From Prompt Engineering to Agentic Orchestration: Evaluating Autonomous Research via Lemma’s FARS

The era of "vibe coding"—where high-level natural language descriptions are translated into functional software by LLMs—is rapidly evolving. As we move beyond simple code generation, a new paradigm is emerging: Vibe Research. This transition shifts the focus from merely generating syntax to orchestrating complex, multi-step scientific inquiry. At the forefront of this shift is Lemma, a multi-agent research platform developed by Anilemma AI, designed to automate the entire lifecycle of scientific experimentation, from literature review and hypothesis formulation to code execution and formal paper synthesis.

Beyond the Wrapper: The Multi-Agent Research Paradigm

Most contemporary AI tools operate on a single-turn or limited-context request-response architecture. Whether they are utilizing RAG (Retrieval-Augmented Generation) to search the web or calling specific API tools, the interaction remains fundamentally linear. Lemma diverges from this "wrapper" architecture by implementing a multi-agent research system.

When a research query is initiated, the platform does not simply generate a response; it instantiating an autonomous team of specialized AI agents. This orchestration layer manages several concurrent workstreams:

1. Planning & Strategy: Deconstructing the prompt into actionable experimental steps.
2. Literature Synthesis: Searching and reading academic papers to establish context.
3. Code Execution & Experimentation: Writing, deploying, and running code within a sandboxed environment.
4. Validation & Iteration: Analyzing results against the initial hypothesis and refining the methodology.

This "AI for AI" approach creates a self-improving loop where the system's output is used to refine its own subsequent research tasks.

The Four Modes of Autonomous Inquiry

Lemma categorizes its operational capabilities into four distinct modes, each varying in computational depth and temporal requirements:

• Explore: A low-latency mode (1–3 minutes) designed for rapid report generation and high-level overviews.
• Survey: A deep-dive mode capable of generating long-form academic surveys with extensive citation coverage across broad datasets.
• Code: An experimental mode that bridges the gap between theoretical research and practical implementation by executing code in a virtualized environment.
• FARS (Fully Automated Research System): The most complex tier, designed for multi-day autonomous experiments that culminate in professional-grade research papers complete with statistical analysis and visualizations.

Case Study 1: Computer Vision Implementation via 'Code' Mode

To demonstrate the efficacy of the Code mode, an experiment was conducted to develop an image classifier capable of distinguishing between AI-generated imagery and authentic photographs. Unlike standard LLM coding assistants that immediately output a script, Lemma’s agentic workflow began with methodology research—analyzing existing computer vision techniques to determine the optimal architecture for the task.

Technical Implementation Details

The system autonomously configured a virtual execution environment (accessible via an integrated VS Code interface and bash terminal) and implemented the following:

• Architecture: MobileNetV1-Small (0.5 scale), chosen for its efficiency in CPU-based training environments.
• Dataset Composition: A curated set of 300 images, split equally between 150 real photographs and 150 AI-generated samples.
• Execution Pipeline: The agents handled dependency installation, dataset generation, model training, and hyperparameter tuning.

Results and Observability

The platform provided full observability into the training process, including loss and accuracy curves. The resulting model demonstrated high precision in classifying the test set, with the system providing a visual grid of predictions (True Label vs. Predicted Label). This capability highlights Lemma's utility as a tool for rapid prototyping of machine learning models without local infrastructure overhead.

Case Study 2: Hallucination Mitigation via FARS and Binary Abstention

The most significant demonstration involved the FARS mode, where an autonomous experiment was designed to investigate whether explicit uncertainty prompting could reduce LLM hallucinations. The research objective was to compare a standard baseline against "uncertainty-aware" prompts using two distinct model architectures: Qwen 2.5 (72B) and GPT-4o.

Experimental Methodology

The FARS agent orchestrated a multi-step experimental design involving three prompting conditions:

1. Baseline: Standard instruction following.
2. Loose Uncertainty: Allowing the model to use hedged language (e.g., "I believe," "It is possible").
3. Strict Binary Abstention: A zero-cost intervention forcing the model into a binary state—either provide a definitive answer or explicitly refuse to answer based on confidence thresholds.

The system managed the entire lifecycle: generating a research proposal, executing 10+ experimental steps (including environment setup and prompt injection), and synthesizing a nine-page formal paper.

Key Findings and Metrics

The research yielded critical insights into Hallucination Coverage Efficiency (HCE). The primary metrics analyzed included coverage, false answer rate, and the impact of prompting on error rates.

• Reduction in False Answer Rate: The implementation of "Binary Abstention" proved highly effective, reducing the false answer rate by an estimated 18% to 51% across both Qwen 2.5 (72B) and GPT-4o.
• The Regime Boundary Trade-off: While binary abstention significantly mitigates hallucinations in structured QA tasks, it introduces a "regime boundary" error. The model began over-abstaining on knowledge-intensive, long-tail queries where the uncertainty threshold was met too easily, thereby reducing overall coverage.

Conclusion: The Democratization of Scientific Research

The emergence of platforms like Lemma represents a fundamental shift in how scientific inquiry is conducted. By automating the heavy lifting of literature review, experimental execution, and statistical synthesis, we are witnessing the democratization of professional-grade research. While the "self-improving loop" of AI researching AI presents complex ethical and reliability challenges, the ability to execute multi-day, high-complexity experiments via a single prompt marks the beginning of a new era in autonomous scientific discovery.