title: "Evaluating Agentic Reasoning in Large-Scale Codebases" date: 2026-10-26 tags: [ai, llm, benchmarking, software-engineering] description: "A deep dive into how different LLMs handle complex, long-running coding tasks within a legacy PHP/Laravel environment."
The Benchmark Problem: Moving Beyond Simple Prompts
Current LLM leaderboards often suffer from a lack of complexity. Most benchmarks rely on short-context prompts and small codebases where models can achieve near-perfect scores by simply following instructions. To truly evaluate the "agentic" capabilities of an LLM—its ability to navigate, reason through, and modify existing architecture—we must move toward long-running tasks involving significant technical debt and large-scale repositories.
In this experiment, I moved away from simple single-file prompts to a task centered on BookStack, an open-source project consisting of approximately 45,000 to 50,000 lines of code (LOC). This codebase is mature, featuring commits dating back over a decade, presenting a significant challenge for models attempting to maintain architectural consistency.
Experimental Methodology
The objective was to simulate a real-world client request: implementing a new feature within the BookStack API that requires hardening validation and extending existing functionality. The prompt was intentionally designed with "ambiguous" requirements—a common occurrence in professional software engineering—forcing the model to rely on codebase discovery rather than explicit instructions.
The Testbed
- Codebase: BookStack (PHP/Laravel), ~5/50k LOC.
- Task Complexity: Implementing new API logic, discovering existing patterns, and ensuring no regressions in a legacy environment.
- The Models: Eight models were selected for testing:
- Anthropic Claude 3 series (Opus, Sonnet, Haiku at varying effort levels).
- Sol (Medium level).
- DeepSeek Flash (High and Max levels).
- Kimi K3.1 Pro.
- GLM 5.3.
Evaluation Framework
The evaluation utilized a dual-layer verification system:
- Deterministic Verification: A suite of 47 hidden PHPUnit tests was executed. These tests were invisible to the models and covered scope, permissions, API integrity, and regression testing (ensuring existing tests remained green).
- AI-Driven Qualitative Analysis: An independent judge—GPT 6.1 Sol Medium—evaluated the code quality based on three primary pillars:
- Architectural Fit: Did the model respect the existing BookStack/Laravel patterns?
- Design Integrity: Was the new code clean, maintainable, and logically sound?
- Verification Rigor: How thorough were the model's own generated tests and its execution of the test suite?
Results: Performance vs. Efficiency
While all eight models successfully passed the hidden PHPUnit functional tests—demonstrating that modern LLMs are highly capable of executing specific instructions—the divergence in code quality and operational cost was profound.
The Leaderboard Breakdown
1. The Gold Standard: Claude 3 Opus
Claude 3 Opus (Medium level) emerged as the clear leader. It demonstrated a superior balance between tool-call efficiency and architectural adherence. Unlike models that "run in circles," Opus achieved high quality with an average number of tool calls, making it the most efficient model for complex, high-stakes engineering tasks where maintainability is paramount.
/2. The Agentic Outlier: Kimi K3.1 Pro
Kimi K3.1 Pro presented a fascinating case study in autonomous agent behavior. It was the only model that automatically instantiated sub-agents without explicit prompting. This resulted in an extremely high number of tool calls (editing files, checking docs, running tests) and a significantly higher time duration. However, despite the heavy computational footprint, its API cost remained surprisingly low compared to GLM 5.3, likely due to optimized token usage or pricing structures for sub-agent workflows.
3. The Cost Disparity: DeepSeek Flash vs. The Field
The most striking metric was the economic variance. For the exact same task, DeepSeek Flash achieved a cost of approximately $0.16, whereas other models like GLM 5.3 reached costs as high as $10.00. While DeepSeek is undeniably the leader in price-to-performance for simple tasks, it struggled with code quality. The "Max" level model suffered from significant point deductions due to:
- Inconsistency: Implementing redundant rules across multiple files.
- Verification Gaps: Writing tests that skipped risky edge cases or failed to run the full suite after changes.
4. Effort Level vs. Model Scale
A critical takeaway was the impact of "effort levels" (e.g., Haiku High vs. Sonnet Medium). We observed that Claude 3 Haiku (High level) outperformed Sonnet (Medium level) in code quality. This suggests that for complex engineering, the depth of reasoning/thinking time allocated to a model can outweigh the raw parameter count or "intelligence" of a larger, less-intensive model instance.
Technical Deductions and Maintainability Risks
The AI Judge's scoring was heavily influenced by "maintainability debt." Points were deducted based on several critical failure modes:
- Pattern Neglect: Models failed to utilize existing Laravel framework functionality or pre-existing BookStack helper methods, opting instead for "hard-coded" or redundant logic.
- Verification Failure: Several models (notably Sonnet and DeepSeek) were penalized for not running the full test suite after implementing changes, or for writing tests that only covered "happy paths" while ignoring destructive edge cases (e.g., deletion vs. restoration of resources).
- Architectural Drift: The introduction of inconsistent logic—where a rule was applied in one controller but ignored in another—was heavily penalized as it poses a long-term risk to the codebase's integrity.
Conclusion
The experiment proves that for large-scale, long-running coding tasks, the metric of success is not merely "passing tests," but "maintaining architecture." While DeepSeek Flash offers unmatched cost efficiency, and Kimi K3.1 Pro demonstrates the future of autonomous sub-agents, Claude 3 Opus remains the most reliable choice for high-quality, maintainable software engineering in complex environments.