Benchmarking LLM Coding Harnesses: A Comparative Analysis of GPT-5/Opus 4.8/Gemini 3.5 Flash

In the rapidly evolving landscape of AI-assisted software engineering, the "model" is only one component of the execution environment. To achieve high-fidelity code generation and autonomous agentic behavior, we must evaluate the harness—the orchestration layer comprising the IDE, CLI, system prompts, tool-calling capabilities (MCPs), and context injection mechanisms.

This technical deep dive evaluates three frontier models—OpenAI GPT-5.5, Anthropic Opus 4.8 Medium, and Google Gemini 3.5 Flash—across five distinct coding harnesses: Cloud Code, Codex CLI, OpenCode, Cursor, and Anti-gravity. The benchmark task involved generating a Filament mini admin panel using PHP enums within a Laravel framework, specifically testing for tool-calling accuracy, state management (database cleanup), latency, and cost efficiency.

The Methodology: Context Injection via Laravel Boost

To ensure the models were not operating in a vacuum, I utilized Laravel Boost. This package automates the generation of agentic context by creating Agents.md and Cloud.xml files, alongside .agents.cloud directories containing specialized skills for the local environment. By providing these instructions, all tested models had access to specific Laravel best practices and documentation via Context 7 and MCP (Model Context Protocol) implementations.

The benchmark task was a standardized CRUD-adjacent operation: creating an admin panel component with strict PHP enum constraints. This allowed for precise measurement of "hallucination" in logic—specifically, whether the model would deviate from provided constraints during implementation.

Experiment 1: Anthropic Opus 4.8 Medium

Harnesses Tested: Cloud Code, OpenCode, Cursor

The first phase focused on the stability and state management of Opus 4.8 Medium.

Execution and State Management

In Cloud Code, the model demonstrated superior lifecycle management. After executing the necessary tinker commands to create records for testing, it proactively performed a database truncation/cleanup. In contrast, both OpenCode and Cursor successfully generated the code but failed to perform post-execution cleanup, leaving residual test data in the database.

Logic Mutation and Cost

A critical failure was observed in Cursor. While the prompt explicitly required an enum value of review, Cursor mutated this to a "human-friendly" version: in review. This seemingly minor semantic change broke the automated test suite.

From a cost perspective, Opus 4.8 Medium remained relatively stable across environments:

• Cloud Code: ~$0.75 (API pricing)
• OpenCode: ~$0.70 (Direct API)
• Cursor: ~$1.07 (Subscription-based overhead)

Experiment 2: OpenAI GPT-5.5

Harnesses Tested: Codex CLI, OpenCode, Cursor

The second phase evaluated the latency and cost of GPT-5.5 within its native environment versus third-party IDEs.

Latency Regression

Interestingly, while previous iterations (like GPT-5.3) had addressed speed bottlenecks, GPT-5.5 showed a noticeable increase in inference latency during this test. In Codex CLI (the model's native environment), the task took approximately 3 minutes—significantly slower than the 1.5-minute execution seen with Opus 4.8.

Cost Analysis

The cost of GPT-5.5 remains a significant factor for high-frequency development. Using direct API pricing in OpenCode, the cost sat at roughly $0.70, mirroring the efficiency of OpenCode's implementation of Opus. However, Cursor again showed higher overhead, with costs reaching approximately $0.95 for the same prompt, suggesting that Cursor’s orchestration layer may involve additional hidden context or multi-step prompting.

Experiment 3: Google Gemini 3.5 Flash

Harnesses Tested: Anti-gravity, OpenCode, Cursor

The final phase tested the high-speed, low-latency Gemini 3.5 Flash model across different agentic workflows.

Agentic Planning vs. Manual Intervention

Anti-gravity introduced a distinct architectural difference: an explicit "Planning" phase. Before executing any code, the harness generated a structured plan for review. While this increased transparency, it also introduced friction; unlike other harnesses, Anti-gravity required manual confirmation for every terminal command and php artisan script execution, extending the total task duration to roughly 3 minutes.

The "Flash" Cost Paradox

Despite being a "Flash" model intended for efficiency, Gemini 3.5 Flash demonstrated surprisingly high API costs in this test. In OpenCode, using the Medium-tier configuration, the cost was approximately $0.84—higher than the Opus 4.8 Medium runs. This suggests that token consumption during tool-calling and context retrieval can scale aggressively even with lighter models.

Comparative Summary of Results

Conclusion: The Case for Native Harnesses

The data suggests that while "table stakes" tasks (like CRUD generation) are now handled reliably by all frontier models, the harness determines the reliability of the side effects—specifically database cleanup and adherence to strict constraints.

There is a strong correlation between model performance and its native environment. The efficiency seen in Codex CLI for GPT-5.5 and Cloud Code for Anthropic suggests that provider-specific system prompts and optimized data processing pipelines (optimized for the specific model's architecture) provide a more robust developer experience than generic third-party IDE integrations. For developers, the choice of harness is no longer just about UI/UX; it is about managing the hidden costs of latency, token inflation, and stateful execution errors.