Benchmarking Frontier LLMs in Complex Software Engineering Tasks: A Comparative Analysis of Agentic Code Generation via Single-Prompt Execution

In the rapidly evolving landscape of Large Language Models (LLMs), a critical question remains for engineers: how do these models perform when tasked with high-complexity, multi-file software engineering projects without iterative human intervention? This study presents an empirical evaluation of several frontier and emerging models—specifically Claude Opus 4.8, OpenAI GPT 5.5, Gemini 3.1 Pro, Composer 2.5, Kimi 2.5, and xAI Grok Build 0.1—to determine their capability in generating a functional, asset-integrated game engine from a single prompt.

Experimental Methodology and Environment

To ensure a controlled environment, the experiment was conducted using the Cursor IDE as the primary coding harness. This approach eliminates variance introduced by different orchestration layers (such as Claude Code or custom agents) and focuses purely on the model's ability to utilize the provided context within a standardized development environment.

The Technical Challenge: Mario Kart Clone

The objective was the generation of a "Mario Kart" style game, selected for its medium complexity, requirement for asset management, and necessity for robust physics/collision logic.

Constraints & Parameters:

• Prompting Strategy: A one-shot prompt containing core requirements, stretch goals, and technical specifications. A single secondary follow-up prompt was permitted only in the event of total functional failure.
• Configuration: All models were set to "Maximum Thinking" mode (where available) and utilized the highest available inference speed/versioning to ensure parity in computational effort.
• Asset Integration: Models were provided with a pre-defined directory containing sprite sheets, tile sets, and a manifest.json defining hazards and items. The requirement was for "zero-setup" execution—the generated code had to be playable immediately upon file creation.
• Metrics of Evaluation:
  1. Code Quality: Evaluated via an independent AI-driven judge focusing on modularity, error handling, and adherence to the manifest.
  2. Latency (Completion Time): Total time from prompt submission to code finalization.
  3. Complexity Metrics: Total Lines of Code (LOC) and File Count.
  4. Efficiency: LOC per minute of execution.

Initial Results: The Failure of One-Shot Complexity

The first pass revealed a significant gap between "chat-capable" models and true "engineering-capable" agents.

• Claude Opus 4.8 & Composer 2.5: Both failed to produce an executable state, resulting in blank canvases or runtime errors within the browser console.
• Gemini 3.1 Pro: Achieved a functional execution state with basic movement and drifting mechanics via spacebar input, but suffered from significant logic regressions and broken gameplay loops.
• Grok Build 0.1: Successfully integrated audio assets but failed to implement any meaningful player controller or physics interaction.
• Kimi 2.5: Produced a static map rendering but lacked all interactive event listeners (keyboard/mouse).
• GPT 5.5: The only model to provide a functional, playable loop in the first pass. It implemented basic movement, item usage (via 'E' key), and rudimentary collision detection with grass textures, though it failed to integrate the full provided asset pack.

Iterative Refinement: Evaluating Agentic Recovery

Following the initial failure, a secondary prompt was issued to all models: "Make it work; make it playable." This test evaluated the "self-correction" capabilities of each model's reasoning engine.

The Emergence of Modular Architecture (Opus 4.8)

The most significant technical achievement was observed in Claude Opus 4.8. Despite having the highest latency (23 minutes, 18 seconds), it demonstrated a level of architectural sophistication far exceeding its competitors. Rather than simply writing a monolithic JavaScript file, Opus generated:

• A modularized game engine architecture.
• Custom Python-based utility scripts designed to parse and analyze the provided asset directory.
• Advanced physics implementation including complex collision detection (grass vs. track) and drifting mechanics via Shift.
• Full integration of the provided sprite sheets and sound effects as defined in the manifest.

The Efficiency/Functionality Trade-off (GPT 5.5 & Kimi 2.5)

GPT 5.5 maintained its position as a highly efficient engineer. With an execution time of approximately 6 minutes, 15 seconds, it delivered a robust, playable experience with functional physics and item usage, though it lacked the deep asset-parsing logic seen in Opus.

Conversely, Kimi 2.5 demonstrated extreme efficiency in terms of LOC per minute but failed to deliver any meaningful software utility. While its execution was rapid (~6 minutes), the resulting code lacked the necessary event listeners for user interaction.

Quantitative Analysis and Conclusion

The following table summarizes the performance metrics observed during the experiment:

Key Technical Takeaways

The experiment highlights a widening divergence in the "Frontier" class of models. We observed that while lightweight or highly optimized models (Gemini, Kimi) excel at rapid code generation and high LOC/min throughput, they lack the reasoning depth required for complex dependency management and asset orchestration.

Claude Opus 4.8 represents a shift toward Agentic Software Engineering, where the model does not just write code but builds the surrounding tooling (e.g., Python parsers) necessary to support the primary application. However, this comes at the cost of significant computational latency. For developers, the choice remains a trade-off between the rapid prototyping capabilities of models like GPT 5.5 and the deep, modular engineering potential of the Claude Opus architecture.