Optimizing Context Engineering: Mitigating Instruction Conflict and Token Bloat in Claude Code via Progressive Disclosure

In the rapidly evolving landscape of AI-driven development, the efficiency of an "AI Operating System" depends heavily on how context is managed. Recent insights from Boris Cherny, Head of Claude Code, suggest a radical approach to maintaining high-performance environments: the periodic deletion and reconstruction of CLAUHD.md files. While this may seem counterintuitive to developers accustomed to building robust instruction sets, the underlying technical justification lies in the complexities of context engineering, instruction conflict resolution, and the optimization of token usage through progressive disclosure.

The Conflict Resolution Overhead

One of the primary challenges in complex AI environments is the presence of overlapping or contradictory instructions across multiple layers of documentation. In a typical setup, an agent might ingest information from CLAUDE.md, various skill files, system prompts, and environmental configuration files.

When these documents contain disparate rules—for example, a rule in CLAUDE.md mandating a specific coding style that conflicts with a more recent instruction embedded within a specialized "skill" file—the model is forced into a computational overhead phase. Before executing the primary task, the LLM must attempt to resolve these logical discrepancies. This process of conflict resolution consumes significant reasoning capacity and can lead to "hallucinated" logic or degraded performance, as seen in some observed regressions in models like Opus 5 when faced with high-entropy instruction sets.

The Constraint Trap: Few-Shot Learning vs. Model Agency

Historically, the standard for prompt engineering involved providing extensive examples (few-shot learning) to ensure output consistency. While effective for pattern matching, over-reliance on these examples introduces heavy constraints that can stifle a model's reasoning capabilities.

By feeding an LLM too many rigid templates, we essentially narrow its operational manifold, preventing it from utilizing the advanced reasoning and "zero-shot" capabilities inherent in frontier models. The goal of modern context engineering should be to provide enough structure for consistency without creating a "constraint trap" that prevents the model from exploring more efficient or creative solutions. As models evolve, instructions that were necessary six months ago may now act as anchors, dragging down the performance of much more capable architectures.

Implementing Progressive Disclosure and Lazy Loading

To combat token bloat, we must move away from monolithic context files toward a paradigm of Progressive Disclosure. This technique is analogous to "lazy loading" in traditional software engineering.

In an optimized AI skill architecture, the model should not ingest the entire payload of every available tool at the start of a session. Instead, the system utilizes a tiered retrieval approach:

1. Metadata/Description Layer: The model first reads only the high-level descriptions of available skills (e.g., "AI News Monitor"). This allows for rapid intent matching with minimal token expenditure.
2. Targeted Ingestion: Once the model identifies the correct skill, it then loads the full Markdown body, associated references, and necessary assets.

This approach significantly reduces the initial context window pressure. If a CLAUHD.md file contains massive amounts of static data (such as pricing, client lists, or stylistic guides) that are not relevant to every task, those tokens are essentially wasted during every single chat initialization. By migrating this "global" context into specialized skills, we ensure that the information is only loaded when the routing logic necessitates it.

Empirical Testing: Retrieval and Routing Accuracy

To validate the efficacy of pruning CLAUDE.md, two critical metrics were tested within a Claude Code environment: Retrieval and Routing.

1. Retrieval Integrity

The first test examined whether the model could still access essential information if it was removed from CLAUDE.md and moved into specialized skill files. The results indicated that for most standard operational parameters, retrieval remained highly accurate. However, a critical edge case was identified: if specific, unique instructions exist only in CLAUDE.md without a corresponding reference elsewhere, the model may produce "confidently wrong" answers rather than admitting ignorance.

2. Routing Efficiency

The second test focused on whether removing instructional bloat affected the model's ability to trigger tools and skills correctly. The findings suggested that as long as the skill descriptions were well-maintained, routing remained largely unaffected. Interestingly, a slight regression was noted in certain script-checking behaviors when instructions were stripped too aggressively, suggesting that some "pointer" instructions (directing the model where to look) are still necessary for complex workflows.

The /doctor Protocol: Auditing your AI Environment

To manage this complexity, we can utilize an automated audit tool—a forward slash doctor skill. This utility performs a comprehensive health check on the AI operating system's environment, focusing on several key technical metrics:

• Token Density Analysis: Identifying overly verbose skill descriptions that contribute to unnecessary token consumption (e.g., detecting when 88 skills alone consume 19.3K tokens just in descriptions).
• Duplicate Detection: Flagging redundant installations or overlapping permission rules (such as multiple Claude versions or conflicting live super-based tokens).
• Hook Error Rates: Monitoring the stability of environment hooks and identifying those that fail during execution.
• MCP Observability: Analyzing Model Context Protocol (MCP) call frequency over a 30-day window to identify underutilized tools.

Conclusion: A Strategy for Maintenance

The objective is not the total eradication of CLAUDE.md, but rather its transformation into a lightweight directory of pointers. Developers should retain "safety gates"—rules that prevent irreversible actions or require human-in-the-loop intervention—within the primary configuration. However, all other operational context, stylistic guides, and data-heavy instructions should be migrated to a modular, progressively disclosed skill architecture. By treating your AI context as a dynamic, scalable system rather than a static document, you enable frontier models to operate at their maximum theoretical capability.