Beyond Auto-Memory: Architecting an LLM-Agnostic Multi-Agent Orchestration System via Local Context Injection and agents.md

In the rapidly evolving landscape of Large Language Models (LLMs), a critical tension has emerged between convenience and control. Most users rely on the "Auto-Memory" features provided by desktop applications like ChatGPT and Claude—features designed to proactively learn from chat histories and generate persistent user profiles. However, for high-stakes professional workflows, this automated approach introduces significant entropy. When an AI decides which context to retain and which to discard, it introduces non-deterministic behavior into what should be a deterministic workflow.

To achieve true reliability in agentic workflows, we must move away from cloud-based "auto-memory" and toward a structured, local, folder-based architecture. By leveraging the recent convergence of instruction standards—specifically Claude’s adoption of agents.md—we can build an LLM-agnostic orchestration layer that remains stable regardless of whether you are running GPT-6 Astra, Fable 5.1, or Gemini.

The Death of Proprietary Instructions: The Rise of agents.md

Historically, implementing custom instructions required model-specific configurations: claude.md for Anthropic models and specific "Custom Instructions" or Codex settings for OpenAI’s ecosystem. This fragmentation made it impossible to swap models without re-engineering the entire context window.

A significant shift is currently underway. Claude has begun recognizing agents.md files, aligning its behavior with the directory-based instruction sets previously utilized by OpenAI's Codex and ChatGPT projects. This convergence allows us to treat a local folder as a "source of truth" for any LLM pointed at that directory. When an agent enters a folder, it doesn't just see text; it sees a structured hierarchy of instructions, roles, and standard operating procedures (SOPs).

The Orchestrator-Sub-Agent Pattern

The core of this architecture is the Orchestrator Pattern. Rather than prompting a single LLM to perform every task—which inevitably leads to "context bloating" and instruction degradation—we implement a hierarchical delegation system.

1. The Orchestrator (The Router)

In my implementation, we utilize an agent called Larry. Larry serves as the primary interface and orchestrator. His role is not to execute tasks but to act as a high-level router. When a prompt enters the system, Larry’s objective is to:

• Analyze the intent of the request.
• Consult the agent_index.md (the registry of available capabilities).
• Identify and launch the appropriate sub-agent.

ually 2. The Sub-Agent (The Specialist)

Once Larry identifies a task—for example, generating a LinkedIn post—he delegates the work to a specialized agent, such as Sage. Unlike the orchestrator, Sage is loaded with highly specific context: tone of voice, hashtag strategies, and formatting constraints. This separation of concerns ensures that the orchestrator remains "light" and capable of managing multiple parallel streams without losing track of high-level objectives.

Hierarchical Context Injection and Token Optimization

One of the greatest challenges in agentic design is the "token tax." If you provide every sub-agent with your entire knowledge base, you rapidly exhaust the context window and increase latency/cost. To solve this, we implement a Hierarchical Knowledge Retrieval system:

1. Root Level (agents.md): Defines the orchestration contract and the existence of the team.
2. Agent Level (agent.md): Defines the specific persona and mission of a sub-agent (e.g., "Sage, the LinkedIn Specialist").
3. Guideline Level (GL003, etc.): Sub-agents pull only necessary guidelines from a centralized knowledge/guidelines directory. For instance, an infographic designer agent would pull GL003: Design System but ignore GL004: Copywriting Tone.
4. SOP Level: The final layer contains step-by-step execution instructions (Standard Operating Procedures) and workstreams.

By only loading the specific .md files required for a particular task, we achieve massive token savings and prevent "instruction drift," where an LLM becomes confused by irrelevant information in its prompt.

Experimental Validation: Terminal vs. Desktop UI

To ensure our architecture was truly model-agnostic and not merely a byproduct of the desktop application's UI/UX, I conducted tests using the CLI (Command Line Interface) via codex and claude. By running both models in a terminal environment within the same directory, we eliminated variables like "search and reference chats" or "generate memory from chats."

The results were telling. When testing GPT-6 Astra against Fable 5.1:

• Astra (Medium Effort): Demonstrated superior capability in generating visual representations and animations for slide decks but tended toward a more generic, less nuanced output when tasked with complex writing.
• Fable 5.1: Showed higher adherence to the specific "decision block" instructions found in my local files, though it occasionally struggled with over-verbosity (the "wordiness" problem common in high-parameter models).

Crucially, both models were able to recognize and execute the agents.md contract. This proves that the intelligence resides not in the model's pre-trained weights alone, but in our ability to inject structured, local context into the inference process.

Conclusion: Building for Resilience

The goal of this architecture is resilience. If a new model—say, Gemini or a future iteration of Claude—is released tomorrow, you do not need to rebuild your workflows. You simply point the new model at your existing folder structure. Your orchestrator, your sub-agents, your SOPs, and your guidelines remain intact.

By treating LLMs as interchangeable "inference engines" and keeping the logic in a local, version-controlled agentic framework, you move from being a "prompt engineer" to an "AI systems architect." This is how we scale productivity without sacrificing precision.