Architecting a Multi-Tiered Memory Engine for Agentic Systems: A Hybrid Approach Integrating Memsearch, Hermes, and G-Brain
In the evolution of agentic workflows, the bottleneck is no longer just raw reasoning capability—it is context persistence. For an autonomous agent to function effectively within a professional ecosystem, it requires "perfect memory": the ability to recall decisions made months prior, retrieve specific semantic meanings without exact keyword matches, and provide verifiable citations for every claim.
Current industry-standard implementations, such as Claude Code, offer foundational memory capabilities, but they suffer from structural weaknesses in three critical dimensions: Storage, Injection, and Recall. To solve this, we must move away from monolithic architectures and toward a hybrid framework that cherry-sifts the best engineering patterns from specialized systems like Memsearch, Hermes, and G-Brain.
The Triad of Agentic Memory: Storage, Injection, and Recall
To build a robust memory architecture, we must first decompose "memory" into three distinct functional requirements. Every decision in an agent's lifecycle depends on how these variables are handled.
1. Storage: The Persistence Layer
Storage involves two primary architectural decisions: Trigger Mechanism (Who decides to save?) and Format (How is it saved?).
- Triggering: Should a hook fire automatically after every turn, or should the agent autonomously decide what constitutes a "fact" worth preserving?
- Formatting: Do we store verbatim transcripts (high fidelity, high token cost/bloat) or agent-generated summaries (low cost, potential loss of nuance)?
2. Injection: The Contextual Loading Layer
Injection is the process of moving stored data into the active context window at session start.
- Mechanism: Do we use a persistent hook that loads files every session, or do we rely on an agent to pull from a
claude.mdstyle index? - Constraints: Is there a character/token cap? Without a strict limit, injection leads to context window bloat and increased latency.
3. Recall: The Retrieval Layer
Recall is the search methodology used to find relevant information within the stored corpus.
- Search Modality: We can utilize Keyword Search (exact string matching), Semantic Search (vector-based meaning retrieval), or Hybrid Search (a combination of both).
Deconstructing the Limitations of Claude Code
While Claude Code provides a functional baseline, its out-of-the-box implementation is fundamentally "leaky."
In terms of Storage, it relies on an agent-decided, summarized approach. Because the agent must proactively flag information for saving, any significant data point not explicitly identified by the model is lost to history. This creates a high degree of selectivity that undermines long-term reliability.
Regarding Injection, Claude Code utilizes a hook-loaded method that is essentially unbounded. It loads an index at every session start without a character cap. As the memory grows, this leads to massive context inflation, driving up costs and degrading the model's ability to focus on the immediate task.
Finally, its Recall mechanism is arguably its weakest link. There is no native search layer; retrieval is limited to resuming previous sessions. If the information wasn't explicitly indexed in a memory.md file, it becomes effectively inaccessible without manual intervention.
effectively Engineering the Hybrid Solution
Our objective was to build an "Agentic Operating System" by layering specialized modules over Claude Code’s functional components.
Phase 1: Robust Storage via Memsearch and Haiku Hooks
To solve the "leaky storage" problem, we replaced agent-decided storage with a deterministic automatic hook. Using the Memsearch framework, a hook fires after every single turn in the conversation. We utilize a lightweight, high-speed model—Claude 3 Haiku—to summarize the interaction and append it to a daily log. This ensures that nothing is left to the agent's "opinion"; if it happened in the session, it is captured, condensed, and stored in a persistent daily memory file.
Phase 2: Optimized Injection via Hermes-style Snapshots
To prevent context bloat during injection, we adopted the Hermes approach. Rather than loading an unbounded index, our system takes a "frozen snapshot" of highly pertinent files—such as user profiles, identity definitions, and recent high-priority memories. We implement a strict cap of approximately 1,300 tokens. Furthermore, this context is cached, ensuring that we only pay the inference cost for these tokens once per session, rather than re-injecting them into every subsequent turn.
Phase 3: Multi-Tiered Hybrid Recall with G-Brain Re-ranking
The most significant upgrade lies in our retrieval architecture. We implemented a three-tier system:
- Tier 0 (Immediate): The system first checks the injected "frozen snapshot" for an immediate answer.
- Tier 1 & 2 (Deep Retrieval): If Tier 0 fails, the system initiates a Hybrid Search using Memsearch’s vector-based semantic search combined with keyword matching. This allows the agent to find information based on meaning even when exact terminology is absent.
- The Re-ranking Layer: Drawing inspiration from Garry Tan's G-Brain architecture, we implement a re-ranker that performs a second pass over all retrieved chunks. The system doesn't just return raw text; it returns a synthesized answer complete with citations. This ensures the agent can point to specific files or conversation logs as sources, and—crucially—it allows the agent to admit when information is missing rather than hallucinating.
Scaling to Enterprise: The "Company Brain" Architecture
A memory system for an individual is a luxury; a memory system for a team is a complex distributed systems problem. As we scale this architecture, we face the challenge of Shared vs. Private memory.
We are implementing a centralized "Company Brain" using a Postgres database (via Supabase). To maintain security and privacy across different clients and departments, we utilize Row Level Security (RLS). Every memory row is tagged with metadata (client ID, project ID, or department). When an agent queries the database, the query is scoped by the user's unique access token. This ensures that Person A can only retrieve memories from authorized projects, while still contributing to a unified, searchable organizational intelligence.
By treating memory as a modular, plug-and-play component rather than a locked framework, we have created an architecture that is portable, scalable, and—most importantly—verifiable.