Beyond Keyword Matching: Engineering a Semantic Memory Architecture for Claude Code via Vector Embeddings and PGVector
The rapid ascent of the Hermes Agent—surpassing 211,000 GitHub stars in just five months—has highlighted a critical deficiency in current AI coding assistants: the "memory gap." While tools like Claude Code offer unparalleled reasoning capabilities, their native ability to maintain long-term context is fundamentally limited. Analysis of developer sentiment reveals that the primary driver for migrating from OpenClaw to Hermes isn't integration depth or autonomous skill execution; it is the persistence of context.
This post details the technical architecture used to rebuild a high-fidelity memory system directly within Claude Code, addressing the specific architectural failures found in existing agentic frameworks like Hermes and implementing a semantic recall engine using PGVector.
The Triad of Agentic Memory: Storage, Injection, and Recall
To build an effective memory system, we must move beyond treating "memory" as a single feature. Instead, it must be engineered through three distinct operational layers:
- Injection (Short-term Context): How information is loaded into the active context window at session initialization to ensure immediate relevance without inducing context bloat.
- Storage (Persistence): The mechanism by which significant decisions, preferences, and facts are extracted from a conversation stream and committed to durable storage.
- Recall (Retrieval): The retrieval logic used to query historical data when the current context lacks the necessary information.
Deconstructing the Hermes Agent Architecture
While Hermes Agent excels at storage and injection, it possesses significant architectural bottlenecks that make it difficult to scale in a production-grade developer workflow.
The FTS5 Limitation
Hermes utilizes an SQLite database with FTS5 (Full-Text Search) for session retrieval. While efficient for keyword matching, FTS5 is fundamentally incapable of semantic reasoning. If a user queries "How do I handle payments?" but the historical conversation used the term "Stripe," the agent fails to retrieve the relevant context. This reliance on exact token matching creates a brittle recall loop.
The Runtime and Security Surface
Hermes operates as a separate runtime, often requiring a dedicated VPS, its own model billing (separate from your primary Claude subscription), and an independent update cycle. Furthermore, its "self-rewriting" capability—where the agent edits its own skill sets—introduces a high risk of information degradation, where the agent may inadvertently overwrite correct instructions with hallucinated or suboptimal logic.
The Proposed Architecture: A Semantic Approach
Our implementation moves away from keyword-based retrieval toward a semantic vector space, integrated directly into the Claude Code environment to maintain portability and minimize latency.
1. Optimized Injection via Frozen Snapshots
To prevent context window exhaustion, we implement a "frozen snapshot" strategy. We maintain a memory.md file with a strict 2,500-character cap. At session start, this file is injected alongside the user profile and current date metadata. By capping the character count, we ensure that high-priority information (e.g., active client lists, pricing models, architectural decisions) is always present in the prompt without triggering the "lost in the middle" phenomenon common in large context windows.
2. Intelligent Storage via Automated Hooks
We do not rely on manual updates. Instead, we implement a post-turn hook that evaluates the importance of the most recent interaction. The agent utilizes a meta memory write skill to determine if a piece of information qualifies as a "durable fact." If a decision is identified (e.g., "Switching from REST to gRPC"), the system:
- Checks for existing duplicates in
memory.md. - Promotes the new fact to the short-term memory file.
- Replaces outdated or less relevant information to respect the 2,500-character limit.
For long-term storage, every conversation turn is summarized and archived alongside a complete transcript in an efficient JSON format. This ensures that while the summary provides quick retrieval, the raw transcript remains available for deep context expansion.
3. Semantic Recall with PGVector and Re-ranking
The core of this rebuild is the transition from FTS5 to a PGVector-based architecture using PostgreSQL. Every conversation summary and transcript segment is transformed into high-dimensional vector embeddings.
Our retrieval pipeline follows a multi-stage process:
- Vector Search: When a query is made, we perform a cosine similarity search within the PGVector set to find semantically related nodes.
- Re-ranking: We implement a re-ranking layer to prioritize results that demonstrate high relevance to the specific intent of the query.
- Context Expansion: Once a relevant node is identified, the system performs "context expansion," pulling in neighboring conversation segments from the JSON transcript tree to provide the agent with the full surrounding dialogue.
4. Legacy Integration: The memory import Pipeline
A significant hurdle in adopting new memory systems is the loss of historical data. We addressed this by building an ingestion pipeline: npm run memory import sessions. This utility scans existing Claude Code history (up to 30 days) and uses Claude Haiku to summarize past interactions. These summaries are then embedded into our PGVector database, ensuring that the agent's "day one" knowledge includes months of previous context.
Conclusion: The Future of Agentic Continuity
The value of an AI agent is not found in its ability to call tools or integrate with third-party APIs; it is found in continuity. By implementing a system that prioritizes semantic recall, manages injection via capped snapshots, and provides verifiable source citations (citing exact transcripts and dates), we transform Claude Code from a stateless chat interface into a persistent, evolving "Team OS."