Beyond Vector Embeddings: Implementing High-Efficiency RAG via Recency Weighting and BM25 Probabilistic Retrieval
The concept of a "Second Brain" has become a ubiquitous marketing trope, often presented as a magical repository for productivity. However, from an engineering perspective, the challenge of building a functional personal knowledge base (PKB) is not one of storage, but of retrieval precision and context management. When we attempt to augment Large Language Models (LLMS) with proprietary or personal datasets, we encounter significant architectural hurdles: context window degradation, information parity issues, and the prohibitive cost of high-dimensional vector searches.
This post explores a specialized architecture—implemented in a system called Ngram—that moves away from the industry's over-reliance on complex vector databases in favor of a streamlined pipeline utilizing recency weighting, "cheat model" summarization, and BM25 plain text retrieval.
The Context Window Paradox and "Context Rot"
Modern LLMs have seen an explosion in context window capacity, with many consumer models now capable of processing the equivalent of 35 to 50 books (potentially exceeding one million tokens). Theoretically, this suggests that Retrieval-Augmented Generation (RAG) might become obsolete; why bother with complex retrieval when you can simply dump your entire corpus into the prompt?
In practice, two critical bottlenecks emerge: cost and context rot.
First, the computational overhead of processing massive context windows is non-linear in terms of both latency and API expenditure. Running inference on a million tokens for every query is economically unsustainable for most applications.
Second, we encounter "context rot." As the density of information within a prompt increases, the model’s ability to effectively attend to specific, granular details decreases. This phenomenon—where performance degrades as context volume grows—makes it increasingly difficult for the model to isolate a single relevant fact from a sea of tokens. Furthermore, large context windows introduce an "information parity" problem: the model treats all text within the window with equal weight. If your knowledge base contains a product price from 2022 and an updated price from 2024, the model lacks the inherent mechanism to prioritize the most recent data, often leading to confident but incorrect hallucinations based on outdated tokens.
Architectural Solution: Recency Weighting
To solve the information parity problem, we implement recency weighting. By stamping every incoming data point with a temporal metadata tag (a timestamp), we can programmatically influence the retrieval process.
The mechanism involves applying a mathematical boost to documents with recent timestamps and a decay function to older entries. This ensures that when the model or the retrieval engine scans the corpus, it prioritizes information that is temporally relevant. This prevents the "stale data" problem where an LLM might confidently return an obsolete instruction simply because it was the first match found in the text stream.
The Ingestion Pipeline: Connectors and Incremental Updates
A knowledge base is only as robust as its ingestion pipeline. A high-performance system requires consistent, automated connectors—scripts designed to interface with various data sources (e.g., Gmail API, Slack, VoIP transcripts, Git history) and pull new data into the central repository.
The key to efficiency here is incremental synchronization. Rather than re-indexing the entire corpus, connectors should only fetch data produced since the last successful execution. For instance, a Gmail connector using read-only API access can be scheduled to run every 15 minutes, identifying only the delta between the current state and the last recorded timestamp. This minimizes compute load and ensures the knowledge base remains "live."
The Summarization Layer: Utilizing "Cheat Models"
Raw text is notoriously difficult to search via keyword or probabilistic methods. To bridge this gap, we introduce a transformation layer using what can be termed a "cheat model."
In our architecture, we utilize a lightweight, cost-effective model—specifically Claude Haiku 5.5 (or its predecessors like 4.5)—to process incoming raw data before it reaches the primary retrieval index. The goal is to transform long-form, unstructured text into highly searchable metadata. This involves two specific outputs:
- Summarization and Rephrasing: The cheat model generates a concise summary of the topic, often rephrasing the content to capture the core intent. It also anticipates potential user queries (e.g., "What is the protocol for X?") and embeds these as part of the searchable metadata.
- Atomization: This involves extracting "atoms"—discrete, high-value units of information such as exact error messages, specific CLI commands, or critical architectural decisions made during a development cycle.
By indexing these summaries and atoms rather than just raw text, we significantly reduce the search space for the primary LLM. When a query is made, the model searches the lightweight summaries first; once a match is found, it is then pointed to the original, full-text source. This two-step process drastically reduces inference time and improves accuracy.
The Death of Vector Embeddings? Implementing BM25
Perhaps the most controversial aspect of this architecture is the move away from vector embeddings and vector databases (like Pinecone or Milvus) in favor of BM25 (Best Matching 25).
While vector embeddings are excellent at capturing semantic similarity, they introduce significant complexity: you must manage embedding models, handle high-dimensional vector storage, and incur the costs of generating embeddings for every new piece of data.
BM25 is a probabilistic retrieval function used in information retrieval to rank documents based on term frequency (TF) and inverse document frequency (IDF). It has been a staple of search engineering since the 1990s. In our testing, implementing BM25 over plain text strings provided:
- A 9/10 reduction in inference latency.
- A 9/10 reduction in maintenance costs associated with vector DB management.
By using Claude (or any capable LLM) to perform an initial, extremely fast bit of inference to pick the correct search terms, we can use BM25 to scan our massive strings of text for those terms. The result is a system that feels instantaneous and provides highly accurate matches without the overhead of a complex neural retrieval architecture.
Conclusion: Building Institutional Memory
The goal of this architecture is to transform ephemeral data—Slack messages, email threads, meeting transcripts—into institutional memory. By combining incremental connectors, recency-weighted metadata, cheat-model summarization, and BM25 retrieval, we create a system that does not just store information but actively understands its relevance and freshness. This approach provides the scalability of an enterprise-grade RAG system with the simplicity and speed of a local text search.