Architecting LLM-Agnostic Intelligence: Implementing a Localized Personal Knowledge Assistant (PKA) via Agentic Orchestration

In the current generative AI landscape, most users are inadvertently "renting" their intelligence. By relying on proprietary features like Claude’s auto-memory or ChatGPT’s custom instructions, users create a state of vendor lock-in where their context, learned preferences, and historical data are trapped within a specific provider's ecosystem. This dependency creates significant risks regarding data portability, security, and long-term knowledge sovereignty.

To solve this, we must shift from a model of "renting" to "owning." The solution lies in the implementation of a Personal Knowledge Assistant (PKA)—a tool-agnostic, local-first folder structure that utilizes an agentic orchestration layer to manage persistent memory, task execution, and knowledge synthesis independently of the underlying Large Language Model (LLM).

The Architecture of Ownership: The Localized Folder Structure

The core of a PKA is not the LLM itself, but the structured filesystem it operates upon. By utilizing a standardized Markdown-based directory, we can create a "Single Source of Truth" (SSOT) that remains functional whether you are using Claude (Sonnet or Opus), Gemini, or OpenAI’s Codex.

The architecture is divided into several critical functional directories:

1. The Deliverables/Inbox Layer: This serves as the owner's intake manifold. It contains the deliverables folder for completed agent tasks and an inbox for raw, unprocessed data (scanned documents, screenshots, or meeting transcripts).
2. The Team Directory: This is the agentic orchestration layer. Rather than relying on opaque, pre-made plugins, this directory contains agent.md files that define the roles, instructions, and capabilities of specific autonomous agents.
3. The PKM (Personal Knowledge Management) Layer: Built on the "My Life" concept, this directory manages the high-level ontology of the user's world, including interests, topics, goals, habits, and key_elements (family, business, etc.).
4. The Document/Asset Layer: A structured repository for images, PDFs, and screenshots, which are referenced via Markdown embeds or Wiki-links, ensuring the AI can surface relevant visual context.

Agentic Orchestration: The Larry-Nolan-Pax Triad

A sophisticated PKA requires more than just a single prompt; it requires a hierarchy of specialized agents. This is achieved through an orchestration pattern where a primary agent manages the routing of tasks to specialized sub-agents.

• Larry (The Orchestrator): Larry acts as the primary interface and router. He maintains the agents.md index. When a task is received, Larry analyzes the request, consults the agent registry, and determines which specialized agent is best suited for the execution.
• Nolan (The Recruiter/Hiring Agent): Nolan provides the system with dynamic scalability. If Larry identifies a gap in the current agent roster (e.g., a need for a front-end developer or a legal expert), Nolan is tasked with "hiring" the agent—essentially generating the necessary .md instruction files and integrating them into the agents.md index.
• Pax (The Researcher): Pax is the interface between the local environment and the external web, capable of performing deep-dive research to feed information back into the local knowledge graph.

This hierarchy allows for a modular expansion of capabilities. You can add specialized agents like Pen (the Journal Writer) or Felix (the Front-end Developer) without ever altering the core orchestration logic.

Achieving LLM Agnosticism via the Adapter Pattern

The most critical technical component for portability is the adapter_prompt.md file. This file implements an "Adapter Pattern" for LLMs. Instead of writing complex, model-specific system prompts, you provide a lightweight instruction set that tells any LLM—be it Claude, Gemini, or Codex—to "initialize" itself within the local folder.

When an LLM is pointed at this folder, it reads the adapter_prompt.md, identifies the agents.md registry, and begins to understand the local directory structure. This approach offers two massive advantages:

1. Token Efficiency: By avoiding the loading of massive, bloated "skill" libraries or plugins, you only load the specific context required for the current task, significantly reducing context window consumption and cost.
2. Model Portability: You can switch from Claude to Gemini mid-session. Because the "intelligence" resides in the local Markdown files and the adapter_prompt, the model is merely a transient compute engine acting upon a permanent, local knowledge base.

Replacing Auto-Memory with Structured Session Logs

The most controversial but necessary move is to switch off AI auto-memory. Proprietary auto-memory is non-deterministic and lacks structural integrity; it "grasps" what it deems useful, leading to fragmented and unorganized context.

Instead, the PKA utilizes Structured Session Logs. Every interaction is recorded in a session_logs directory within the team_knowledge folder. These are standard Markdown files that document the conclusions, decisions, and outcomes of every session.

This provides a "Time Machine" capability. Because these logs are structured and text-based, you can instruct an LLM to "review the session logs from last Tuesday" to reconstruct the context of a previous project. This is far more efficient and reliable than relying on a black-box memory feature.

Data Interconnectivity: From Markdown to SQLite

While the initial implementation relies on Markdown and Wiki-links (making it natively compatible with tools like Obsidian for visual knowledge graph exploration), the architecture is designed for professional-grade scaling.

For high-density data management, the system can be upgraded to an SQLite database structure. By transitioning from flat Markdown files to an SQLite backend, the AI can perform complex relational queries, making it much more efficient at surfacing deep connections between entities (e.g., linking a person in the CRM to a specific meeting note and a subsequent project task).

By decoupling the knowledge from the model, we move from a state of digital dependency to a state of digital sovereignty. The PKA is not just a way to use AI; it is a way to build a permanent, scalable, and model-agnostic cognitive infrastructure.