Architecting an Autonomous Research Engine: Automating the Karpathy LLM Wiki Workflow via Claude Code and MCP

In the evolving landscape of Large Language Model (LLM) utilization, the transition from manual prompting to autonomous, scheduled workflows represents the next frontier of productivity. Inspired by Andrej Karpathy’s "LLM Wiki" concept—which emphasizes the structured conversion of raw data into a curated, searchable knowledge base—this post explores the implementation of a fully automated research engine. By leveraging Claude Code, the Model Context Protocol (MCP), and a custom-built wiki skill, we can transform fragmented data sources (YouTube, Gmail, Slack) into a structured "Second Brain" within Obsidian.

The Core Paradigm: Raw vs. Curated Data

The fundamental architecture of this system rests on a dual-directory paradigm: the raw/ folder and the wiki/ folder.

1. The raw/ Directory: This serves as the landing zone for unprocessed, high-entropy data. It contains the direct outputs from data-fetching agents—such as YouTube metadata, transcripts, or email threads—before any LLM-driven synthesis has occurred.
2. The wiki/ Directory: This is the curated, low-entropy output. It contains the results of the "ingest" phase, where an LLM has processed the raw/ data to produce summaries, trend analyses, competitor deep-dives, and synthesized reports.

By maintaining this separation, we preserve the integrity of the original data while providing a highly structured, navigable interface for research.

The wiki Skill: Orchestrating Agentic Workflows

The centerpiece of this implementation is a custom-developed wiki skill. This skill acts as a wrapper for the entire lifecycle of the research engine, handling everything from project scaffolding to agentic dispatching.

Phase 1: Scaffolding and Contextual Interviewing

When initializing a new research project via the slash wiki command in Claude Code, the skill executes a multi-phase setup. Rather than requiring a complex configuration file, the skill utilizes an interview-based initialization. The agent queries the user to define:

• Topic/N/Niche: The specific domain of research.
• Scope/Directories: The organizational structure (e.g., competitors/, trending/, my_channel/).
• Data Sources: The specific endpoints to be monitored.
• Page Types: The specific templates for the wiki (e.g., "Video Overviews," "Weekly Outlier Digests").

This process automatically generates a standardized file structure, including .env for environment variables, .mcp configurations for tool integration, and a cloud.md file. The cloud.md file is critical; it serves as the system prompt that defines the operational boundaries and instructions for the Claude Code agent during every new context window.

Phase 2: Data Ingestion via MCP and Parallel Sub-Agents

The most technically complex aspect of the workflow is the "farming" phase. Using the Model Context Protocol (MCP), specifically the vidIQ MCP server, the system can interface directly with YouTube's API to extract deep metrics.

To handle large-scale data extraction efficiently, the wiki skill implements a dispatcher pattern. For instance, when tasked with monitoring 57 different YouTube channels, the system does not process them linearly. Instead, it calculates the workload and dispatches the task into parallel sub-agents. In our implementation, the 57 channels are split into four distinct batches, each handled by a parallel sub-agent. This significantly reduces the wall-clock time required for the wiki farm command to complete.

The workflow follows a strict execution chain:

1. wiki farm [source]: Triggers sub-agents to fetch raw metadata and populate the raw/ folder.
2. wiki ingest: An LLM-driven process that reads the raw/ files, performs summarization, and writes the processed information into the wiki/ folder.
3. wiki health check: A maintenance routine to identify broken links, stale notes, or redundant data.

Automation and Scheduled Routines

A research engine is only as useful as its consistency. To move beyond manual triggers, we utilize Claude Code routines. By prompting the agent to "run this on a schedule," the system creates a cron-like routine within the Claude environment.

For example, a routine can be configured to trigger every Monday at 2:00 AM. This routine automatically invokes the wiki farm and wiki ingest pipelines, ensuring that by the time the researcher starts their week, the wiki/ folder is already populated with the latest trending topics, competitor updates, and performance outliers.

The Final Output: Obsidian as the Visualization Layer

The ultimate destination for this processed intelligence is Obsidian. Because the wiki/ folder is structured as a standard Markdown repository, it integrates natively with Obsidian’s powerful features.

The use of a structured index.md and organized subfolders allows for the generation of a Knowledge Graph. Researchers can visualize the relationships between different competitors, trending topics, and video formats through Obsidian's Graph View. This transforms a simple collection of notes into a dynamic, interconnected "Second Brain," where the "Outlier Digest" can be cross-referenced with "Competitor Deep-Dives" to identify emerging market shifts.

Conclusion

By moving away from manual data collection and toward an agentic, MCP-driven architecture, we can build a research engine that operates autonomously. The combination of Claude Code's orchestration, the scalability of parallel sub-agents, and the structured permanence of Obsidian creates a powerful, self-updating knowledge ecosystem.