Engineering Robustness in Agentic Workflows: A Multi-Layered Architecture for Evidence-Based Skill Refinement

In the current landscape of generative AI, there is a pervasive and often misleading narrative that autonomous agents are on the verge of self-rewriting their own source code to achieve perfect alignment. The promise is seductive: an AI that reads your Slack, adopts your persona, and optimizes its own procedures while you focus on higher-level strategy. However, for engineers building production-grade systems, "self-improving" without a rigorous architectural framework is a recipe for catastrophic failure.

The challenge isn't just making an AI learn; it is ensuring that the learning process—what we term Skill Refinement—is controlled, verifiable, and does not increase the system's "blast radius." This post explores a structured approach to building a skill refinement pipeline that transforms real-world feedback into reusable AI behavior without compromising system integrity.

Defining the "Skill" and the Lifecycle of Improvement

In this architecture, a skill is defined as a deterministic procedure—a workflow or Standard Operating Procedure (SOP)—that an agent follows repeatedly. The goal of skill refinement is to take rejected outputs, edge cases, or new environmental data and codify them into the skill's logic.

It is critical to distinguish between Evaluations (Evals) and Refinement.

• Evals are a grading mechanism. They measure performance against static, known benchmarks to determine if a skill meets a predefined "Definition of Done."
• Refinement is the process of updating the skill's instructions based on observed usage and real-world data.

A robust system follows a strict lifecycle:

1. Initial Construction: Building the skill with a clear, AI-verifiable "Definition of Done."
2. Evaluation: Running initial evals against static datasets to ensure baseline competency.
3. Deployment/Usage: Implementing the skill in live environments (e.g., writing LinkedIn content or processing client data).
4. Feedback Collection: Gathering evidence from rejected drafts, call transcripts (via tools like Fathom), or user corrections.
5. Refinement: Processing that evidence to propose updates.
6. Re-Evaluation: Running evals against the new version of the skill to ensure the refinement hasn't introduced regressions.

The Three-Layer Architecture

To manage this lifecycle, we implement a three-layer system designed to decouple data ingestion from logic execution and orchestration.

1. Signal Capture

The first layer is responsible for ingesting raw telemetry from external ecosystems (Slack, Notion, Fathom, etc.). This layer populates an Evidence Inbox. It doesn't just collect text; it collects "signals"—structured observations that indicate a deviation from the expected behavior or a change in environmental context.

2. The Refinement Engine

This is the processing core. Once signals are captured, the engine analyzes them to decide if they warrant an update to a skill.md file, a context file (e. Far as client-specific data), or long-term memory components. This layer prevents "agent sprawl"—the tendency for autonomous agents to create unnecessary and unmanaged sub-skills that clutter the system architecture.

3. Cadence and Orchestration

The final layer manages the execution frequency. While some updates can be triggered via webhooks (e.g., at the end of a VS Code session or Claude Code interaction), most production systems benefit from a scheduled cadence—daily or weekly reviews. This allows for batch processing of evidence, reducing API overhead and providing a window for human oversight.

The Pipeline: From Raw Signal to Proposed Diff

The technical implementation of this pipeline can be broken down into four distinct stages:

Stage 1: Signal Capture & Evidence Ingestion

Raw data enters an intake folder. This might include a rejected LinkedIn draft or a Fathom transcript from a discovery call with a client like "Acme Robotics." The system processes these raw events and transforms them into Evidence Cards. An evidence card contains the source, the signal type (e.g., changed fact), a summary of the observation, and the direct verbatim evidence required for verification.

Stage 2: The Evidence Router

Not all information belongs in a skill update. The Evidence Router evaluates each piece of evidence to determine its destination. It uses an intelligent routing logic to decide if an update should target:

• skill.md: For behavioral changes (e.g., "Avoid using 'let that sink in'").
• Context Files: For environmental changes (e.g., "Acme Robotics now requires SOC 2 compliance").
• Memories/References: For persistent factual updates.

Stage 3: Skill Self-Update and The Judge

This stage proposes the actual modification to the markdown files. To prevent "hallucinated" improvements, we implement an AI Gate known as The Judge. The Judge evaluates the proposed change against the skill's original "Definition of Done." It calculates a confidence score and generates a diff (the delta between the current state and the proposed state).

Stage 4: Human-in-the-Loop (The Proposal Gate)

Crucially, we do not allow the system to commit changes directly to production files. Instead, all updates are routed to a proposals folder. This mimics a Git Pull Request workflow. A human operator reviews the proposed diffs in an environment like VS Code or via a GitHub PR.

Risk Mitigation: The Three M's Framework

When deciding whether to allow auto-refinement or require manual intervention, engineers should evaluate the Blast Radius using the "Three M's" framework:

1. Megaphone: How does this change impact your audience? (High risk if it changes public-facing content).
2. Money: Does this change affect financial logic, procurement rules, or pricing structures? (Critical risk).
3. Meaning/Mission: Does this change alter the fundamental identity or direction of the system (e.g., updating an ICP—Ideal Customer Profile)?

If a proposed change scores high on any of these metrics, manual approval is non-negotiable. An error in an ICP context file can cascade through every downstream skill that relies on that data, leading to systemic failure.

Conclusion

True autonomous improvement isn't about removing the human; it’s about augmenting the human with a structured pipeline for learning. By treating AI skills as versioned, verifiable code and implementing a robust routing and judging architecture, we can build systems that evolve with our business without losing control of their fundamental logic.