Architecting a Self-Improving Software Factory: Multi-Agent Orchestration via Telemetry-Driven Autonomous Development

The paradigm of software engineering is shifting from manual, human-centric development to autonomous, agentic workflows. The goal is no longer just "writing code with AI assistance," but building a closed-loop Software Factory: a system capable of detecting its own failures, formulating tasks, executing changes, and verifying them without human intervention.

This post explores the architecture of a self-improving software factory—a multi-agent system that leverages telemetry, specialized coding skills, and hierarchical verification to maintain a production-grade codebase.

The Feedback Loop: Telemetry as the Trigger

A truly autonomous system cannot rely on manual ticket creation. It requires an automated ingestion layer. In this architecture, we utilize telemetry-driven feedback loops. By integrating monitoring tools like Sentry (for error tracking) and PostHog (for product analytics), the system can detect runtime exceptions or degraded user experiences in real-time.

The component responsible for this is Super Collect. This skill operates on a scheduled cadence, polling telemetry providers to identify regressions or new feature opportunities. Once an anomaly or requirement is identified, it automatically generates a structured GitHub Issue. This transforms raw error logs into actionable developer tasks within the GitHub project board, effectively bridging the gap between production monitoring and development backlog.

The Agentic Framework: Specialist Orchestration

The core of the factory is the Super Board, an agentic framework that manages a pipeline of specialist agents. Unlike generic LLM prompts, this system mimics a professional software engineering team by separating concerns into three distinct roles: Builder, QA (Quality Assurance), and Reviewer.

1. The Builder Phase: Optimization and Context Injection

The "Builder" stage is where the most significant technical complexity resides. To prevent common LLM pitfalls—such as code duplication and excessive token consumption—the builder utilizes a pre-computation layer consisting of two critical skills: Ponytail and Context Seven.

Reducing Complexity with Ponytail

One of the primary costs in agentic development is the "token tax" incurred by writing redundant or bloated code. The Ponytail skill implements a 7-layer ladder designed to minimize complexity before a single line of new code is written:

1. Feature Existence Check: Does this feature already exist in the codebase?
2. Reusability Analysis: Can existing utility functions or components be leveraged?
3. Standard Library Audit: Are there native platform features available?
4. Dependency Check: Is there an existing, installed dependency that solves this?
5. One-Liner Optimization: Can the logic be implemented with minimal footprint?
6. (And subsequent layers of abstraction reduction).

By traversing this ladder, the agent avoids "hallucinating" new functions when a simple import would suffice, significantly reducing both the codebase's technical debt and the token overhead of the LLM.

Context-Aware Development via Context Seven

To ensure accuracy in modern ecosystems (e.g., React, NestJS, Vercel), the agent utilizes Context Seven. This skill performs a deep read of third-party library documentation before the coding phase begins. By injecting up-to far-reaching API specifications into the prompt context, we mitigate the risk of the model using deprecated methods or incorrect implementation patterns.

The actual implementation leverages spec-driven development skills (specifically curated Matt Pocock skills), which provide lightweight, best-practice implementations for feature deployment, bug diagnosis, and codebase refactoring.

2. The QA Phase: Hierarchical Verification

A critical failure point in autonomous systems is the "blind" application of patches. To prevent this, the Super QA skill implements a bottom-up testing hierarchy. Rather than jumping straight to End-to-End (E2E) tests, which are brittle and slow, the system follows a rigorous dependency-aware sequence:

1. Unit Testing Layer: Verifying fundamental logic and pure functions.
2. Integration Testing Layer: Ensuring that service layers and database interactions function correctly.
3. Component Testing Layer: Utilizing tools like React Testing Library to verify component hooks and props.
4. E2E/UI Testing Layer: The final stage, utilizing Playwright or Cypress to simulate user journeys through the UI.

This hierarchical approach ensures that by the time an agent reaches the E2E stage, the underlying architectural primitives are already validated, drastically reducing the "retry" loops in the factory.

3. The Review and Refinement Loop: Eliminating "AI Slop"

The final stage involves a Super Review process, which includes code reviews and codebase design audits. Furthermore, to address the aesthetic and UX degradation often seen in AI-generated interfaces (often termed "AI slop"), we implement UI Refinement Loops.

Using specialized skills like Impeccable, the system enters an iterative loop: Diagnose $\rightarrow$ Critique $\rightarrow$ Audit $\rightarrow$ Fix $\rightarrow$ Harden $\rightarrow$ Clarify $\rightarrow$ Polish.

This loop allows the agent to critique its own UI design, adding motion (animations) and polishing CSS/styling until the interface meets high-fidelity standards.

Guardrails and Production Readiness

An autonomous factory must be self-governing. The architecture includes several critical guardrails:

• Secret Detection: Automated hooks to prevent the accidental commit of API keys or sensitive credentials.
• Automated Evals: Setting up skill evaluations to measure the accuracy of agent outputs over time.
• Documentation Sync: Using Boost to automatically update README.md and other technical documentation as the codebase evolves.

By treating AI agents not just as coders, but as specialized components within a structured CI/CD-like pipeline, we can build software that doesn't just run—it improves.