Architecting a Multi-Modal Generative Pipeline: Orchestrating Whisper, Gemini, and Google Veo via OpenRouter

In the evolving landscape of agentic workflows, the ability to orchestrate disparate generative models into a unified, autonomous pipeline is becoming a critical skill. This post explores the implementation of a Python-based multi-modal agent capable of transforming raw voice input into fully synthesized video content—complete with scripted narration, generated imagery, and motion synthesis—using a single API gateway for model orchestration.

The Orchestration Layer: OpenRouter and Dependency Management

A significant challenge in building multi-modal agents is "API key sprawl"—the management of various credentials for OpenAI (text/audio), Google (image/video), and specialized providers (Recraft). To mitigate this, we utilize OpenRouter as a unified inference gateway. This allows us to interact with a diverse array of models—ranging from Whisper Large V3 Turbo to Google Veo—through a single standardized endpoint, simplifying authentication and providing centralized cost observability.

For environment management, the pipeline leverages uv, an extremely fast Python package manager. The dependency stack includes:

• numpy: For high-performance audio buffer manipulation.
• sounddevice & soundfile: For real-time PCM audio capture and WAV serialization.
• python-dotenv: For secure management of the OpenRouter API key.
• requests: To handle the RESTful interactions with the inference endpoints.

Phase 1: Speech-to-Text (STT) via Whisper Large V3 Turbo

The pipeline begins with an audio capture module. Using sounddevice, we implement a context manager that captures audio streams into a NumPy array. The raw PCM data is buffered in chunks and subsequently written to a .wav file.

To convert this unstructured audio into actionable text, the agent performs a request to the OpenAI Whisper Large V1/V3 Turbo model via OpenRouter. The implementation requires encoding the audio file into Base64 format to facilitate transmission within a JSON payload. This step transforms the user's verbal intent into a string that serves as the seed for the subsequent LLM-driven logic.

Phase 2: Structured Script Generation using JSON Schema

The core intelligence of the agent lies in its ability to transform a simple transcription into a structured execution plan. We utilize an LLM (with fallback capabilities, such as transitioning from GPT-4o to Gemini 1.5 Flash) to generate a highly structured response.

To ensure downstream modules can parse the output without error, we implement Structured Output using a strict JSON Schema. The schema defines a video_plan object with three mandatory string properties:

1. script: A 40–60 word voiceover script containing SSML-like audio tags (e.g., <whisper>, <excited>).
2. image_prompt: A high-fidelity prompt optimized for a 16:9 aspect ratio cover image.
3. video_prompt: A motion-centric prompt describing the temporal dynamics of an 8-second clip.

By setting additionalProperties: false and utilizing OpenRouter's provider-specific parameters, we enforce strict adherence to this schema, preventing the "hallucination" of extra text that would break the pipeline.

Phase 3: Image Synthesis and Asset Generation

Once the image_prompt is extracted from the JSON payload, the agent invokes an image generation model (e.g., Gemini or Recraft). The implementation involves a POST request to the /images/models endpoint.

A critical technical detail here is handling the response format; most modern generative APIs return images as Base64-encoded strings within a JSON object. Our pipeline decodes this string and writes it to a .png file, which serves as the "first frame" for the subsequent video synthesis step. This ensures visual consistency between the static cover art and the generated motion.

Phase 4: Asynchronous Video Synthesis via Google Veo

The most computationally intensive stage is the generation of the video clip using Google Veo. Unlike text completion, video generation is an asynchronous process. The API does not return a completed file immediately; instead, it returns a polling_url.

We implement a robust polling architecture:

1. Initial Request: Send the video_prompt and the encoded "first frame" (the image from Phase 3) to the /videos endpoint.
2. Status Polling Loop: The agent enters a loop, periodically querying the polling_url. We monitor the status field for transitions between pending, in_progress, and completed.
3. Exponential Backoff/Sleep: To avoid rate-limiting and unnecessary overhead, we implement a 15-second sleep interval between status checks.
4. Final Retrieval: Once the status reaches completed, the agent performs a final GET request to the provided unsigned URL to download the .mp4 binary content.

Phase 5: Observability and Cost Tracking

A production-grade agent must be economically viable. Because we use OpenRouter, every API response includes metadata regarding token usage and precise USD cost. Our helper.py module implements a global cost_tracker dictionary. By parsing the usage field from every request (STT, LLM, Image, Video), the pipeline generates a real-time "cost receipt" upon completion. This allows developers to identify high-latency or high-cost bottlenecks—such as the significant delta between a sub-cent transcription and a multi-dollar video generation task.

Conclusion

By combining structured JSON outputs, asynchronous polling patterns, and unified API orchestration, we have moved beyond simple prompting into the realm of automated content engineering. This pipeline demonstrates that with the right architectural approach, complex multi-modal workflows can be managed through a single, cohesive Pythonic interface.