Architecting Agentic Workflows on the Edge: Leveraging LiteRT-LM, Gemma 4, and Task-Specific Tiny LLMs

The paradigm of Large Language Model (LLM) deployment is undergoing a fundamental shift. While the industry has focused heavily on massive, cloud-based models, a parallel revolution is occurring at the edge. As demonstrated by the work coming out of Google AI Edge, the frontier of mobile and embedded AI is moving toward two distinct but complementary architectures: System-level Generative AI and In-app Generative AI powered by Tiny LLMs (TLMs).

The Edge Imperative: Latency, Privacy, and Cost

Deploying models on-device is no longer just a technical curiosity; it is a requirement for specific high-utility use cases. The primary drivers are:

• Latency-Critical UX: Real-time applications, such as live voice translation (as seen in Pixel's implementation), cannot tolerate the round-trip time of cloud inference.
• Privacy-Preserving Computation: For messaging and sensitive data processing, keeping the inference loop entirely on-device ensures end-to-end encryption and user trust.
• Offline Availability: Enabling intelligence in disconnected environments (e.g., IoT, remote robotics).
• Operational Cost Reduction: Offloading inference from expensive GPU clusters to the user's local NPU/GPU/CPU significantly reduces the TCO (Total Cost of Ownership) for developers.

Two Paradigms of Edge Intelligence

1. System-Level Gen AI (The OS Approach)

This approach involves integrating larger, foundational models (typically in the 2B to 5B parameter range) directly into the Operating System. Examples include Android’s AI Core and Apple Intelligence. These models are preloaded on premium devices and are customized via prompting or "skills." They provide a centralized API for developers to leverage high-quality, built-in intelligence without bloating individual application binaries.

2. In-App Gen AI (The TLM Approach)

In-app Gen AI utilizes Tiny LLMs (TLMs)—models generally under 500M parameters. Unlike system-level models, these are task-specific and bundled with the application or webpage. While they require more intensive fine-tuning to achieve production-level reliability, they offer unparalleled reach across a wide spectrum of hardware, from premium smartphones to low-power IoT devices.

Deep Dive: Gemma 4 and Effective Parameter Optimization

The recent launch of Gemma 4 introduces a breakthrough in how we manage the memory constraints of mobile devices. The architecture focuses on "effective" parameter counts, specifically the E2B and E4B variants.

The "E" designation refers to the amount of RAM required to keep the model resident. While the model may have a higher total parameter count, the E2B model is optimized to require only ~2B parameters in RAM. This is achieved through sophisticated memory management of the embedding tables. In the LiteRT-LM runtime, while the weights for the transformer layers must stay resident, the per-layer embedding tables are memory-mapped. During the autoregressive loop, the runtime only loads the necessary slices of the embedding table (a few hundred to thousands of bytes) for the current token inference. This allows the OS to efficiently evict older, unused parts of the table, drastically reducing the active memory footprint.

Gemma 4 E2B and E4B are also multimodal, supporting text, image, and audio inputs, and are released under the Apache 2.5 license, facilitating widespread adoption.

The Agentic Skill Pattern: Progressive Disclosure

One of the most significant advancements in edge agentic workflows is the implementation of Agent Skills. Traditionally, providing an agent with tools (like Function Calling) requires including all tool definitions in the system prompt. For small models, this "context blover" leads to high token costs and degraded reasoning performance.

To solve this, we utilize a pattern of Progressive Disclosure. The model is initially provided only with a one-line metadata description of available skills. The architecture follows this structure:

1. skill.md: Contains the metadata and high-level description.
2. scripts/: Contains the executable logic (e.g., JavaScript).
3. assets/: Contains any necessary supporting files.

When the model identifies a relevant skill via its metadata, it triggers a load_skill function call. The runtime then dynamically injects the detailed function definitions and the JavaScript logic into the context window. This keeps the initial prompt condensed, maximizing the "batting average" of the model's reasoning capabilities.

Furthermore, we utilize Constraint Decoding during tool calls. By applying strict, tool-specific constraints during the generation phase, we can significantly increase the reliability of small models (like the 2B class) when they are performing complex JSON or function-calling tasks.

The Tiny Model Workflow: From PyTorch to LiteRT-LM

Deploying models to the edge requires a robust optimization pipeline. The workflow for TLMs typically follows this trajectory:

1. Training/Fine-tuning: Using LiteRT-Torch, developers can leverage PyTorch-native optimizations. For models like Function Gemma (270M), the key is using a larger "teacher" model in the cloud to generate massive amounts of synthetic data, which is then used to instruction-tune the tiny "student" model.
2. Quantization & Optimization: The model is optimized for the target hardware using libraries like XNNPACK (for CPU) and MLDRFT (for GPU).
3. Export: The model is exported into a single LiteRT-LM file (an evolution of the TFLite format) that contains the weights, tokenizer, and the autoregressive loop logic.
4. Deployment: This single artifact is cross-platform, running on Android, iOS, Linux, Windows, and even embedded IoT platforms.

Performance Benchmarks

The efficiency of the LiteRT-LM runtime is evident in real-world metrics:

• High-End Mobile/Desktop: Gemma 4 models can achieve thousands of tokens per second on modern Android GPUs and Apple Silicon.
• Edge/IoT: On a Raspberry Pi, we have demonstrated performance of approximately 133 tokens per second, which is sufficient for real-time image analysis and simple text tasks.
• NPU Acceleration: Using specialized NPU compilation for Qualcomm silicon, we see significant performance leaps, enabling complex Vision-Language Models (VLMs) like Fast VLM (500M) to run with high-speed video input processing.

Conclusion

The future of mobile intelligence lies in the modularity of the edge. By combining the foundational power of system-level models like Gemma 4 with the task-specific precision of fine-tuned Tiny LLMs, developers can build agentic ecosystems that are fast, private, and incredibly extensible. Whether through the "progressive disclosure" of JavaScript-based skills or the deployment of specialized transcription engines like AI Edge Eloquent, the tools to bring high-performance AI to every device are now available.