ai claude-code observability opentelemetry llmops cost-optimization anthropic agentic-workflows telemetry

Auditing Agentic Token Expenditure: Leveraging OpenTelemetry to Uncover Hidden Costs in Claude Code Workflows

5 min read

title: "Auditing Agentic Token Expenditure: Leveraging OpenTelemetry to Uncover Hidden Costs in Claude Code Workflows" date: 2026-10-21 tags: [ai, observability, claude-code, llmops, opentelemetry] description: "An analysis of token-based cost leakage in Claude Code and the implementation of OpenTelemetry-native observability to track tool-call failures and model routing efficiency."

In the era of agentic workflows, the transition from per-token pricing to flat-rate subscriptions—such as the $200/month Claude Code tier—has introduced a dangerous "observability gap." While subscription models provide budget predictability, they obscure the granular unit economics of individual coding sessions. When your IDE's internal usage reporter (/usage) only tracks percentage-based quota consumption rather than monetary value or token throughput, you lose the ability to identify high-cost, zero-output (ZOO) sessions.

This post explores a recent discovery of a $44 "leak" within a single Claude Code session and details how an OpenTelemetry-native approach can provide the necessary telemetry to audit LLM agent performance and cost-efficiency.

The Observability Gap in Subscription-Based Agents

The native /usage command in Claude Code is designed for quota management, not financial auditing. It provides metrics such as:

  • Session Block Totals: Displayed as $0 (due to the subscription model).

  • Usage Percentages: Current session and weekly consumption relative to a fixed limit.

  • Context Drivers: Identifying if usage is driven by sub-agent heavy sessions or high-context windows (e.g., >150,000 tokens).

While these metrics are useful for preventing quota exhaustion, they fail to answer the most critical question in LLM engineering: What was the ROI of this session? A session can consume massive amounts of context and trigger hundreds of tool calls without ever resulting in a merged Pull Request (PR). Without external observability, these "zombie sessions" remain invisible on your balance sheet.

Implementing OpenTelemetry-Native Auditing

To bridge this gap, I implemented Dash Zero, specifically utilizing the Darkplane/AISDLC Insights component. The architecture relies on an OpenTelemetry (OTLP) standard, allowing for a non-intrusive plugin-based approach that reads coding sessions rather than server logs.

Technical Setup and Integration

The implementation involves deploying an OTLP endpoint and configuring Claude Code to emit traces via a specialized plugin. The setup follows a standardized pattern:

  1. Authentication & Endpoint Configuration: Obtain the auth_token and OTLP endpoint from the Dash Zero settings.
  2. Plugin Installation: Install the official telemetry plugin directly into the Claude Code environment:
    /plugin install dash-zero @cloud/dash-plugins-dash-zero
  3. Scope Configuration: It is critical to configure the scope at the user level rather than the project level. Because project scopes are keyed on specific folder paths, using a user-level scope ensures that worktrees and disparate repositories are captured under a unified telemetry stream.
  4. GitHub Integration: By connecting GitHub via OAuth, the system can correlate agentic sessions with downstream Git events (PR creation, merges, or branch deletions).

Because this is OpenTelemetry native, the resulting data is queryable via standard PromQL and SQL, enabling complex analytical queries regarding tool-call success rates and model latency.

Analyzing the $44 Leak: A Case Study in Tool-Call Failure

The primary value of this observability stack was revealed when sorting sessions by estimated cost (calculated by applying public pay-as-you-go API rates to the captured token counts).

A single session was identified as a significant outlier, costing approximately $44 for a duration of four hours. Upon deep inspection of the telemetry data, the following metrics were uncovered:

  • Prompt Count: 10 prompts.

  • Tool Call Volume: 470 total tool calls.

  • Error Rate: 35 failed tool calls (approx. 7.4% failure rate).

  • Outcome: The session worked on a specific branch that was ultimately deleted without ever reaching the PR or merge stage.

This "leak" is often caused by an agent entering a loop of failed tool executions—likely attempting to resolve a dependency or file-system error through repetitive, high-context prompts. Without this level of granularity, the $44 loss would have been attributed to standard subscription usage.

Model Routing and Economic Optimization

The audit also revealed significant opportunities for Model Routing optimization. By analyzing "Cost by Model," I found that Claude 3 Opus accounted for 58% ($124) of my total two-day spend, despite representing only a small fraction of the total prompt count (approx. 27 prompts).

The data showed that many high-cost Opus prompts were being used for trivial tasks—such as simple file renaming—which do not require the reasoning depth of an Opus-class model. This led to a new operational strategy:

  1. Architectural Planning: Route complex, multi-step architectural decisions and initial project scaffolding to Claude 3 Opus.
  2. Implementation & Execution: Default all subsequent implementation tasks, file manipulations, and unit testing to Claude 3 Sonnet to minimize token expenditure while maintaining high throughput.

Measuring Agent Autonomy: Uninterrupted Runtime

Finally, the telemetry provided a new metric for agentic productivity: Uninterrupted Agent Runtime. This measures the duration an agent operates before human intervention (a "human-in-the-loop" interruption) occurs.

The data showed a significant trend toward high autonomy, with the largest segment of sessions exceeding 40 minutes without manual intervention. While high autonomy is generally desirable for delegation, the correlation between long runtimes and high cost/low output (as seen in the $44 session) serves as a critical warning sign for developers to implement stricter timeouts or intermediate checkpoints in their agentic workflows.

Conclusion

As LLM agents move from simple chat interfaces to autonomous coding entities, observability must evolve from monitoring "uptime" to monitoring "utility." By leveraging OpenTelemetry and tools like Dash Zero, engineers can transform the black box of subscription-based AI usage into a transparent, auditable, and optimizable pipeline.