AI Observability Stack Cost: OpenTelemetry + Grafana + Custom Traces for Coding Agent Fleets

Why AI Observability Is Different

Traditional application observability tracks requests, latency, and error rates. For AI coding agents, you need to track those plus:

• Token consumption per request, session, user, and workflow.
• Cache hit rates on prompt caching.
• Model routing decisions (which model handled which request and why).
• Tool call sequences within an agent trajectory.
• Retry patterns and their cost.
• Cost attribution to business unit or feature.

Without these, you cannot answer basic operational questions like "why did our AI bill spike last Tuesday" or "which repos consume the most tokens." This post prices out the observability stack that gets you those answers.

The Three Stack Layers

Every AI observability stack has three layers, whether you buy them separately or as an integrated product:

1. Instrumentation. The code in your agent that emits spans, metrics, and events. OpenTelemetry is the standard.
2. Transport and storage. Where the telemetry data goes: Grafana Cloud, Datadog, Honeycomb, New Relic, or self-hosted.
3. Query and visualization. Dashboards, alerts, and ad-hoc queries. Grafana, Datadog, or custom.

Instrumentation Cost

Instrumentation itself is nearly free — OpenTelemetry SDKs are open source. The real cost is engineer time:

• Basic instrumentation: Wrap Anthropic/OpenAI SDK calls to emit spans with token counts, model name, and latency. About 2–4 engineer-hours.
• Agent trajectory tracing: Emit spans for each tool call and thought step, with parent-child relationships. 8–16 engineer-hours.
• Cost attribution: Add user, session, workflow, and cost-center attributes to every span. 4–8 engineer-hours.
• Ongoing maintenance: Updates as new agent features ship. 2–4 hours/month.

One-time cost: about $3,000–$6,000 in engineer time. Ongoing: about $400/month.

Transport and Storage Cost by Provider

Cost varies dramatically by ingest volume. Assume a 20-developer team generating 100 traces per developer per day, each with 30 spans averaging 400 bytes: about 24M spans/month or roughly 10 GB/month of trace data plus metrics and logs.

The Custom Traces You Actually Need

Off-the-shelf observability tools capture what any HTTP client would produce. For AI coding agents you need custom span attributes:

• llm.model — which model handled the request.
• llm.tokens.input, llm.tokens.output, llm.tokens.cached_read — token accounting.
• llm.cost_usd — computed cost of this specific call.
• agent.trajectory.id — group all spans for one agent task.
• agent.tool.name, agent.tool.arguments_size — which tools ran and with what.
• agent.retry.count — how many times we retried this task.
• business.workflow, business.cost_center — for chargeback.

Standard OTel semantic conventions cover some of this in the gen_ai.* namespace. Custom attributes fill the rest.

Full Stack Cost for a 20-Developer Team

Grafana Cloud total: about $550/month. Self-hosted total: about $800/month when engineer time is honestly counted. The apparent savings of self-hosting evaporate on smaller teams.

When Self-Hosted Wins

The math flips in favor of self-hosting only at larger scale:

• Above 200 developers: Ingest volumes push Datadog and Grafana Cloud into the $2K+/month range. Tempo on Kubernetes handles the same volume for $300/month plus fixed engineer time.
• Compliance requirements: HIPAA, FedRAMP, or SOC2 workloads sometimes mandate specific data residency that vendor tools do not support cleanly.
• Team already runs Prometheus and Grafana: Adding Tempo alongside is trivial, so the marginal ops cost is near zero.

The One Dashboard You Actually Need

Do not build ten dashboards. Build one: Cost per successful task, sliced by model and workflow. That single view answers 80% of the operational questions. Additional dashboards are worth building only when a specific question comes up repeatedly.

Alert on three things:

1. Any workflow's daily cost exceeds 2x its 7-day average.
2. Prompt cache hit rate on any workflow drops below 40%.
3. Retry count on any workflow exceeds 3x its 7-day average.

Recommendation

• For teams under 50 developers, use Grafana Cloud. Best cost, minimal setup, complete stack.
• For teams above 200 developers or with compliance mandates, use self-hosted Tempo + Prometheus + Grafana. Costs pass through Grafana Cloud around 100 developers.
• Do not use Datadog unless you already do. It is the best UX but 3x the cost — for AI observability specifically, that premium is hard to justify.
• Custom span attributes are essential. Standard OTel gen_ai semantic conventions cover the basics but miss cost attribution and business-unit fields.
• Build one great dashboard before ten mediocre ones. Cost per successful task is the answer.