What Is KV Cache and Why It Determines Your LLM Bill

Every time you send a long prompt to an AI coding assistant, there's a hidden mechanism that determines most of your bill: the KV cache. Understanding it won't just make you a better engineer — it'll directly explain why your 100K-token context call costs far more than 10x a 10K-token call, and what you can do about it.

What Is KV Cache? The 60-Second Version

When a transformer model generates text, it processes your input token by token. For each token, every attention layer computes a Key vector and a Value vector. These K and V vectors are stored so the model doesn't have to recompute them when generating subsequent tokens. That storage is the KV cache.

Think of it like this: if you're writing a long document, the KV cache is the model's "working memory" of everything it has already read. Without it, the model would need to re-read your entire prompt from scratch for every single output token — making generation impossibly slow.

The problem? This working memory grows linearly with sequence length. Every additional token in your context adds another set of K and V vectors across every attention head in every layer. For a model like Claude Sonnet 4 or GPT-4.1 with 80+ layers and 64+ attention heads, that's enormous.

Why KV Cache Makes Long Contexts Expensive

Here's the math that explains your bill. For a standard transformer:

• KV cache memory per token = 2 (K and V) x num_layers x num_heads x head_dim x bytes_per_param
• For a 70B-class model (80 layers, 64 heads, 128 dim, FP16): ~2.6 MB per token in the sequence
• At 10K tokens: ~26 GB of KV cache memory
• At 100K tokens: ~260 GB of KV cache memory

That 260 GB doesn't fit on a single GPU. Providers must shard it across multiple GPUs, coordinate memory, and often can't batch your request with others. This is why long-context requests are disproportionately expensive to serve — and why providers charge premium prices for them.

The cost impact isn't just linear — it's super-linear in practice. A 100K context request often costs 12-15x a 10K request (not 10x) because of reduced batching efficiency, memory pressure, and slower generation speed tying up hardware longer.

How Models Reduce KV Cache Costs: GQA, MLA, and Sliding Window

Model designers have developed several techniques to shrink KV cache requirements. Understanding these helps you pick models that are cheaper to run at long contexts:

Grouped Query Attention (GQA) — used by Llama 4, Qwen 3, and Gemini models — shares Key and Value heads across multiple Query heads. Instead of 64 separate KV heads, a model might use 8 KV groups shared across 64 query heads. This cuts KV cache by 8x while barely affecting quality. It's why Llama 4 Maverick can handle 1M context tokens on fewer GPUs than older architectures.

Multi-head Latent Attention (MLA) — DeepSeek's innovation used in V3 and R1 — compresses K and V into a low-rank latent space before caching. Instead of storing full K and V vectors, it stores a compressed representation and decompresses on the fly during generation. This achieves 5-10x KV cache reduction with minimal quality loss and is a key reason DeepSeek offers competitive pricing at long contexts.

Sliding Window Attention — used in Mistral models and hybrid architectures — limits how far back each layer "looks." Instead of caching the entire sequence, only the last N tokens (typically 4K-32K) are cached per layer. Some layers use full attention while others use windowed attention. This caps KV cache growth but sacrifices some long-range recall ability.

The Pricing Impact: What This Means for Your Bill

API providers price tokens to reflect their serving costs. KV cache is the dominant factor in those costs for long contexts. Here's how it shows up in pricing:

• Input tokens cost less than output tokens because input KV cache is computed in one parallel pass, while output tokens are generated sequentially (each needing the full cache read).
• Cached/prompt-cached tokens cost 50-90% less because the provider has already computed and stored the KV cache — they just reload it instead of recomputing.
• Long-context surcharges (implicit or explicit) reflect the extra GPU memory and reduced batching efficiency of large KV caches.

This is why prompt caching is the single biggest cost optimization available. When you reuse the same system prompt or file context across multiple requests, the provider stores that KV cache and charges you the discounted cached-token rate instead of recomputing everything.

Practical Tips: Minimize KV Cache Costs in Coding Workflows

Armed with this understanding, here's how to reduce what you pay:

1. Keep context focused. Don't dump entire codebases into context. Every unused file you include inflates KV cache and your bill. Include only the files the model actually needs for the current task.
2. Leverage prompt caching. Structure your prompts so the static portion (system prompt, reference docs, boilerplate context) comes first. Providers cache the KV state for this prefix, so subsequent requests using the same prefix get the 50-90% cached discount.
3. Break long conversations. After 20-30 messages, start a fresh conversation. The accumulated KV cache from the full conversation history is resent with every new message — and it's growing exponentially in cost.
4. Choose models with efficient KV architectures. For long-context tasks (analyzing large codebases), prefer models using GQA or MLA over standard multi-head attention. They're cheaper to serve at length and providers pass some savings through.
5. Use summarization for old context. Instead of keeping 50K tokens of conversation history, have the model summarize earlier work into 2-3K tokens. You lose some detail but cut KV cache by 90%.

Estimating Your KV Cache Costs

To understand how KV cache impacts your specific workflow, track your average context length per request. If you're typically sending 30K-50K tokens of context for coding tasks, you're paying 3-5x what someone sending focused 10K-token requests pays — even if you generate the same amount of output.

Use our AI cost calculator to model the cost difference between your current context sizes and optimized shorter contexts. The gap is usually larger than people expect — cutting average context from 50K to 15K tokens can reduce your monthly bill by 60-70%.