Does Reranking Save or Cost Money? Token Math for RAG Rerankers

In a RAG-based coding agent, retrieval pulls candidate code chunks out of your repository and stuffs them into the prompt. A reranker sits between retrieval and generation, re-scoring those candidates so only the most relevant ones reach your expensive model. The question every team eventually asks: does that extra step save money, or is it just another line item on the bill? The honest answer is that it depends on your generation model's price and your query volume, and the token math makes the break-even point surprisingly clear.

What a Reranker Actually Does

A typical RAG pipeline runs in two stages. First, a cheap embedding search scans your indexed repo and returns the top candidates, say the 50 chunks whose vectors sit closest to the query. Embeddings are inexpensive: you pay a tiny amount to embed the query, and the vector search itself is nearly free compute. Second, without a reranker, the agent naively stuffs the top 15 to 20 of those chunks into the generation prompt and hopes the answer is somewhere in there.

A reranker changes that second stage. It re-scores all 50 candidates with a model tuned for relevance, then keeps only the best 5 to 10 to send onward. Fewer chunks, higher signal. The reranker can be a dedicated cross-encoder reranker API or a simple cheap-model scoring pass. Either way it adds a small, dedicated cost that the naive pipeline does not pay.

The Cost Trade-Off in One Sentence

Reranking adds a small fixed cost per query but can sharply cut the number of chunks you feed the expensive generation model, and input tokens on a premium model are exactly where RAG bills balloon. Whether you come out ahead depends almost entirely on how expensive that generation model is.

Worked Example: Reranking Into Claude Opus 4.8

Suppose your embedding search returns 50 candidate chunks averaging about 750 tokens each. Without reranking, you stuff the top 20 chunks, roughly 15,000 tokens, into Claude Opus 4.8 (input $5 per million tokens) on every query. With reranking, you send only the best 5 chunks, about 4,000 tokens.

The gross input savings are $0.055 per query. Now subtract the reranker's own cost. Model it as a cheap scoring pass with DeepSeek V4 Flash (input $0.09/M, output $0.18/M) that reads all 50 candidates, about 37,500 tokens, and emits a short ranked list of roughly 250 tokens. That works out to about (37,500 x $0.09/1M) + (250 x $0.18/1M), or roughly $0.003 per query.

Net savings: $0.055 minus $0.003, about $0.052 per query. At 1,000 queries a day that is roughly $52 a day, or about $1,500 a month, saved simply by sending five tight chunks instead of twenty loose ones. The reranker overhead is a rounding error against what you save on the premium model's input.

The Break-Even Rule

Reranking pays off when the input tokens you save on the generation model are worth more than the reranker costs to run. Two variables dominate: the generation model's input price and your query volume. Because the reranker overhead is roughly fixed per query, the pricier your generation model, the faster it pays for itself, and the more queries you run, the more that per-query win compounds into real monthly money.

When Not to Bother: Cheap Generation Models

Flip the generation model to a budget option and the math inverts. Run the same query on DeepSeek V3.2 (input $0.229/M): the 20-chunk path costs about $0.0034 and the 5-chunk path about $0.0009, so the gross savings are only around $0.0025 per query. But the reranker scoring pass still costs about $0.003 per query, more than you saved. Net result: you lose roughly $0.001 on every query. Qwen3 Coder (input $0.22/M) lands in the same spot.

The lesson is blunt. When your generation model is nearly as cheap as the reranker itself, the input savings are too small to cover the overhead, and reranking becomes a cost, not a saving. Budget models already make context cheap, so trimming context buys you almost nothing.

Quality Is a Bonus, Not the Point

Cost aside, feeding five high-signal chunks instead of twenty noisy ones often improves answers: less distraction, fewer hallucinations, and fewer expensive retries when the model cannot find the answer buried in irrelevant context. Those second-order savings can tip a borderline case toward reranking even on a mid-priced model, but they are hard to predict, so treat them as upside rather than the core justification.

How to Decide for Your Stack

Start from your generation model. If you generate with a premium model such as Opus 4.8, GPT-5.6 Sol, or Fable 5 and run meaningful query volume, add a reranker and expect real savings. If you generate with a budget model like DeepSeek V3.2 or Qwen3 Coder, skip reranking or use the cheapest scoring pass you can find. Before you wire up an extra pipeline stage, price both paths for your own model in the AI cost calculator so the break-even is a number, not a guess.