Prompt Cost Regression Testing: Stop AI Coding Changes from Quietly Doubling Token Spend

A two-paragraph system-prompt change looks harmless in code review. Multiplied across millions of requests, it can become a permanent input-token tax. Prompt cost regression testing treats instructions, tool schemas, context selection, and agent loops like performance-sensitive production code.

The test does not need to predict an invoice exactly. It needs to detect material changes before deployment: a larger static prefix, more tool definitions, longer outputs, extra turns, reduced cache reuse, or lower completion that triggers retries. Pair the cost check with a quality threshold so developers cannot win by deleting necessary context.

Create a Stable Cost Fixture Set

Select representative conversations for common workflows: explain code, fix a small bug, generate tests, refactor a file, use two tools, and recover from a failing command. Store the inputs, repository snapshot, tool responses, model snapshot, and acceptance rubric. Remove secrets and unstable external dependencies.

Include short and long sessions. Static prompt growth has the largest percentage effect on short tasks, while loop behavior dominates long tasks. Tag fixtures by risk and production frequency so the final result can be weighted using actual traffic.

Measure More Than Total Tokens

• Uncached input, cache writes, cache reads, output, and billed reasoning tokens where reported.
• Number of model turns, tool calls, failed tool arguments, and repair attempts.
• Wall-clock time and external execution cost such as CI minutes or sandboxes.
• Task success, test result, reviewer edits, and policy compliance.
• Estimated cost under the exact provider rate card used in production.

Keep raw token counts separate from prices. A provider can change rates without a code change; storing both lets you reprice old runs. It also shows whether a regression came from more usage or a pricing update.

Use Deterministic Checks Before Live Runs

CI can count tokens for system instructions, tool schemas, and static context without calling a model. Fail a pull request when the prefix grows beyond an explicit budget, or require an explanation for the increase. Snapshot normalized tool definitions so an accidental verbose description or duplicate schema is obvious in the diff.

These checks are fast and stable, but they cannot see behavioral changes. A clearer prompt may be 200 tokens longer and save two repair turns. That is why a scheduled or pre-release live evaluation must measure the complete agent loop.

Control Randomness in Behavioral Tests

Pin model versions where possible, use consistent effort settings, and run multiple repetitions for stochastic tasks. Compare medians and distributions rather than a single run. Record provider request IDs and timestamps because service updates, rate limits, and cache state can change results outside your code.

A useful gate might block a release when weighted median cost rises more than 15% without a quality gain, or when cost per accepted task rises more than 10%. Exact thresholds depend on traffic and margins. Allow an explicit override with a written reason, owner, and expiry rather than encouraging developers to weaken the test.

Catch the Common Regressions

Watch for chat history appended twice, full files included after a targeted retrieval, tool results echoed in both messages and state, schema descriptions repeated across agents, and retry prompts that resend unchanged context. Another frequent problem is breaking cache affinity by inserting timestamps or request-specific identifiers near the beginning of a stable prompt.

Output regressions matter too. A change from patch-only answers to explanations plus complete files can multiply output cost. Enforce structured response limits and verify that downstream code does not ask the model to repeat information already available through tools.

Review cache behavior explicitly. A prefix can keep the same token count while becoming more expensive because a variable build ID moved to the top and destroyed reuse. Run cold and warm cases, record cache writes and reads, and preserve the exact normalized prefix hash. This separates a genuine model-behavior regression from a routing or prompt-layout regression.

Make the report easy to review: show the old and new weighted cost, quality, turns, and the three fixtures with the largest changes. A compact diff helps maintainers catch intent, while raw traces remain available for investigation.

Close the Loop with Production Data

Fixtures drift. Compare their weighted cost with production telemetry each month, add new task patterns, and retire obsolete ones without rewriting history. Alert on real cost per accepted task, then reproduce unexpected shifts in the fixture suite. CI prevents known regressions; production monitoring finds unknown ones.

Prompt cost tests make optimization durable. Without them, a one-time token reduction slowly disappears as features accumulate. Estimate fixture prices with our AI Cost Calculator, commit the baseline, and require every material increase to buy a measured improvement.