Cost per Passing Test: A Better KPI for AI-Generated Test Suites Than Token Spend

An agent can generate 100 tests for a few dollars and still create negative value. Some will not compile, some repeat existing coverage, and some pass without asserting meaningful behavior. Cost per accepted passing test connects AI spend to the test assets your team keeps.

The metric is simple: add model usage, tool infrastructure, repair attempts, CI execution, and reviewer time, then divide by tests that pass, meet the rubric, and are merged. The strict acceptance condition prevents a generator from winning by producing a large number of trivial assertions. It also makes different models and prompting strategies comparable on the same repository.

Define an Accepted Passing Test

A test must compile, pass in a clean environment, and fail when the intended behavior is deliberately broken. That mutation check proves the assertion can detect something. It should avoid network flakiness, uncontrolled time, random order dependence, and duplicate coverage. Finally, a reviewer should judge that the test names behavior clearly and belongs at the chosen test layer.

Your rubric may be lighter for low-risk utility code and stricter for payments or authentication. Write it before generation. If acceptance changes after seeing output, cost comparisons become subjective and easy to manipulate.

Build the Numerator Honestly

• Model spend: all input, output, cached, and reasoning usage across initial generation and repairs.
• Execution: sandbox, CI minutes, database fixtures, browsers, and external test services.
• Human review: prompt preparation, failure triage, patch review, and manual edits.
• Rework: retries, reverted tests, flaky-test investigation, and maintenance during the observation window.

Suppose an agent run costs $8 in model usage, $4 in CI, and 45 minutes of review valued at $60 per hour. Total cost is $57. If it proposes 40 tests but only 30 pass and 20 satisfy the full acceptance rubric, cost is $2.85 per accepted passing test. Dividing by all 40 proposals would claim $1.43 and hide half the quality problem.

Pair Cost with Coverage Value

Passing tests are not identical. A useful companion metric is cost per mutation killed, per previously uncovered branch, or per defect found. Mutation testing is especially helpful because it changes code and checks whether the suite detects the change. A test that passes but never fails under relevant mutations may add maintenance without protection.

Do not pay agents to chase a coverage percentage blindly. The final points of line coverage often target generated, unreachable, or defensive code and can cost far more than their risk value. Segment results by component criticality so a payment invariant and a formatting helper are not treated as equivalent units.

Compare Strategies, Not Only Models

Test one-shot generation against an iterative loop, cheap-model generation plus premium review, and human-written specifications followed by agent implementation. Keep the task set and acceptance rubric constant. A stronger model may cost more initially but need fewer repair cycles and less review, lowering the complete numerator.

Also compare context approaches. Supplying the implementation, existing nearby tests, fixtures, and test conventions can raise input cost while reducing invalid output. The metric reveals whether the extra context pays for itself through higher acceptance.

Measure Maintenance After Merge

A generated test can look cheap on day one and become expensive after five flaky failures. Track a 30- or 90-day adjusted cost that adds investigation and update time. Flag tests disabled shortly after merge. If one strategy produces more brittle mocks or over-specified snapshots, its adjusted cost per retained test will rise.

Use cohorts so the maintenance window is fair. Do not compare a new batch with six months of history against another batch observed for one week. Report initial and adjusted metrics separately.

Prevent gaming by reporting the attempted-test count and rejection reasons beside the KPI. A workflow could lower cost per accepted test by proposing only obvious happy paths while leaving risky branches untouched. Track required behaviors from the test plan, and show which remain uncovered. Efficiency is meaningful only after the suite meets the agreed scope.

Use the KPI as a Guardrail

Set a target by test type and risk, then cap retries when projected cost exceeds the threshold. Escalate difficult cases to a human or stronger model rather than allowing an autonomous loop to keep spending. Review a sample of accepted tests even when automation passes, because a flawed rubric can approve cheap nonsense at scale.

Cost per accepted passing test does not replace defect metrics, but it gives engineering teams a concrete production unit for AI spend. Estimate candidate model charges with our AI Cost Calculator, then include CI and reviewer time before selecting a test-generation workflow.