How Many Eval Tasks Do You Need? Budgeting AI Coding Tests with Confidence Intervals

Choosing a coding model from 10 tasks feels empirical, but a result of 7 wins versus 5 wins can be mostly noise. A useful evaluation budget must pay for enough representative tasks and repeated runs to distinguish a real cost-quality advantage from luck.

The goal is not statistical perfection. It is a decision with bounded uncertainty. You want to know whether a cheaper model clears an acceptable success threshold, whether a premium model earns its extra cost, and how much confidence the sample supports. That requires planning the sample before seeing the results, because adding tasks only when your preferred model is losing creates bias.

Define One Primary Outcome

Start with a binary outcome that matters: the patch passes tests and review, or it does not. Record secondary measures such as tokens, wall time, retries, and reviewer minutes, but avoid changing the winner after results arrive. A model that writes elegant prose but fails the repository checks is not a successful coding run.

Write the acceptance rubric so two reviewers would reach the same conclusion. Include allowed tools, maximum attempts, time and token limits, and whether a small human edit still counts. Freeze the task prompt and starting commit. Without those controls, the evaluation measures changing conditions rather than models.

Why Small Samples Mislead

If a model succeeds on 7 of 10 tasks, the observed rate is 70%, but uncertainty is wide. One extra success changes the headline by 10 percentage points. With 70 successes in 100 tasks, one result moves it by only one point. Larger samples narrow the range around the observed rate and make comparisons more stable.

A practical rule is to begin with at least 30 diverse tasks for screening and move toward 100 or more for an expensive default-model decision. This is guidance, not a magic statistical threshold. Rare task classes need their own representation. If only two of 100 tasks involve database migrations, the overall sample tells you little about migration performance.

Estimate the Eval Bill Before Running

For each model, estimate input tokens, output tokens, average attempts, and tool infrastructure. Multiply by the number of tasks and repetitions. Then add reviewer time. If one run averages $0.40 of API spend and eight minutes of review valued at $1 per minute, the human component is twice the model cost. An evaluation of 100 tasks across three models would cost about $120 in API usage and $2,400 in review under those assumptions.

Reduce review cost with deterministic checks first: compilation, tests, linting, and patch constraints. Humans should inspect runs that pass machines or fail ambiguously. Do not remove human review entirely, because agents can satisfy tests with brittle behavior or inappropriate changes.

Use Paired Tasks and Repeated Runs

Run every candidate on the same tasks. This paired design controls for task difficulty better than giving each model a different random set. Randomize execution order when rate limits or repository caches may matter. If models are stochastic, repeat a subset two or three times to estimate run-to-run variance. A model that succeeds once and fails twice is operationally different from one that is consistently adequate.

Keep task categories visible: bug repair, feature work, tests, refactors, documentation, and tool use. Report both the overall result and category results. Weight categories using your production workload rather than equal counts if the final decision is a default routing policy.

Set Stop Rules in Advance

• Stop a candidate early if it cannot meet a predeclared minimum quality threshold even under an optimistic remaining result.
• Stop expanding when the confidence range is narrow enough that every plausible value leads to the same purchasing decision.
• Continue when two models remain close and the annual spend difference makes the uncertainty economically material.
• Rerun after a major model, prompt, tool, or repository change; an old confidence interval does not cover a new system.

Choose on Expected Cost per Success

Divide mean run cost by the accepted-success rate as a first approximation, then add review and retry effects. Show uncertainty around success rather than one decimal that implies false precision. If a cheap model's plausible cost-per-success range overlaps a premium model's range, route low-risk tasks to the cheap tier and gather more production evidence instead of declaring a universal winner.

A good evaluation is an investment decision, not a leaderboard. Size the sample to the money at risk, preserve representative tasks, and stop when additional certainty would not change the choice. Estimate the token portion for each candidate with our AI Cost Calculator, then add validation and review to the eval budget.