AI Evaluation Dataset Maintenance Cost: Budget per Reliable Test Case

A coding-agent evaluation set is a maintained product, not a one-time spreadsheet. Repositories evolve, dependencies disappear, tests become flaky, task wording leaks answers, and formerly difficult cases become trivial. If maintenance is not budgeted, the reported benchmark slowly stops measuring production value.

The useful unit is cost per reliable test case per evaluation cycle. Include authoring, repository preparation, labels, execution compute, model judging, human adjudication, and periodic refresh. Divide only by cases that still run deterministically, represent current work, and distinguish model quality.

Map the Lifecycle

A case begins with a real task or carefully designed synthetic task. A maintainer freezes a repository state, writes acceptance tests, removes accidental clues, validates licenses and secrets, and records expected behavior. Independent reviewers then solve or inspect the case. After publication, every change in runtime, dependency availability, or product policy can require maintenance.

Separate creation cost from recurring cost. Creation includes task selection and initial validation. Recurring cost includes storage, environment rebuilds, flaky-test triage, label disputes, contamination checks, and updates when production distributions shift. A cheap initial set can become expensive if every run needs manual rescue.

Calculate Cost per Reliable Case

Add monthly maintainer hours at a loaded rate, sandbox and CI charges, judge-model tokens, data services, and review time. Divide by the number of cases that pass a reliability gate. If a 500-case suite costs $12,000 per quarter but only 400 cases are stable and current, the true quarterly maintenance cost is $30 per reliable case, not $24.

Then divide by useful decisions. A suite run every day can be economical even at a high maintenance cost if it prevents expensive model regressions. A suite consulted once a year may not justify elaborate infrastructure. Track cost per release decision or routing change as a second-level metric.

Measure Reliability Explicitly

• Environment reliability: setup succeeds without manual intervention.
• Test reliability: repeated runs give the same pass or fail result.
• Label reliability: reviewers agree on acceptable outcomes.
• Discrimination: the case reveals meaningful differences between systems.
• Relevance: the task still resembles current production work.

Quarantine cases that fail any dimension rather than silently keeping them in aggregate scores. Publish active, quarantined, and retired counts. A smaller stable suite provides more decision value than a large headline number filled with noise.

Use Refresh Tiers

Refresh high-value gating cases on every dependency or policy change. Review representative monitoring cases monthly or quarterly. Keep archival cases for trend analysis but do not let them block releases when their environment is obsolete. Assign an owner and next-review date to every active case.

Sample new tasks from production telemetry with privacy safeguards. Compare language, framework, difficulty, tool use, and failure modes with the existing distribution. Add cases where coverage is weak and retire redundant cases that no longer change decisions. This keeps the suite compact enough to run frequently.

Control Judge and Human Review Cost

Use deterministic tests for compilable behavior, exact outputs, and security invariants. Reserve model judges for qualities that cannot be encoded cheaply, such as maintainability or explanation usefulness. Calibrate judges against blinded human labels and route low-confidence disagreements to adjudication instead of reviewing every case manually.

Cache immutable repository setup and judge prompts where supported. Run a cheap smoke subset on every change and the full suite only for candidates. Adaptive evaluation can save compute, but preserve a fixed core so comparisons remain statistically interpretable.

Plan Capacity and Ownership

Assign a maintainer budget proportional to suite criticality. A release-blocking security set needs an on-call owner and repair target; an exploratory benchmark can tolerate longer quarantine. Publish a maintenance backlog with broken environments, disputed labels, missing coverage, and planned retirements so score consumers understand current limits.

Reserve 10% to 20% of evaluation capacity for refresh rather than spending every cycle on new cases. New volume is attractive, but neglected old cases compound operational debt. Review whether vendors, package registries, and base images used by the suite have stable availability. Mirror necessary artifacts when licensing allows and verify that mirrors do not change task difficulty.

Guard Against Benchmark Leakage

Keep a private holdout for routing and release decisions, rotate a portion regularly, and monitor suspiciously large gains on public cases. Contamination does not merely inflate scores; it causes teams to buy a model whose production advantage may not exist. Treat leakage review as part of maintenance cost.

Bottom Line

Budget evaluations by reliable, current test cases and the decisions they support. Quarantine decay, assign refresh tiers, and automate deterministic checks before paying model judges or reviewers. The cheapest benchmark is not the one with the lowest run cost; it is the one that prevents the most expensive wrong model decision.