AI-Generated Test Cleanup Cost: Why Test Files Need Their Own Budget

Tests Are Not Free Verification

AI-generated tests are often sold as cheap coverage, but test files concentrate framework APIs, mocks, fixtures, async timing, environment assumptions, and repository conventions. A test that does not compile or asserts the implementation rather than the requirement creates cleanup work without reducing risk. Budget tests as a separate deliverable with their own acceptance rate and defect categories.

Split the task into discovery, generation, static cleanup, runtime repair, flake investigation, review, and maintenance. Record input and output tokens, tool calls, test runner minutes, retries, changed fixtures, and reviewer edits. Divide by tests retained after review or by meaningful behavior covered, not by raw test count. A hundred trivial assertions can be more expensive than ten durable contract tests.

Repository Conventions Drive Cost

Before generation, retrieve the nearest accepted test, fixture builders, mocking policy, naming conventions, and exact library versions. Small, relevant examples reduce invented APIs and redundant setup. Do not send an entire test tree when a focused pattern answers the question. Version drift between documentation and installed dependencies is a common source of repair loops.

Create a test steering file that distinguishes unit, integration, browser, and contract boundaries. Specify what may be mocked, required cleanup, clock and network controls, and commands for each tier. Review it like code. When the agent repeatedly fails the same setup, fix the shared context rather than adding an explanation to every prompt.

Language and Framework Matter

Java tests may expose import, generic, checked-exception, and dependency-injection errors early. Rust reveals ownership and trait constraints. TypeScript catches interface and mock-shape mismatches, while Python and JavaScript may defer similar problems until execution. A low static-error rate in a dynamic language does not mean low cleanup cost; it often shifts spend from compiler feedback to runtime debugging.

Baseline each stack separately. Measure compile failures, runtime failures, flaky failures, setup duration, and human changes per retained test. Route complex framework setup to a model proven on that repository while using cheaper models for repetitive table cases. The right model mix for production code may not be the right mix for tests.

Validate in Layers

Run formatting and static diagnostics first, then the smallest targeted test, then related packages, and finally required full gates. Return concise failures to the agent and stop identical retries. If a browser or integration test requires unavailable infrastructure, report the missing precondition rather than asking the model to rewrite valid code around an environment problem.

Check mutation value or fault sensitivity on a sample when coverage numbers rise sharply. Ensure assertions fail when the target behavior is intentionally broken. Review tests for overspecified implementation details, broad snapshots, hidden network access, and excessive sleeps. These patterns pass today while increasing future maintenance cost.

Forecast and Improve

Estimate expected test cost from cases requested multiplied by generation cost, divided by first-pass retention, plus cleanup and execution. Add a flake reserve based on historical rates. Report a range because integration tests have long tails. Compare the estimate with the risk reduction and reviewer time of writing tests manually, not with the token price alone.

Track retained tests, meaningful branches covered, failures caught before merge, escaped regressions, cleanup minutes, and maintenance changes after three months. Remove generated tests that provide no durable signal. Test generation pays when repository context is precise, validation is layered, and success means stable defect detection. Treating every emitted test as value merely transfers cost into future CI and maintenance.

Avoid Test-Volume Incentives

Create a decision record for the retained AI-generated test suite. Capture meaningful behavior covered, mutation or fault sensitivity, compile and runtime repairs, flake rate, execution minutes, reviewer changes, regressions caught, later edits, skips, and deletions. Store the record with the accepted artifact rather than only in a temporary model transcript. This evidence lets engineering reproduce an outcome, finance explain the bill, and reviewers distinguish a real improvement from a lucky run. It also makes exceptions visible instead of allowing them to become undocumented defaults across future sessions.

The main failure to avoid is that targets based on raw test count or coverage reward trivial assertions and large snapshots. Define an owner, a measurable threshold, and an expiry date for the decision. sample tests after several releases and feed maintenance outcomes into routing. When evidence falls outside the approved range, stop expansion, preserve state, and investigate before buying more attempts. This operating discipline converts a one-time technique into a bounded policy that can remain economical as models, tools, teams, and repositories evolve.

Keep one rejected test example beside each accepted pattern so future agents learn what the repository does not consider useful coverage.