Risk-Based Test Selection for AI Coding: Save CI Without Buying Defects

Running every test after every agent edit can dominate task cost and feedback time. Running too few converts CI savings into escaped defects and rework. Risk-based selection chooses a smaller early suite from the changed behavior and consequence, while keeping clear conditions that require the full gate.

Measure the baseline first: test duration, compute cost, queue time, flake rate, failure yield, and defects caught by suite and change class. Include model tokens spent waiting for or interpreting results and human interruption. A slow suite may still be economical if it catches expensive failures; a fast flaky suite can trigger costly repair loops.

Build a Change-to-Test Map

Map files and components to unit, integration, contract, security, migration, and end-to-end tests using imports, build graphs, ownership, runtime traces, and failure history. Add semantic triggers: authentication, billing, persistence, permissions, public APIs, schemas, and infrastructure deserve tests beyond simple file reachability.

Version the map and record why each test was selected. If the system only returns a test list, engineers cannot diagnose missing coverage. Unknown dependencies should expand the suite rather than be interpreted as no risk. Generated selection must fail safe when repository metadata is stale.

Use Test Tiers

Run local static checks and directly affected unit tests first. Add integration and contract tests for dependency boundaries. Run end-to-end, security, migration, and performance suites when risk triggers appear. Full suites remain required for release branches, dependency graph changes, broad refactors, selection uncertainty, and scheduled coverage audits.

Set a maximum time saved, not a minimum number of tests. One high-value contract test can matter more than hundreds of trivial units. Preserve test setup and teardown dependencies; selecting an individual test without its environment can create false failures or false confidence.

Account for Flakes

Track flaky tests separately from genuine failures. Limit automatic retries and preserve the first failure evidence. Repeatedly rerunning a flaky suite can erase all selection savings. Quarantine only with an owner, issue, and expiry, and keep risk coverage through an alternative check when possible.

When an agent changes production code to satisfy a flaky result, cost and risk compound. Require evidence that the failure reproduces and relates to the diff before editing. Log rerun count, tokens, compute, and wall time under the original task.

Evaluate Missed Failures

Shadow the selector: run the chosen suite, then run the full suite without blocking users during evaluation. Count failures missed, detection delay, and consequence. Sample clean full-suite results too. Continue periodic full runs after launch because repositories and dependency graphs evolve.

Calculate savings from compute, queue, developer wait, and agent runtime. Subtract selector maintenance, shadow runs, missed defects, reruns, and investigation. Report cost per accepted change and escaped-defect rate by risk class. Do not celebrate CI-minute reduction alone.

Give the Agent a Stop Rule

If selected tests fail twice with the same unexplained signature, the agent should stop guessing. If changed files exceed the planned scope, public behavior changes, or dependency reach is unknown, expand the suite. At final release, run the repository's required quality gates regardless of early selection savings.

Risk-based selection is best used for fast iteration, not for weakening release assurance. Build explainable mappings, tier tests by consequence, measure misses through shadow full runs, and make uncertainty expand coverage. This cuts feedback cost while preserving the evidence needed to trust agent-generated changes.

Audit the Selector Itself

Treat selection code, dependency graphs, and risk rules as production assurance tooling. Require review, tests, versioning, and rollback. A change to the selector can affect every repository task while producing no application diff. Log its version with each chosen suite so missed coverage can be reproduced.

Review false negatives, excessive selections, unknown mappings, and tests that never run. Add repository owners to the audit because automated graphs may miss semantic dependencies known to maintainers. Retire obsolete rules and refresh coverage after major architecture changes. Measure the selector's own compute and maintenance cost.

Set a service objective for explanation: engineers should see why a test was included and why a high-risk suite was omitted. If the system cannot explain a surprising choice, run the broader suite. Explainability is an operational fallback that prevents opaque optimization from quietly becoming reduced assurance.

Include test-environment failures in the decision. An unavailable database, exhausted runner, or broken fixture does not prove the change is safe and should not be classified as a passing selection. Retry infrastructure failures under a bounded policy, then stop or move to a healthy environment. Preserve uncertainty in the final gate instead of converting missing evidence into green status to save queue time.