Kiro Studied 1.5M Coding Chats: Static Diagnostics Are a Cost Lever

What the Data Actually Measures

Kiro analyzed about 1.5 million internal coding conversations and 406,000 diagnostic-tool invocations, giving teams an unusually concrete view of where static analysis saves repair work and where it still misses costly defects. The study covers seven Claude variants across TypeScript, Python, Java, Rust, Go, Kotlin, C++, and Swift. It observes the agent's own diagnostic calls during generation rather than claiming that every final patch is correct.

The August 26 Kiro analysis reports that proactive diagnostic use occurred in roughly 3% to 22% of conversations. Newer variants converged around 1.2 reported errors per checked file, while unresolved imports represented about one-third to more than half of categorized errors. These observational results mix model, prompt, orchestration, language server, and workspace effects.

Price the Generate-Validate-Repair Loop

Every diagnostic loop consumes model context, tool runtime, and additional output, but it can avoid a much more expensive build failure or human review cycle. Record the cost of the initial edit, each analyzer call, the error output returned to the model, repair tokens, and final verification. Compare that with the cost of allowing the same error to reach CI. The cheapest policy is the one that minimizes accepted-change cost, not analyzer calls.

Run analyzers incrementally after related edits and a complete required set before completion. An agent that checks after every line wastes runtime; one that checks only at the end accumulates interacting errors. Group source, interface, and test changes into a logical unit, then return concise diagnostics with file, line, code, and message. Trimming repeated tool output preserves context without hiding evidence.

Test Files Need a Separate Budget

Kiro found test files consistently produced far more diagnostic errors than source files. Reported averages ranged from 3.56 to 9.13 errors per checked test file across the studied models, compared with lower source-file figures. Mock frameworks, assertion APIs, fixtures, and project-specific setup create a denser compatibility surface. A routing policy based only on production-code performance will underbudget test generation and cleanup.

Measure test work separately: setup discovery, fixture reuse, compile failures, runtime failures, flaky retries, and reviewer corrections. Give the agent examples of current test patterns and installed library versions before generation. Run the smallest relevant test target before the full suite. If a budget model repeatedly fails framework-specific setup, escalate only test construction instead of rerunning the entire task on a premium model.

Language Mix Changes Economics

In the study, Java had a 26.7% file error rate, Rust 15.1%, C++ 12.2%, TypeScript 8.6%, Python 4.0%, and JavaScript 1.6% for the highlighted Opus 4.6 sample. These figures do not prove that Java agents are universally worse; stricter analyzers expose more issues and dynamic languages defer faults to runtime. They do show why one blended team average is a poor forecast.

Build cost baselines by language and task class. Include compile diagnostics for static stacks and runtime, type-check, and contract tests for dynamic stacks. A low editor error count in JavaScript can hide missing validation rather than superior quality. Standardize analyzer installations in benchmark workspaces so a model is not rewarded because one environment reports fewer categories of mistakes.

Turn Diagnostics Into a Gate

Availability alone does not guarantee tool use. Kiro cites related experiments where blocking completion until diagnostics were resolved reduced acceptance of incorrect code dramatically compared with optional availability. Implement a completion contract: the agent must report the analyzer command, result, remaining warnings, and any justified exception. Do not let it declare success after merely editing files.

Track errors per checked file, diagnostic invocation rate, repair success, files checked per call, CI escapes, and total cost per accepted patch. Review changes after model or prompt upgrades because tool-use behavior can move independently of raw generation quality. Static diagnostics are not complete verification, but they are a fast, explainable feedback channel whose economic value can be measured directly.

Audit the Analyzer Environment

Create a decision record for the language-server and static-analysis environment. Capture compiler and extension versions, lint configuration, workspace root, generated-code exclusions, file hashes, diagnostic codes, and the downstream tests used after a clean result. Store the record with the accepted artifact rather than only in a temporary model transcript. This evidence lets engineering reproduce an outcome, finance explain the bill, and reviewers distinguish a real improvement from a lucky run. It also makes exceptions visible instead of allowing them to become undocumented defaults across future sessions.

The main failure to avoid is that one workspace can report fewer errors only because its tooling is weaker or configured differently. Define an owner, a measurable threshold, and an expiry date for the decision. pin benchmark environments and repeat sampled runtime tests after analyzer upgrades. When evidence falls outside the approved range, stop expansion, preserve state, and investigate before buying more attempts. This operating discipline converts a one-time technique into a bounded policy that can remain economical as models, tools, teams, and repositories evolve.