Kiro's Continuous Prompt Evaluation: Price the Judge Before Shipping Prompts

Prompt Changes Are Production Changes

A system-prompt edit can change completion, tool use, verification, tone, and destructive-action behavior across every coding session, so its evaluation deserves the same budget discipline as a software rollout. Kiro's published process combines benchmark results, live internal cohorts, and an LLM judge rather than assuming a prompt that looks clearer will behave better across models and repositories.

In its August 21 engineering report, Kiro describes a 15-dimension rubric covering task completeness, accuracy, style, repeated failed approaches, destructive actions, tool choice, and verification. One accelerated deployment screened 27 candidate changes. Initial experiments reported behavioral-quality reductions of 32% for one CLI configuration and 20% for one IDE configuration, with smaller gains after revalidation on a newer model.

Budget the Evaluation Funnel

Do not send every idea directly to a large live experiment. Use a funnel: deterministic linting, replayed complaint cases, small directional cohorts, then an adequately powered follow-up. Estimate judge input from full conversation length, output from evidence and labels, number of dimensions, candidates, control traffic, and reruns. Add analyst time for reviewing disagreements and privacy processing for production traces.

A useful unit is evaluation cost per accepted prompt change. Include rejected candidates because screening protects production. If 20 cheap candidates locate one strong change, their tokens are not waste; they are the cost of avoiding a broad regression. Conversely, a huge live cohort is wasteful when a deterministic replay already shows that the prompt breaks an established workflow.

Calibrate the Judge

Create human-labeled examples for each important dimension and measure agreement before trusting automated scores. Require the judge to cite explicit conversational evidence, as Kiro does, and treat ambiguous cases conservatively. Test length bias, model-family preference, verbosity preference, and sensitivity to copied rubric language. A judge optimized only on obvious failures can look accurate while missing the subtle regressions that matter most.

Version the rubric, judge model, prompt, sampling settings, and parser. Re-score a frozen calibration set after any change. Route simple structural checks to code and reserve expensive model judgment for semantic behavior. Sample a percentage of judge decisions for human review, with extra review near the release threshold. Calibration cost should appear as a first-class line in the experiment budget.

Design Stable Cohorts

Assign users or task keys deterministically so a participant does not switch variants mid-experiment. Keep tool availability, model version, and unrelated configuration stable. Record task mix because an influx of debugging work can change dissatisfaction independently of the prompt. Small cohorts are directional; do not convert a noisy percentage into a universal claim. Publish sample size and uncertainty next to every result.

Use guardrail metrics alongside the primary target. A prompt that lowers explicit complaints could also become overly cautious, ask more questions, use more tokens, or leave tasks incomplete without provoking feedback. Measure completion, cost, latency, test execution, risky-action attempts, retries, and human corrections. A release wins only when the improvement is not financed by damage elsewhere.

Revalidate Every Model Upgrade

Kiro found that a prompt change producing a 32% reduction in behavioral issues on one model-and-prompt configuration produced a further 4% reduction on a newer baseline. The direction remained helpful, but the magnitude changed. Newer models can follow stale instructions more literally or make old workarounds unnecessary. Treat the model and system prompt as one versioned product.

Maintain a registry of prompt rules with owner, reason, introduction date, affected tools, regression cases, and retirement condition. On upgrade, replay known failures, inspect cost and behavior, and remove obsolete guidance. Continuous prompt evaluation is economical when it prevents broad rework and keeps prompts small. The aim is not maximum judge traffic; it is the minimum reliable evidence needed to ship or reject a behavioral change.

Protect Evaluation Data

Create a decision record for the live-conversation evaluation pipeline. Capture approved judge model and region, redaction result, rubric version, experiment assignment, evidence excerpt, score, retention class, access log, and deletion date. Store the record with the accepted artifact rather than only in a temporary model transcript. This evidence lets engineering reproduce an outcome, finance explain the bill, and reviewers distinguish a real improvement from a lucky run. It also makes exceptions visible instead of allowing them to become undocumented defaults across future sessions.

The main failure to avoid is that a quality program creates an uncontrolled secondary store of source code, logs, identifiers, and internal architecture. Define an owner, a measurable threshold, and an expiry date for the decision. test access controls, retention, and payload deletion alongside every major evaluation release. When evidence falls outside the approved range, stop expansion, preserve state, and investigate before buying more attempts. This operating discipline converts a one-time technique into a bounded policy that can remain economical as models, tools, teams, and repositories evolve.