Prompt Experiment Sample Size: Stop Overspending on Noisy Agent A/B Tests

Small Samples Create Expensive Confidence

A prompt A/B test can spend thousands of model and judge calls while producing a result too noisy to guide a release. Sample size is therefore a cost-control decision, not only a statistics exercise. Before generating traffic, define the baseline rate, smallest improvement worth shipping, acceptable false-positive risk, desired power, unit of assignment, and primary outcome.

The unit should usually be a user, repository, or stable task key rather than an individual turn. Assigning each turn independently allows one developer to experience both prompts and creates correlated observations that look larger than the true sample. Stable assignment costs no extra tokens and makes the experiment easier to interpret. Record repeat users and clustered tasks when calculating uncertainty.

Define the Minimum Useful Effect

Do not power a test to detect any mathematically nonzero difference. Choose the smallest effect that repays rollout, migration, and maintenance cost. A 0.2-point improvement in acceptance may be real but operationally irrelevant if it adds latency or requires a longer prompt on every request. Express the threshold in completed tasks, saved review minutes, avoided failures, or dollars per accepted change.

Use historical data to estimate the baseline and variability, then calculate a sample range with a standard statistical tool. Inflate the range for cluster assignment, missing outcomes, judge ambiguity, and expected task-mix imbalance. If required traffic is too large, choose a more sensitive measure, a stronger preproduction replay, or a larger practical effect. Do not simply run an underpowered test and label the result directional after seeing it.

Stage the Funnel

Begin with deterministic checks: prompt length, prohibited instructions, tool schema compatibility, and known regression cases. Next replay a frozen set of representative conversations. Then use a small live cohort to detect obvious harm, followed by the powered comparison. Each stage should have explicit stop conditions. Candidates that fail early do not deserve more traffic merely because a schedule reserved it.

Reuse common control results only when model, tools, repository snapshot, and evaluation rubric are unchanged. Otherwise, stale controls create false savings. Cache immutable judge inputs and deterministic outputs, but preserve privacy and retention limits. Record which evaluation artifacts were reused so the experiment bill and audit trail remain understandable.

Budget Primary and Guardrail Metrics

Pick one primary metric such as verified task completion. Add a small set of guardrails: destructive-action attempts, test execution, output tokens, latency, retries, and human corrections. Do not run an expensive semantic judge for facts available from logs or tests. Code can count tool calls and gate results; reserve model judgment for behavior that deterministic checks cannot classify.

Estimate cost before launch: conversations per cohort multiplied by average input and output, judge passes, rerates, model rates, sandbox time, and analyst review. Add a contingency for long conversations and disagreements. Set a hard ceiling and specify whether the test stops, samples, or moves to a cheaper judge when the ceiling is approached. Changing rules midstream compromises both budget and inference.

Read Results Without Chasing Noise

Report the estimated effect with an interval and raw counts. A result crossing the release threshold is more useful than a bare significant or not-significant label. Inspect segment consistency, but avoid dozens of unplanned slices that manufacture a winner. If one language appears different, treat it as a new hypothesis and run a dedicated follow-up rather than rewriting the conclusion.

Store the prompt, model, assignment logic, sample calculation, rubric, exclusions, cost, and release decision. Revalidate after model or tool changes because the same prompt can behave differently. A disciplined sample plan prevents two kinds of waste: experiments too small to answer the question and experiments far larger than needed. Both spend tokens; only one looks rigorous on a dashboard.

Plan Sequential Decisions Carefully

Create a decision record for the prompt experiment stopping rule. Capture fixed-horizon or sequential design, expected and maximum samples, interim boundaries, safety stop conditions, exclusions, raw counts, effect interval, and actual evaluation cost. Store the record with the accepted artifact rather than only in a temporary model transcript. This evidence lets engineering reproduce an outcome, finance explain the bill, and reviewers distinguish a real improvement from a lucky run. It also makes exceptions visible instead of allowing them to become undocumented defaults across future sessions.

The main failure to avoid is that daily dashboard peeking lets an experiment owner stop at the most flattering random moment. Define an owner, a measurable threshold, and an expiry date for the decision. prespecify boundaries and apply the same rule to positive and negative results. When evidence falls outside the approved range, stop expansion, preserve state, and investigate before buying more attempts. This operating discipline converts a one-time technique into a bounded policy that can remain economical as models, tools, teams, and repositories evolve.