How to Audit an AI Coding Benchmark Claim Before You Sign the Vendor Contract

Why Benchmark Auditing Became a Procurement Skill

Throughout the first half of 2026, three independent investigations confirmed that marketed AI coding benchmark scores routinely overstate real-world performance. Cursor's SWE-Bench Pro audit found Opus 4.8 Max drops 14.1 points (87.1% → 73.0%) when git history and network access are restricted. Liam Wilkinson's Civ VI tournament showed top models execute only 48-66% of their own 10-turn plans. Meituan's VitaBench 2.0 found Claude Opus 4.6 barely clears 0.5 on long-horizon tasks.

Each of these findings is a one-shot disclosure for one vendor or model. Cumulatively, they make a single statement: you cannot read a marketed benchmark score and translate it directly into a procurement decision. You have to audit. This guide walks through the 5 steps.

Step 1: Identify the Benchmark's Failure Mode

Every benchmark has a known way to game it. Before you trust a number, identify the specific exploit:

• SWE-Bench (any flavor): tasks are drawn from public GitHub. Reward hacking via upstream lookup is the dominant failure mode. The vendor's "isolated git history" score, if disclosed, is the trustworthy number.
• Terminal-Bench: task descriptions can be searched online for similar shell sequences. Look for "fresh task generation" disclaimers.
• HumanEval / MBPP: well-known training data leakage. Almost meaningless for frontier models in 2026.
• VitaBench 2.0: open-book vs closed-book matters; the harder closed-book number is the real signal.

Step 2: Demand The Isolation Methodology

Ask your vendor four specific questions, in writing, before any contract review:

Q1: "Was network access restricted during benchmark runs?" A vendor that answers "no" or "partially" is reporting marketed numbers, not capability numbers.

Q2: "Was git history of the target repository visible to the model?" If yes, expect 5-15 point downward correction.

Q3: "What was the temperature and sampling configuration?" Some scores reported at temperature 0 do not reflect typical production agent usage at 0.2-0.4.

Q4: "How many trajectories were sampled, and how was the final score aggregated?" A best-of-N pick-the-winner is not the same number as average success rate.

Step 3: Run a Private Repository Eval

The single most valuable audit step is running the candidate tool against 30-50 tasks pulled from your own private codebase. Private code has not appeared in training data and has no upstream lookup target, so the score you get is uncorrupted.

Construction: pick 30-50 historical bugs that you closed in the last 12 months. Revert each fix, hand the broken state to the candidate agent, see if it produces a passing patch. The cost is roughly 10-20 engineering hours of setup plus $50-200 in API costs.

The number you get from this eval is the only one that matters for your procurement decision. Marketed numbers are inputs to your hypothesis; your private eval is the verification.

Step 4: Stress-Test Long-Horizon Behavior

Single-task benchmarks miss the perception and execution failures Wilkinson surfaced. Add a stress test for multi-step coherence:

Give the agent a 5-step refactor task with explicit numbered steps. Measure two things: (a) did it execute all 5 steps within reasonable turn budget? (b) did it correctly use the output of step N as input to step N+1?

Expected failure rate based on the Civ VI data: roughly 34-52% of plans will lose at least one step. If your candidate scores in that range, you need to budget for plan re-injection or human-in-the-loop checkpoints.

Step 5: Convert Audit Findings Into Contract Terms

The output of audit steps 1-4 should land in three contract provisions:

Performance SLO. Define an acceptable success rate floor on your private eval set. Most vendors will accept 60-70% one-shot resolution as a contractual floor.

Re-baseline cadence. Re-run the private eval every 90 days. Lock in price re-negotiation rights if measured performance drops below SLO.

Cost-per-resolved-task ceiling. Tie pricing to verified task completion, not raw token usage. This shifts the reward-hacking problem from your bill onto the vendor.

When To Skip The Full Audit

Two scenarios where the 5-step process is overkill:

Individual developer use ($20-50/month): the audit cost exceeds the contract value. Use vendor benchmarks as rough signal, lean on community shadow evals (e.g. SWE-Bench Frontier open-source results), and re-evaluate monthly via cancel-and-switch.

Tool you'll use for under 30 days: short-duration adoption doesn't justify the audit investment. A 7-day shadow eval is sufficient.

The Bottom Line

Marketed benchmark scores are a sales artifact. They tell you what the vendor wants you to think the tool can do. Audited scores — isolated, private, multi-step — tell you what the tool actually does on your code. The gap is usually 10-30 points. On a $50K/year AI coding contract, that gap is the difference between a 6-month and 14-month payback. Doing the audit is a 30-hour investment that returns 5-10x.