Beyond $/M Tokens: Token Efficiency Metrics That Actually Predict AI Coding ROI

Why $/M Tokens Alone Misleads

Every AI coding vendor publishes their model price as "$X per million input tokens, $Y per million output tokens." It's a clean, comparable number — and it's insufficient for making real decisions. Two examples that show why:

• Model A costs $2/$6 per M tokens with a 40% first-attempt success rate on your typical tasks.
• Model B costs $5/$25 per M tokens with a 75% first-attempt success rate.

On the sticker price, Model A is 3-4x cheaper. On cost per completed task, Model B wins by ~20% because Model A needs 2-2.5 attempts on average. Add developer wait-time for retries and Model B saves another $10-20 per task. The $/M comparison hid all of that.

This guide walks through four metrics that reveal real ROI, how to instrument them, and how to use them to make model-selection decisions.

Metric 1: Tokens Per Completed Task

Tokens per completed task is the median number of input + output tokens consumed across all successful task attempts. It's not "tokens per attempt" — attempts that fail don't count.

To measure it, you need:

• A definition of "task complete" (tests pass, PR merged, feature deployed — pick one and be consistent).
• Session-level token accounting (most gateway tools give this for free).
• A way to tag tasks with an outcome (successful/failed).

Once instrumented, tokens per completed task lets you compare models on an "apples to apples" basis. A model that uses 30% fewer tokens per successful outcome is 30% cheaper regardless of sticker price. This is the metric that lets you objectively evaluate OpenAI's "54% token efficiency" claims for GPT-5.6.

Metric 2: Cost Per Verified Output

Cost per verified output takes tokens per completed task one step further: instead of counting any task marked "complete," it counts only tasks whose output passed a verification step (tests, human review, or an LLM judge).

The distinction matters because "complete" is often self-reported by the agent. A coding agent that claims "I've fixed the bug" doesn't necessarily produce a fix that actually works. Cost per verified output measures the price of actual value delivered.

Instrumentation requires a verification pipeline — running tests after each agent-generated change, using a code-review LLM (Claude Haiku is a common choice at $0.80/$4 per M tokens), or human sign-off. This is more work than tokens-per-completed-task but the payoff is higher: you catch models that "complete" tasks with subtle bugs that eat developer time downstream.

Metric 3: Retry-Adjusted Cost

Retry-adjusted cost sums the token cost of all attempts (successful and failed) and divides by the number of successful outcomes. It's the metric that captures the true cost of trying tasks with a cheap-but-error-prone model.

Formula: Retry-adjusted cost = (Total tokens across all attempts × price per token) / (Successful completions)

Applied to our earlier Model A / Model B example:

• Model A: Average 2.5 attempts × 15K tokens × $6/M = $0.225 per successful task
• Model B: Average 1.3 attempts × 15K tokens × $25/M = $0.488 per successful task

In this case, Model A still wins on pure retry-adjusted token cost. But this ignores time, which is where Metric 4 comes in.

Metric 4: Time-Weighted Spend

Time-weighted spend adds the cost of developer time to the retry-adjusted cost. Formula:

Time-weighted spend = Retry-adjusted token cost + (Avg minutes of developer wait time per task × Developer hourly rate / 60)

For a $100/hour developer waiting 15 minutes on a failed attempt (Model A) versus 3 minutes on a successful first attempt (Model B):

• Model A time cost: 15 minutes × $100/60 = $25 in wait time per failed attempt × 1.5 avg failed attempts = $37.50 in wasted developer time per completed task
• Model B time cost: 3 minutes × $100/60 = $5 in wait time per completed task
• Model A total: $0.225 tokens + $37.50 time = $37.72 per completed task
• Model B total: $0.488 tokens + $5 time = $5.49 per completed task

When developer time enters the equation, Model B wins by 7x. This is why "cheap models save money" is often wrong for productive engineering teams — the developer time saved by a higher success rate typically dwarfs the token savings.

When Each Metric Matters

Not every team should optimize the same metric:

• Solo indie developers: Retry-adjusted cost matters most. Your time is not fungible — a failed attempt costs you no monetary time, just satisfaction.
• Startup engineering teams: Time-weighted spend. Every 15 minutes of developer time costs real dollars; retries eat productivity.
• Enterprise coding teams with formal review: Cost per verified output. Bugs that pass model self-verification but fail human review are hidden costs.
• Autonomous / async agent workflows (Devin, Cursor Composer): Tokens per completed task, since no human is waiting.

Practical Implementation

Instrumenting these metrics requires:

1. A gateway or wrapper that tags every API call with task ID, session ID, developer ID, model, and timestamp.
2. A definition of "task complete" that your team applies consistently (tests pass, PR merged, code review approved).
3. An outcome tagging step: at the end of each session, mark it successful, failed, or abandoned.
4. A dashboard rolling up token cost, attempt count, wait time, and outcome across model choices.

Even a rough version — a spreadsheet where developers self-report task outcomes for two weeks — will beat comparing models on $/M tokens alone. The point isn't perfect measurement; the point is escaping the trap of choosing models based on sticker price when the real economic drivers are elsewhere.