Same Code, 73% More Tokens: Why $/Token Doesn't Compare Across Claude, GPT & Gemini
By Eric Bush · July 14, 2026 · 7 min read
The Comparison Everyone Gets Wrong
When developers compare AI coding models, the first number they reach for is the sticker price: dollars per million tokens. Claude Opus 4.8 is $5 input / $25 output, GPT-5.6 Sol is $5 / $30, Gemini 3.1 Pro is $2 / $12. Easy, right? Cheapest per token wins.
Except a token is not a token. A widely-shared analysis circulating this week highlighted an uncomfortable fact: the same file produces a wildly different token count depending on which model's tokenizer counts it. And since you pay per token, that means the real cost of the identical task can differ far more than the sticker prices suggest.
681 vs 1,178 Tokens for the Same File
The example that made the rounds: a single TypeScript source file tokenized to 681 tokens under a GPT-5.x tokenizer but 1,178 tokens under Claude's newest tokenizer — a 1.73x difference for byte-identical input. The claim extends further: the newer Claude tokenizer reportedly produces roughly 30% more tokens on code than its predecessor while list prices stayed flat, which the author framed as a de-facto price increase.
A word of caution: this is a community analysis, not an official statement from any provider. Tokenizer behavior varies enormously by content type — dense code with lots of symbols, indentation, and camelCase identifiers tokenizes very differently from prose. The exact ratios depend on the specific files tested. Treat "73% more" as a real phenomenon worth measuring on your codebase, not a fixed universal constant.
Why Tokenizers Diverge
Every model family trains its own tokenizer — a lookup table that maps text fragments to integer IDs. The choices that go into that table decide how efficiently a given kind of text compresses:
- Vocabulary size — a larger vocabulary can pack more characters into a single token, lowering counts for common patterns
- Training corpus mix — a tokenizer trained heavily on code learns efficient merges for
=>,const, and indentation runs; one trained mostly on prose does not - Whitespace handling — how leading spaces and tabs are grouped matters enormously for indented code
- Multilingual coverage — supporting more human languages can crowd out code-specific merges, inflating code token counts
Two models can advertise the same "128K context window," but if one tokenizer is 30% less efficient on your code, you effectively get 30% less usable context — and you pay 30% more to fill it.
The Real Cost Math
Suppose you feed a 10,000-"GPT-token" codebase context into two models with identical $5/M input pricing. If Model A's tokenizer is 1.5x less efficient on that code, you are actually billed for 15,000 tokens instead of 10,000. The effective input price is no longer $5/M — it is $7.50/M for the same information delivered.
This flips naive rankings. A model that looks 20% cheaper on paper can be more expensive in practice if its tokenizer is 40% less efficient on your language. Output is affected too: reasoning traces and generated code are billed in the model's own tokens, so a "cheaper" output rate can be eaten by a chattier tokenizer.
How to Compare Models Honestly
The only fair comparison is cost per completed task, measured on your own code:
- Take a representative sample of your actual source files
- Tokenize the same sample with each candidate model's official tokenizer or token-counting endpoint
- Multiply each token count by that model's per-token price to get the effective price for your content
- Run a handful of real tasks end-to-end and compare total spend, not sticker rates
Our cost estimator works in task-level terms precisely because sticker prices lie. Two providers at "$5/M input" are not offering the same deal if one silently charges you for 50% more tokens on the exact same file.
The Takeaway
Tokenizer efficiency is one of the most under-discussed variables in AI coding costs. It hides behind a clean-looking per-token price and quietly rewrites your bill. Before you standardize a team on the "cheapest" model, tokenize your real files and compute the effective rate. The winner on paper is often not the winner on the invoice.
Want to calculate exact costs for your project?
Frequently Asked Questions
Why does the same code produce different token counts on different models?
Each model family trains its own tokenizer with different vocabulary sizes, training data, and whitespace rules. A tokenizer trained heavily on code compresses source files more efficiently than one optimized for prose or many human languages, so the identical file can be 30-70% more tokens on one model than another.
Does a lower per-token price always mean a cheaper model?
No. If a model's tokenizer is less efficient on your content, you are billed for more tokens per task, raising the effective price. A model advertised 20% cheaper per token can cost more in practice if its tokenizer produces 40% more tokens on your code.
Is the claim that Claude's tokenizer raised prices 30% official?
No. That figure comes from community analysis, not an official provider statement. It reflects a real phenomenon — tokenizers differ — but the exact ratio depends on the specific files tested. Measure it on your own codebase rather than treating any single number as universal.
How should I compare AI coding models fairly?
Compare cost per completed task, not sticker price per token. Tokenize a sample of your real source files with each model's tokenizer, multiply by that model's price to get the effective rate, then run real tasks end-to-end and compare total spend.
Does tokenizer efficiency affect the context window too?
Yes. A less efficient tokenizer means the same code consumes more of the context window, so a '128K' window holds less of your actual code — and costs more to fill — than a more efficient tokenizer's window of the same nominal size.
Related Articles
AI-Assisted i18n Translation Cost: Token Math per 1,000 UI Strings (Claude, GPT, Gemini 2026)
Translating 1,000 UI strings with an LLM sounds cheap. Then variable placeholders, brand terms, and re-review overhead multiply the token count 4-6x. Here is what it actually costs across Claude, GPT, and Gemini.
AI Documentation Generation Cost: README, JSDoc, Docstrings Compared Across Claude, GPT, and Gemini
Auto-generating docs for a 50K-line repo is one of the highest-ROI AI coding workflows — if you pick the right model. We break down cost per 1000 lines of code documented across providers and doc styles.
Prompt Caching Across Claude, GPT, and Gemini: A 2026 Cost-Saving Playbook for Coding Agents
Prompt caching is the single biggest cost lever for AI coding agents in 2026 — but every provider implements it differently. We compare Anthropic's explicit breakpoints, OpenAI's new GPT-5.6 30-minute contract, and Gemini's implicit prefix caching. Numbers, decision rules, and the migration trade-offs for switching between them.