What Is JSON Mode / Structured Output — and Its Hidden Token Costs

JSON mode — also called structured output — is a feature that constrains a model to emit valid JSON, usually matching a schema you supply. Instead of hoping the model returns parseable text, you guarantee it. It is the backbone of tool calling, function calling, and any agent that has to feed model output into downstream code. It is also a quiet source of token cost that most developers never audit.

What It Is and Why Coding Agents Use It

When a coding agent decides to "edit a file" or "run a test," it does not write prose — it emits a structured object naming the tool and its arguments. Structured output makes that reliable: the model is forced to produce fields your runtime can parse without regex hacks or retries on malformed responses. That reliability is why every serious agent framework leans on it. The alternative — parsing free-form text and hoping the model wrapped its answer in the delimiters you asked for — fails often enough that the retries cost more than the schema ever would. The tradeoff is that the schema defining those fields is not free.

The Hidden Cost, Part 1 — Your Schema Is Input Tokens

Here is what surprises people: the JSON schema you attach counts as input tokens on every single call. A verbose schema with long field names, nested definitions, and a paragraph of description on each property gets re-sent with every request. If your schema is 800 tokens and you make 500 calls, that is 400,000 input tokens spent on the schema alone — before the model answers anything.

On Sonnet 5 at $2 per million input tokens, those 400,000 tokens cost $0.80 just to transmit the schema across 500 calls. That is a fixed tax you pay for structure, and it scales linearly with call volume. Multiply across a busy agent making tens of thousands of tool calls a day and the schema line item becomes real money.

The Hidden Cost, Part 2 — Structure Is Output Tokens

The other half of the bill is on the output side. Every brace, quote, colon, comma, and escaped character the model emits is an output token — and output tokens cost 4–5x more than input on most models. The worst offender is repetition: in an array of 100 objects, every long key name is generated 100 times. Structural characters carry no semantic payload, yet you pay premium output rates for all of them.

Terse vs. Verbose Keys — A Worked Comparison

Consider a schema returning a list of code issues, each with a severity, a file path, and a description. Compare the key choices:

• Verbose: "severityLevel", "affectedFilePath", "detailedDescription" — around 12 tokens of keys per object.
• Terse: "sev", "path", "msg" — around 4 tokens of keys per object.

Across 100 objects, the verbose version generates ~1,200 tokens of keys versus ~400 for the terse one — an 800-token difference of pure overhead, per response. On Sonnet 5's $10 output rate, that is roughly $0.008 wasted per call, or $8 per thousand calls, for nothing but longer names. The verbose schema also inflates every input request because the schema definition itself is bigger.

How to Cut Structured-Output Cost

Structured output is worth its cost when it prevents parse failures and retries. The goal is to keep the structure and drop the waste:

1. Keep schemas minimal. Include only the fields you actually consume. Every optional field you "might use later" bills on every call today.
2. Use short, consistent key names. When a key repeats thousands of times across array items, three characters beats fifteen.
3. Cache the schema with prompt caching. If your provider supports it, cache the unchanging schema so repeated calls read it from cache at a fraction of the input rate instead of re-billing full price.
4. Avoid re-sending unchanged schemas. If the schema is identical across a session, it is a prime caching candidate rather than a fresh payload each time.
5. Prefer enums over free text. A constrained "sev": "high" from a fixed set is cheaper and more reliable than an open string the model has to compose and you have to validate.

Tie It Back to Model Selection

Structured-output overhead interacts with model choice. Because the schema tax lands on input tokens, a model with a low input rate and a fat schema can still be cheaper than a premium model with the leanest schema. For high-volume tool-calling loops, a cheaper tier with prompt caching often wins outright. When the structured reasoning itself is hard, a premium model earns its rate — but pay for the intelligence, not for verbose key names.

Want to see how schema size and call volume translate into a monthly total across models? Plug your per-call input and output estimates into the AI cost calculator and watch how trimming a schema or switching tiers moves the number.