How to Count Tokens Before Sending: Tokenizer Tools, Prompt Sizing, and Cost Control for Coding Agents

Why Token Counting Matters Before You Hit Send

Most AI coding bill surprises come from one mistake: sending more tokens per call than you thought. A 50K-line repo you dragged into context "to give the model the full picture" is 200K-400K tokens of input. On Claude Opus 4.8 at $15/M input, that single call is $3-6 — and if your agent retries five times you've spent $30 on what felt like a free operation.

The fix is boring and reliable: count before you send. Three categories of tools make this practical.

Tokenizer Library by Provider

Quick Wins You Can Apply Today

Five small habits that cut prompt size without quality loss:

1. Strip comments and blank lines from code context. A typical commented codebase has 20-40% non-code lines. Most coding tasks don't need them. The savings on a 100K-token codebase context: 20K-40K tokens per call.

2. Exclude generated files. package-lock.json, yarn.lock, build outputs, vendor directories, type-generated files. Easy to include accidentally; high token cost; near-zero information content for the model.

3. Send file summaries, not full files, on the first pass. A 500-line file's first-pass summary is 100-200 tokens. Send the summary; let the agent ask for the full file if it needs it. Most tasks never need the full file.

4. Use diffs instead of full files for change-review tasks. git diff output for a typical PR is 500-3000 tokens. The before+after of the same change is 10-50K tokens. Same information.

5. Cap conversation history at N turns. Most coding agents pile on conversation history as input on every turn. After 20 turns this dominates input cost. A rolling window of the last 5-8 turns plus a summary of older content keeps cost flat.

Pre-Send Sizing Workflow

A simple, reliable workflow before sending any large prompt:

Step 1 — Assemble the full prompt locally. System prompt + tool descriptions + context files + user message + conversation history.

Step 2 — Run the appropriate tokenizer. Get a real token count, not a character-based estimate.

Step 3 — Multiply by input price. Token count × price/M tokens. For an estimated output of 2-5K tokens, add that to the bill.

Step 4 — Compare against a threshold. Per-call cost threshold of $0.50 / $1 / $5 depending on team comfort. Above the threshold: prompt for confirmation or auto-truncate.

Real-World Savings

A team running a coding agent at scale that adds a pre-send tokenizer + per-call cost preview typically sees:

The headline: 60% cost reduction with no measurable quality loss. The savings come from sending the right amount of context, not more.

Specific Tools Worth Knowing

Tiktokenizer.app — web-based, supports OpenAI, Claude, and Llama tokenizers. Paste text, see token count and visual tokenization. Free.

Anthropic count_tokens API endpoint — POST to /v1/messages/count_tokens with the same payload you'd send to messages. Returns token count. Free to call.

OpenAI tiktoken CLI — pip install tiktoken, then count tokens in a file or piped stdin. Useful in build scripts and CI.

ai-cost-estimator — for ballpark project-level cost estimates without per-prompt counting; useful for "is this whole project going to cost $5K or $50K?" planning.

Pre-send token counting is one of the least exciting and most reliably effective AI coding cost interventions. Build it into your agent's pipeline once; benefit from it on every call thereafter.