Load Shedding for AI Coding Agents: Prevent Queue Spikes from Becoming Token Bill Spikes

An agent fleet can turn a normal traffic spike into a cost incident. A burst of commits starts reviews, reviews trigger tools, tools hit rate limits, and retries add more work to the same queue. Backpressure and load shedding keep demand inside a budget before token usage compounds.

Traditional services protect latency and availability with queues and admission control. AI coding systems need the same controls plus economic awareness because every queued item may carry a variable model bill. The system should decide what to run now, delay, downgrade, merge, or reject based on value, freshness, risk, and remaining spend.

Why Agent Queues Amplify Cost

Agent tasks are not fixed-size requests. One code review may finish in two calls; another can inspect dozens of files, invoke tests, and repair failures. When the provider throttles, workers wait or retry. If each retry includes the conversation history, input cost grows. Meanwhile, duplicate repository events may enqueue work that is stale before execution begins.

Autoscaling workers without a budget signal can make the situation worse. More workers consume the backlog faster but also send more simultaneous model and tool traffic, increasing throttling and external CI load. Capacity must consider downstream limits and the maximum acceptable dollars per minute.

Admission Control Comes First

Estimate each task before accepting it using repository size, changed files, event type, model tier, and historical runs. Reserve a budget envelope rather than assuming the mean. If the queue lacks capacity or spend headroom, return a visible delayed state instead of pretending the task started.

Use idempotency keys for repository, commit, workflow, and purpose. A force-push or repeated webhook should replace or deduplicate stale work, not create another full review. Cancel jobs tied to superseded commits before they spend more tokens.

Define a Degradation Ladder

• Normal: preferred model, full repository tools, and normal iteration limit.
• Constrained: efficient model, smaller context, fewer parallel workers, and tighter output limits.
• Essential only: security, release blockers, and explicitly requested work; routine summaries wait.
• Paused: no new autonomous work when the daily budget or provider health threshold is breached.

Publish which level is active. Silent downgrades make model-quality changes look like random failures and can cause users to resubmit, adding more load.

Prioritize by Value and Freshness

A security fix blocking production should outrank documentation on an old branch. Use separate queues for interactive and background work so a nightly migration cannot consume every slot. Add deadlines: a pull-request summary for a superseded commit has zero value and should expire automatically.

Fairness matters in shared fleets. Give teams weighted quotas and borrowing rules rather than one global first-in queue. A noisy repository should not starve every other project. Return unused reservations when tasks complete below estimate so capacity becomes available quickly.

Make Queue Cost Observable

Expose queued tasks, reserved dollars, actual dollars, age, priority, attempts, and cancellation state. A dashboard that shows only completed token spend reacts too late because a large liability may already be waiting or in flight. Forecast the worst permitted completion cost of accepted tasks and compare that committed amount with the remaining budget.

Trace one task across the model, tool, and CI systems with the same identifier. This prevents separate retry layers from hiding multiplication: the queue retries a worker, the worker retries the provider, and the tool retries a job. Central policy should cap the combined tree, not each layer independently.

Retry with Budgets, Not Hope

Classify failures. A rate limit deserves delayed exponential backoff with jitter. An invalid tool argument may justify one corrected attempt. A deterministic test failure needs new evidence or a different strategy, not the same prompt. Set a maximum incremental cost and elapsed time for retries, and carry the original task's reservation through the entire loop.

Circuit breakers should watch provider errors, tool errors, queue age, tokens per completed task, and dollars per minute. Open the circuit before the monthly cap is exhausted. A daily or hourly burn-rate limit reacts faster than a single monthly budget.

Test Overload Deliberately

Replay webhook bursts, provider throttling, slow CI, and tool outages in a staging environment. Verify deduplication, cancellation, priority order, degradation notices, and budget reservations. Measure the maximum cost incurred after the pause threshold; delayed cancellation can leave many in-flight calls spending after the dashboard says stopped.

Load shedding is not refusing valuable AI work. It preserves the highest-value tasks and prevents retries or stale events from buying nothing. Use historical task costs from our AI Cost Calculator scenarios to size reservations, then enforce spend rate alongside concurrency.