Bytedance's 'Don't Optimize for Code Contribution Rate' Reflection: A New AI Coding Cost KPI Framework

A Year of Receipts From a 100K-Engineer Org

On June 24, 2026, Bytedance VP of Engineering 洪定坤 (Hong Dingkun) published a long reflection on the company's first year of pushing AI-assisted coding through its 100K-engineer org. The piece covered missteps, successful patterns, and the specific infrastructure (Harness) Bytedance built to measure agent impact. Several findings in the piece directly contradict how most engineering orgs are currently tracking AI coding ROI.

The most important reframing: code contribution rate is a vanity metric, not a cost-effectiveness signal. Most companies use "what percentage of merged code came from AI?" as a top-line metric. Bytedance's Harness data showed this metric is loosely correlated with actual cost savings and tightly correlated with overspending.

Why Code-Contribution Rate Misleads

The mechanism: when teams are graded on AI-generated-code percentage, engineers route low-difficulty tasks (boilerplate, getter/setter, formatting) to AI to pump the number. These tasks were already cheap; routing them through AI adds tokens without adding value. Meanwhile, the high-leverage tasks — debugging, architecture, complex refactors — get routed away from AI because using AI on them is slower than experienced humans.

Bytedance's data showed teams with the highest AI contribution rates often had: (1) higher token spend per shipped feature, (2) more reverts and follow-up commits per AI-generated PR, (3) flat or negative impact on engineering velocity. Teams with moderate AI contribution rates and disciplined task routing showed the opposite pattern.

The Three Metrics Bytedance Switched To

Harness — Bytedance's internal AI coding observability platform — replaced contribution-rate dashboards with three cost-aware metrics:

Cost per shipped feature. Total agent token spend divided by features shipped (gated by acceptance criteria, not just merged PRs). This metric is honest because it accounts for re-work, re-roll, and tasks where the agent's output had to be discarded. Bytedance's stated target: keep this trending downward quarter-over-quarter.

Time-to-first-mergeable-output. From "engineer asks agent for help" to "agent produces something the engineer was willing to commit." This captures both the speed and quality dimensions, and penalizes agents that get stuck in retry loops. Bytedance reported this metric improved 60-80% over the year as workflows matured.

Token-spend-per-engineer-hour-saved. The flagship cost-effectiveness metric. Engineers self-report (or telemetry estimates) hours saved per agent interaction, divided by token spend. Bytedance found teams in the top decile were 3-5× more cost-effective than the bottom decile — same agents, same models, vastly different operational discipline.

What Cost-Effective Teams Actually Do

The piece identified four behaviors that distinguished top-decile from bottom-decile teams:

Task selection. They routed tasks to agents based on difficulty match, not blanket policy. Hard reasoning tasks went to Claude Opus; routine refactors went to Haiku or DeepSeek; boilerplate went to in-house cheaper models or templates. Top-decile teams had 5-8× more model variety in their routing.

Aggressive context pruning. Bottom-decile teams sent the entire codebase as context. Top-decile teams sent the minimum: relevant files, function signatures, recent diffs. This single discipline cut their input token spend 40-70%.

Hard token caps per task. Top-decile teams set a maximum token budget per agent run — typically $0.50-$2.00. Runs hitting the cap were terminated and either re-scoped or escalated to human attention. This eliminated runaway-loop costs that dominated bottom-decile bills.

Quality gating before merge. Top-decile teams treated agent output as draft, not finished work. They ran tests, code reviews, and lint passes — then either merged or fed errors back to the agent for one more attempt. Bottom-decile teams either merged optimistically (and paid in production bugs) or re-prompted aggressively (and paid in tokens).

A Reusable KPI Template for Other Teams

Most engineering orgs are not Bytedance scale. The KPI framework still translates cleanly. A practical adoption path:

• Stop reporting AI contribution rate as a top-line metric
• Start tracking cost-per-shipped-feature monthly
• Add time-to-first-mergeable-output to your team retros
• Estimate engineer-hours-saved-per-token-spend with a quarterly survey
• Set hard per-task token caps in your agent tooling

Even a rough version of these metrics will reveal more about your AI coding cost-effectiveness than any contribution-rate dashboard.

The Underlying Lesson

The Bytedance piece is a useful reminder that the cost story for AI coding is not just "models got cheaper." It is "operational discipline matters as much as price-per-token." Two teams running identical models and tools can have 3-5× different bills, and the difference is in how they select tasks, prune context, cap runs, and gate output. None of those levers require waiting for the next model release.

For developers and team leads watching their AI coding budget grow, the actionable takeaway is uncomfortable: the dashboard you've been showing leadership is probably wrong, and switching to cost-aware metrics will reveal both wins and embarrassments. Worth doing anyway.