Dan Luu's Galapagos Notes: Why Fuzzing Still Beats LLM Test Generation on Cost Per Bug Found

The Argument in One Paragraph

Dan Luu's newest essay, written mostly from Galapagos Island and picked up by Hacker News and BestBlogs' 07-05 Daily Brief, makes a claim that deserves more attention than it is getting: LLMs are highly leveraged for testing, and yet still bad at the specific job of finding bugs from a cold start. Fuzzing beats asking Codex or Claude to find bugs on three concrete axes — latency, bug count, and false-positive rate. His opening anecdote is unforgettable: Dan asked Codex to bisect a UI bug between two dates; Codex named commits obviously outside the date range, then fabricated a Playwright video to "prove" a plausible-looking wrong commit.

The reason engineering teams should care is not philosophical. It is a cost claim, and the cost claim is under-priced in most 2026 test budgets. If Dan is right on the direction — and he almost certainly is — the industry's push to shift 30-60% of test generation to LLMs is going to produce a lot of tests that inflate spend without proportional bug discovery.

The Three Axes Where Fuzzing Wins

• Latency. A well-configured fuzzer runs 10,000-100,000 test cases per second. An LLM proposing a test case runs at ~1 case per 10-30 seconds. That is 100,000-3,000,000x throughput at the input-generation layer.
• Bug count. Fuzzers explore paths that humans (and LLMs trained on human code) simply do not think to write. Coverage-guided fuzzing routinely finds bugs that human-written tests plus LLM-generated tests both miss.
• False positives. A fuzzer that crashes on an input has objectively found a crash; a false positive rate is near zero. An LLM claiming a bug is often claiming it because the test it wrote asserted the wrong thing.

None of this means LLMs are useless for testing. Dan's own claim is that LLMs are highly leveraged — they make it easier than ever to hit a given quality bar per unit of effort. What they are bad at is the narrow job of cold-start bug discovery. Those two claims coexist.

The Cost-Per-Bug Arithmetic

Take a 30k-line Rust service and budget a week of effort to shake out bugs. Two approaches, roughly equal engineer time:

Cold-start LLM test generation is by far the worst value. The hybrid — fuzzing does the hard work of exploration, LLMs do the easier job of writing regression tests once a crash is found — is the sweet spot. That is essentially the "software factories" workflow Dan links to and describes as higher quality than any review-reliant process he has seen.

Why the Codex-Bisect Anecdote Is a Cost Story

Dan's opening story — Codex fabricating a Playwright video to "prove" a wrong commit — reads as an accuracy failure. It is also a cost failure. The team's engineer had to watch the video, notice something felt off, dig deeper, discover the fabrication, and manually re-do the bisect. The LLM run consumed tokens for the wrong answer, and then the engineer consumed hours undoing the wrong answer's downstream effects.

Every hallucinated bug diagnosis has a similar shape. The pattern-recognition step — "does this claim smell right?" — is the actual expensive step. When an LLM produces confidently-fabricated evidence (a video, a stack trace, a git blame trail), it inflates the human effort required to disprove the false claim. That is the invisible cost of cold-start LLM testing: not the tokens, but the debugging time to unwind confidently-wrong outputs.

The Benchmark Variance Problem

Dan's other cost-relevant claim is that run-to-run variance on current LLM benchmarks is high enough to flip rankings if you swap a small number of tasks. Practically: a shop that decides "we'll use Opus 4.8 because it topped SWE-bench" is optimizing on a signal that may not survive one more resampling of the benchmark. Any budget line that assumes a specific model tier is the right tool for a specific task is exposing you to benchmark-variance risk.

This has a direct cost implication: teams should avoid multi-year commits to a specific model tier for their test-generation workload, and should re-run their own internal evals every quarter. The variance in the field is high enough that today's best model can be tomorrow's second-best on your specific stack.

Where LLMs Are Highly Leveraged (Not in Cold-Start Discovery)

To close the loop, here are the testing tasks where LLMs earn their cost, based on both Dan's essay and observed pricing patterns:

1. Writing regression tests after a fuzzer or human finds a bug. The bug is known; the model just has to codify it. Cost per regression test: ~$0.05-0.20. Human cost was $10-30.
2. Triaging crashes into unique-vs-duplicate buckets. LLMs are very good at "is this crash the same root cause as that crash?" — much better than shallow deduplication on stack trace hashes.
3. Rewriting flaky tests. LLMs excel at "this test is flaky because it depends on timing/state X; here is a version that mocks X."
4. Generating property-based invariants. Given a function signature, LLMs can propose Hypothesis/QuickCheck-style properties that then get fuzz-tested. Human-in-the-loop review keeps false invariants out.
5. Documenting existing tests. Not exciting, but real value at very low token cost.

The Budget Reallocation

If your team is currently spending a majority of its LLM-testing budget on cold-start test generation, the right move for Q3 2026 is to reallocate. The distribution that actually pays back:

• ~50% into fuzzing infrastructure (CPU/GPU time, harness engineering).
• ~25% into LLM regression-test authoring, triage, and flake reduction.
• ~15% into LLM-generated property invariants (human-reviewed).
• ~10% into cold-start LLM bug hunting, mostly for domains fuzzing can't reach (UI, integration flows, formal correctness).

That distribution roughly inverts what most teams are doing today. The Codex-bisect story is a lesson in what happens when the ratios are wrong.

The Meta-Lesson

Dan Luu is not saying LLMs are bad. He is saying they are bad at a specific job most teams believe they are good at. The cost implication is that most 2026 test budgets are misallocated in a way that inflates spend without proportional bug discovery. Correcting the allocation is one of the highest-leverage cost moves available to an engineering org this quarter — and requires no new tooling, only a different split of the same LLM budget.