Mistral Leanstral 1.5: The Cost of AI Formal Verification vs Manual Security Audit

What Mistral Just Released

On July 3, Mistral AI released Leanstral 1.5 under an Apache-2.0 license: a 119B-total, 6B-active-parameter model specifically trained for formal verification in Lean 4. The headline numbers are unusually crisp for a benchmark launch. It saturates miniF2F. It solves 587 of 672 PutnamBench problems. It hits state-of-the-art on FATE-H (87%) and FATE-X (34%). Most importantly for engineering teams, when Mistral pointed it at 57 open-source repositories, it uncovered five previously unknown bugs — including in Rust codebases, where formal verification tooling has historically been thin.

A free HuggingFace weight drop and a free API tier means the marginal API cost of running Leanstral 1.5 is $0 today. That is the number worth staring at. For the first time, a serious formal-verification workflow can run at zero LLM cost per property, changing the break-even math against manual audit engagements.

Cost Baseline: The Manual Audit

A formal security audit of a mid-sized Rust codebase (say, 30-50k lines) from a reputable shop like Trail of Bits or Sigma Prime typically runs $40,000-$120,000 for a 4-6 week engagement, delivering a report covering 20-60 specific properties. That works out to roughly $1,500-$3,000 per verified property, plus the engineering time to hand back the report, argue findings, and land fixes.

Now compare Leanstral 1.5 in an agent loop. On the Mistral team's own run — 57 open-source repos, 5 bugs found — the ratio was 11.4 repos per bug and, if we assume roughly 8-12 hours of engineering time per repo to set up the harness and review outputs, roughly 100-140 hours of human time to produce those 5 findings. At a fully-loaded rate of $150/hour, that is $15,000-$21,000 for 5 verified bugs, or $3,000-$4,200 per bug. On the surface, that looks worse than the audit shop — but the important detail is that the bugs Leanstral found were novel to already-audited codebases. The audit price does not include finding what the audit missed.

Where the Free API Actually Costs You

"Free API" is a marketing headline, not an operational reality. A serious verification workflow reveals four real cost lines:

• Rate limits. Free tiers cap concurrent requests, which stretches a wall-clock timeline. A 500-property sweep that a paid API can chew through in 45 minutes may take 20-30 hours on the free tier.
• Lean 4 harness engineering. Leanstral operates as a multi-turn prover that iterates against Lean compiler feedback. Getting that loop set up on an arbitrary codebase means writing translations from Rust invariants to Lean specifications — the biggest hidden cost.
• Human triage of false leads. The model reports failed proof attempts as candidate bugs; a human still has to distinguish "your spec is wrong" from "your code is wrong."
• GPU cost if you self-host. Running the 119B/6B active model at reasonable throughput needs one 80GB H100 or two 48GB L40S GPUs — roughly $2-4/hour on cloud pricing, or a fixed capex hit if on-prem.

Realistic Per-Property Cost With Leanstral 1.5

Assume a 40k-line Rust codebase with 30 formal properties you want to verify. Rough estimates:

Roughly a 5-8x cost reduction, and a 3-4x wall-clock reduction. The comparison is not perfectly apples-to-apples — an audit shop delivers a reputation-backed report you can hand a regulator, and Leanstral delivers machine-checked proofs that an engineer still has to interpret — but for regression verification, invariant checking, and pre-audit hardening, the ratio is real.

The Bugs Leanstral Actually Found

Mistral was careful to note that the five bugs uncovered in the 57-repo sweep came from codebases that had received prior human review. That is the critical selling point for cost-conscious teams: the model is not competing with an unaudited codebase, it is competing with the residual bug population that survives audits. Two of the five bugs were in Rust projects — the industry's canonical "provably safe" language — and one involved an incorrect assumption about lifetime bounds that a human reviewer had signed off on.

This mirrors what agentic testing research (like Dan Luu's recent Galapagos notes) has been arguing: LLMs are not competitive with fuzzing on cold-start bug discovery, but they are extremely leveraged at closing the gap in properties humans failed to spec correctly.

How to Fold Leanstral 1.5 Into an Existing Budget

1. Pilot on a bounded module first. Pick a single 3-5k line library (a crypto primitive, a parser, a state machine) with well-understood invariants. Time-box the harness setup to one engineer-week.
2. Run Leanstral before commissioning a paid audit, not after. A pre-audit sweep either finds low-hanging bugs cheaply or gives the audit team a scoped starting point.
3. Budget for the harness, not the tokens. The Lean 4 translations are 70-80% of your true cost.
4. Self-host if free-tier throughput is your bottleneck. An H100 rental for a 5-day sweep costs ~$400; skipping the rate-limit wait is worth it for any project on a deadline.
5. Do not skip human review of proof attempts. A "verified" property is only as strong as the specification it was verified against.

The Meta-Point

For a year the argument against "LLMs for formal verification" has been that the domain is too rigorous, the proof search too specialized, and the model outputs too hallucination-prone to trust. Leanstral 1.5 collapses two of the three objections: proof search now works at benchmark-saturating levels, and the outputs are machine-checked by Lean rather than trusted directly. What remains is the harness engineering — which is a solvable, one-time cost per codebase.

Any team spending more than $40k/year on external audits should have a Leanstral pilot on the roadmap this quarter. The free tier gets you the proof-of-value at zero API cost; the moment it earns its keep, the paid tier pays for itself with the wall-clock reduction alone.