AI agents formalize 16% of CPython in Lean, 613K lines and 418 bug reports
- Opus 5.5 rewrote CPython functions in Lean line by line: 613,318 lines across 1,539 files with 18,898 theorems and lemmas, and 4,071 functions have passed independent review, covering 16% of CPython 3.15.2.
- The proofs turned up 418 bug reports against CPython. Most are small, but about 55 can crash the interpreter or corrupt memory from ordinary Python code.
- Every function goes through four steps: a line-by-line Lean copy, a side-by-side test against real Python on at least 50,000 tricky inputs, a machine-checked proof, then review by a second agent that did not write it.
- The first breadth-first attempt failed review: fewer than half the accepted functions held up and one file only 21%, because a proof like 'if it returns, the result is at least zero' also passes a function that always crashes.
- Coverage is uneven: grammar rules 100%, built-in types 35%, parser 31%, C modules 19%, interpreter core 10%, standard library 9%.
Hacker News 의견들
The Julia folks ran basically the same experiment on their stdlib, and the prompt for this Python work actually points at the julia-in-lean repo.
The bug hunting is clearly promising, it finds a ton. The problem is making it routine. This took a lot of human plus agent work, so it needs to be opt-in properties checked in the test suite, not a one-off project.
At that point you might as well just use Lean and skip the two language problem entirely.
Only 418 bugs? I would have guessed closer to 10k. Professional Python programmer, for what it's worth.
They don't have full coverage yet, it's 16% of CPython so far.
Cite an AI-written document? Hell no.
Serious question: does anyone actually read articles written by AI?
Stupid question, but if the Lean code is a line-by-line copy of the C, how does it know which behavior is a bug and which is intended?
Because the proof only goes through with strange assumptions, like every number staying under 2^32. That assumption is exactly the condition that triggers the bug, so it needs a human look either way.
Wouldn't the tokens be better spent writing ACSL annotations and verifying with Frama-C? You'd check the actual runtime instead of an AI translation, and fixes could go upstream directly.
Two things I don't get. If the rewrite is line by line, how do the two implementations ever disagree in step 2? And how does it decide what should always be true in step 3?
The agent prompt lists sources in order: the implementation disagreeing with itself, undisputed invariants like no crash or no hang, then doc sentences read in context. But how the interesting statements get picked is still pretty vague in the post.
You can write solvers for the gnarly bits and wire them into your test runs. I do that for AI-written code where drift wouldn't be obvious to me.