Verification Is Not Knowledge: Jessica Hullman argues AI for science still needs human accountability
- Northwestern's Jessica Hullman argues in a Cosmos Institute guest essay that AI has made correctness and accountability look separable, and that peer review should keep treating papers as human exchange rather than pure output checking.
- Hullman builds the case on Terry Tao's thought experiment: an AI-derived cancer treatment with a formal certificate of correctness and a passed phase 3 trial but no human understanding of how it was found, where critics replied that a stage 4 patient would take the drug anyway.
- The failure mode she names is not anyone's mental state but what happens afterward. Tao's scenario turns on the AI having passed the trial by exploiting a weakness in the trial itself, and a paper whose authors cannot explain its claims leaves no one to consult when they fail.
- Tom Dietterich flagged arXiv's growing problem with submissions by human authors who likely lack the expertise to understand their own claims; Hullman suggests reviewers reserve expert judgment for texts not flagged as heavily AI-written.
- She quotes Bill Thurston on math research, whose measure of success is whether the work enables people to understand and think more clearly about mathematics, and asks where the belief that scientific value lies in correctness alone came from.
Hacker News 의견들
I'd restate it as 'verification is not human knowledge', but fine. We're going to be several steps behind AI once it goes well past human capability, like trying to outplay a chess engine. Eventually no human will reach that level of knowledge in a lifetime.
How do we even know when that happens if it's beyond us to comprehend? An open-ended claim like that can always come true someday, same as flying cars.
As long as humans are in charge we need to understand what's going on, in science or in code. Give up all the knowledge to AI and you can be manipulated and fooled with ease.
No reason to worry about superintelligence until AI actually reaches human level, if it ever does, and that could be very far off. Everything it has comes from human data it was trained on. Crunching numbers faster isn't intelligence.
I've never heard such drivel in my life. Current AI isn't intelligent, it beats humans at some things the way Deep Blue did. I don't believe LLMs reach human intelligence, let alone superintelligence.
You don't need an LLM for a chess engine or fast computer vision, we've had those forever. A good word predictor happens to be great at closed systems with defined rules and a huge corpus. Ask it for a poem or a riddle and see how little intelligence it has.
Is science about humans achieving insight, or the archiving of verifiable solutions? I'm not sure that dichotomy captures the value of discoveries. Practical impact can be independent of whether humans understand how they were reached.
Impact has always been independent of understanding. Roman concrete and borosilicate glass were sophisticated tech with zero theory behind them, and lasers sat for decades with deep understanding and no use.
The real problem is short term versus long term. A solution alone helps now, but if humans don't build understanding it probably blocks long-term progress. It's much harder to build on top of something you don't understand.
Most of pharma, honestly. That's the whole industry.
If a machine had told Heisenberg the critical mass for a bomb was much lower than he thought, we could have had a different world regardless of whether he understood it.
This reminds me of math class as a kid. The teacher wanted to see how I got there, not just the right answer, even though the right answer is right either way. AI is to science what the calculator was to elementary math. But verification is knowledge itself, and the verifiers can explain the answers to us.
Meaning requires integration, and I assume AI can make meaning of it now or later. I just don't see the point of knowledge without meaning.
I'm not convinced LLMs can explain. They reverse engineer amazingly but are terrible at explaining and abstracting, which is probably a training artifact. Still, it shows they can do things they can't explain to us.
We're not there yet for science. Maybe for math proofs, but that's a very different domain since we know the rules of the game in a way we don't with science.
I sucked at remembering trig identities but was fast at testing a few options on my calculator to check which one I was using. Without the calculator I was nowhere.
There's also ambiguity in what counts as a right answer. In hard sciences we drop second-order details and ignore them, and someone explaining why they dropped that detail is the valuable part.
My teachers kept saying they were teaching the method, not the answer. I always thought I didn't need the method if I could find the answer anyway. Nowadays I think that wasn't quite right.
Everyone knows Tao, but do you know Peter Scholze? Tao lists him among the most important mathematicians of our age. He generates understanding rather than just proofs, and he might be the template for future mathematicians.
Even Scholze isn't always sure of the correctness of extremely complicated statements and welcomes computer verification. Look at the Liquid Tensor experiment. He spent much of 2019 obsessed with the proof and still had lingering doubts. Math isn't far from software engineering, per Curry-Howard.