Dermatologist's test: a 2015 Inception v3 CNN beats GPT-6 Astra and Claude Fable 5.1 at skin cancer diagnosis, 62% to 42% on 200 biopsy-proven lesions
- A dermatologist gave 200 biopsy-proven lesions to GPT-6 Astra, Claude Fable 5.1 and a fine-tuned Inception v3: the CNN named the right first-choice diagnosis for 125 (62%), the two LLMs for 84 and 85 (42%)
- With a second choice allowed, the CNN reached 173 of 200 against 134 and 138, and it caught 13 of 30 melanomas against 8 for GPT-6 Astra and 11 for Claude Fable 5.1
- The widest gap was on harmless lesions: the CNN recognised 37 of 50 moles and 26 of 50 seborrhoeic keratoses, against 16 and 13 for GPT-6 Astra and 33 and 6 for Claude Fable 5.1
- Both LLMs called an invasive melanoma under 0.8 mm on the ear a seborrhoeic keratosis and said the patient could be discharged; the CNN called it a melanoma
- The CNN is Inception v3, published by Google in 2015, fine-tuned on 68,011 ISIC Archive images on a single H100 for about 20 minutes with all 200 test lesions stripped from training and validation, and it runs in a phone browser through WebGPU
Hacker News opinions
Give the LLM a calculator MCP and the calculator's advantage disappears. Same story here, except nobody has a plug-in that reads a dermoscope.
Colleagues keep asking why bother training new models when you can just ask an LLM. Doesn't matter what the domain is, the answer from some researchers is always LLM.
I did expect better from the LLMs though. There are tens of thousands of public skin cancer images on the internet, so a general model should have picked up more of this than 42%.
This isn't surprising. LLMs are spiky on the stuff frontier labs benchmark, mostly code and math, and nobody claims they perfect every existing workflow. I'm sure a frontier LLM could improve that CNN.
As an MLE this article isn't useful. It's a fine-tuned CNN against general purpose models on a tiny 200-lesion dataset.
Frontier models aren't good at most tasks. They're good at programming because there's a mountain of training data and the task rhymes with writing reports. Images of skin cancer probably don't map onto an LLM's internal representation at all.
For anything at scale a dedicated classifier with the right architecture wins. LLMs just push the cost of building one close to zero, so use the LLM to make the thing that's actually good at the job.
It's the pareto principle. 80% of applications need 20% of the intelligence, and if I can drop something in without curating high quality training data, that's great. Skin cancer probably isn't one of those.
You're making the curse of knowledge mistake. You know which words get an LLM to build and train a classifier. An average person doesn't, and people are lazy.
An off-the-shelf API has way less friction. If the results are good enough, why bother? This is like saying we don't need Dropbox because we have FTP servers.
Training needs labels and data. Getting running with an LLM needs a prompt and nothing else. That's the whole tradeoff.
LLM decision models are still new and flaky. The order of the choices changes the probabilities, they start breaking down past 20 classes, and rewording the prompt moves the answer.
There's a significant engineering cost to a custom classifier, agreed, but I've been stunned by how well LLMs code and do data analysis, so 42% on a dermoscopy image caught me off guard for a model this recent.
For a lot of applications the general model isn't the destination, it's the labeling crew. Use it for exploration, then train the task-specific model on what it produced.
Most imagined agentic software ends the same way: the AI builds the software, with a classifier sitting in the awkward spots as the expansion joint.
Plenty of software people have no information theory background, so LLMs look like magic to them. When you can't reason from first principles, you bolt duct tape onto the wrong approach.