LeCun co-authored paper proves Platonic Representation Theorem for JEPAs, says good views need 42x fewer samples than next-token prediction
- Randall Balestriero and Yann LeCun prove a Platonic Representation Theorem: JEPAs trained separately on any number of modalities learn the same representation up to one rotation, as long as positive pairs are two observations of one scene and misalignment is bounded otherwise.
- Next-token prediction is a JEPA with bad views, parts of a single document, which the authors give as the reason language models align with vision only through scale.
- Good views need 42x fewer training samples than next-token prediction on the same corpus to reach the same alignment, in runs validated from 145k captions to the texts of 940M web images.
- A 109M-parameter text JEPA aligns with DINOv2 and CLIP image encoders better than every language model they evaluated up to 72B parameters, and better than CLIP's own text encoder.
- The theory also produces a captioner: one rotation turns separately trained encoders into an image captioner scoring CIDEr .838 on COCO, and the text JEPA reaches 64% top-5 on ImageNet-100 against at most 56% for zero-shot language models up to 8B parameters.
Hacker News 의견들
Not sure how I feel about the Platonic Representation Hypothesis. Half of me thinks it's profound, the other half thinks it just says there's an optimal dimensionality reduction and bigger models approximate it better.
It matters because it says something about training smaller models, no? If the representation is shared, maybe you don't need the scale to get there.
It sounds a lot less profound if you swap 'representations of the world' for 'compressed representations of data' or 'decision boundaries in a classifier over data'.
What if we end up with two internally consistent but mutually incompatible representations? We already see versions of that in deployed systems, and it's not obvious there's one truth to converge on.
Feels inevitable to me once you think about it. Every network is modeling the same reality, so it would be weirder if they ended up incompatible. It's like reducing different equations to the same simplest form.
As a physicist I'd switch from Euclidean to spherical coordinates or to a Lagrangian and the problem dissolves. Whether there's a true platonic frame or just task-dependent tradeoffs between error and compute, I'm not sure. Reading further, their PRH is an explicitly defined theorem, so my hand-waving doesn't really apply.
I thought the hypothesis was way more ambitious, like applying to any model including human minds, not just the same architecture trained on different modalities. I suspect the assumptions kill the interesting part.
It's clickbait. They don't prove the platonic representation hypothesis. That said the training efficiency results look genuinely interesting, the paper is worth a read.
To keep a model from ever forming concepts like theft or a robot uprising, you'd have to cut so many primitive concepts that the model stops being useful. That's why I don't buy the 'just leave it out of the training data' idea.
The title coincidence is wild. I sent investors a deck on Oct 27 with 'AI infrastructure for proving and improving...' and Claude helped me pick that phrasing. Two days later it's the title of this paper.
We're all talking to the same model, so we converge on the most plausible phrasing and then believe it was our own idea. Two days isn't enough time for a retrain, so it's more subtle than stealing. Also 'proving and improving' has plenty of hits in Google Books.
A 33M-parameter text JEPA hits .125 on the PRH benchmark versus .034 on Flickr captions, level with Qwen2.5-3B and -7B with far fewer samples. Bullish on JEPA. The success of next-token prediction looks like a stroke of luck.
The real question is which domains have good views to pretrain on. Next-token prediction works because it can eat the entire unstructured web. Can JEPA assemble a training set like that at scale?
Every example in the HF demo has a wrong answer. 'A man in a black shirt and hat with a yellow neckerchief has his hand raised' and no hand is raised. The sheep one is just cake and sheep, and none of them are right.
It's two models trained separately on images and captions, so of course it's bad at captioning. Most numbers in the paper are bad in absolute terms, they're just surprisingly good given zero paired training data, which might help where paired data is hard to get.
I'm tired of theory papers that bolt human interpretation onto a math result to hit whatever buzzword is in fashion. Ironically ChatGPT is the fastest way to strip the marketing off and see what's actually new.
The math is the result. Swapping terminology gets you an intuition and nothing materially useful.