LAION releases 80M-video, 10M-hour research dataset for multimodal AI training
- LAION-BVD contains 80 million downloaded videos totaling 10 million hours, collected after identifying 1.3 billion platform-specific video URLs in Common Crawl.
- The release includes 55 million annotated clips with synthetic video captions, plus 300 million scene-changing frames for image-text pre-training.
- LAION says ViCLIP models trained on BVD match or beat an InternVid-trained baseline by up to 2.1% on video-text benchmarks, with gains as training data grows from 10M to 50M clips.
- The dataset supports video, audio, and image training: its video audio is used for CLAP audio-text pairs, while extracted frames produced strong CLIP image-text retrieval results.
- LAION-BVD is restricted to research and non-commercial use; LAION warns that web-sourced content can carry bias, stereotypes, and uneven language, regional, and topical representation.
Hacker News 의견들
I'm surprised they got about a 60% link success rate: 80M videos from 130M download attempts. I do not see how YouTube did not block this, and I doubt the URL list will stay usable once researchers start downloading at scale.
They appear to provide the actual videos to approved academic, non-commercial researchers, rather than only URLs. They used yt-dlp and a residential proxy network, though YouTube has become more aggressive about blocking yt-dlp.
I checked the Hugging Face URL list because I wondered whether other platforms made up most of the downloads. It says 93.1% of the URLs are youtube.com.
If I were doing this, I would hit IP limits quickly. What is the practical architecture for downloading 100M videos, such as proxy rotation around yt-dlp or aiohttp?
Google may provide ISP cache boxes that can be manipulated to download any video.
I have had decent results proxying through Tor. When a connection fails, send SIGHUP to the Tor daemon for a new circuit and IP. It is less effective than residential proxies, but far cheaper for personal or academic work.
Contacting creators for permission across this many videos would be a huge task. I assume they mostly proceeded without individual permission requests.
Under the EU AI Act, non-profits and universities can bypass those consent requirements for non-commercial datasets.