AI crawlers consume 14 CPU cores rendering kernel commits that could be cloned
- Across five geo-distributed git.kernel.org nodes, scrapers consume more CPU rendering Git commits into HTML than all legitimate access combined, including Git clones, and continuously occupy about 14 CPU cores.
- The linux.git repository has about 1.48 million commits and 922 forks on git.kernel.org; crawlers can fetch duplicate HTML for the same commit histories across billions of valid URLs instead of cloning shared Git objects.
- cgit can render commits, patches, plain views, and arbitrary diffs, creating about 1.2 metric bajillion valid URLs for one linux.git fork and a page space too large for conventional caching.
- Operators first blocked identifiable bots with fail2ban and ASN bans, but crawlers shifted to millions of residential and mobile IPs that mimic browsers, make four or five requests, then disappear.
- The Anubis proof-of-work gate initially stopped crawlers at SHA-256 difficulty 4, but bots later solved it, forcing higher difficulty and more friction for legitimate users.
Hacker News opinions
I have seen Blogger traffic arrive from tiny countries, obscure browsers, and outdated operating systems. VPNs plus header-masking tools such as ModHeader, BrowserMask, Crawlee Python, and Masquerade-Spoofer make this easy for scrapers and costly for hosts.
I would put Varnish, now called Vinyl, in front of it. Most Git content is effectively static, so a cache should cut far more rendering work than generating every cgit page repeatedly.
The cache cannot retain this workload. The old commits are revisited after eviction, and cgit has a combinatorial page space that is far too large to store, roughly a quadrillion crawlable pages by the article's calculation.
If the deterministic data is only about 7 GB, there ought to be a way to optimize the renderer or precompute it. Still, the server should not have to redesign itself because crawlers choose the worst access pattern.
A special cloning path only helps for the small share of the web that is cloneable. A crawler processing billions of pages is unlikely to build a source-specific optimization for every site with a better export mechanism.
I find the "AI is dumb because crawlers are dumb" line unhelpful. These bots may use AI-written code, but they are not running inference on every fetched page, and doing so at pretraining scale would greatly raise training cost.
Dynamic Anubis difficulty per IP does not solve a botnet that makes four or five requests per residential IP and then rotates away. Without a persistent identifier, defenses must either burden every visitor or remove expensive cgit functions.
I suspect these crawlers have mixed purposes: pretraining data, security-bug searches, and coding agents. The worrying question is whether free community services now need identity-bound requests or redesigns for hostile access patterns.
I have run CVSweb, Subversion, and cgit at home for years, and this is the first time scraping has made my bandwidth annoying. Blocking one IP just makes another continue the same crawl a second later.
I took down my cgit repository a few months ago because the load made other VMs on the same machine unusable. I have had better results with git-arr.