Stealth Bot Bills Would Sweep In Journalists and Archivists, Not Just Bad Crawlers
- More than 300 media representatives went to Capitol Hill for the Stealth Bot Prohibition Act (H.R. 9915), which would require every automated website visitor to disclose its identity and purpose, with FTC penalties up to $53,000 for a stealth bot reasonably likely to damage, impair, or burden a site's technical or commercial operation.
- The bill defines a bot as software that retrieves, scans, indexes, scrapes, or otherwise accesses a website, but gives no standard for what counts as reasonably likely to impair a site.
- New York's Stealth Crawler Prohibition Act (S9934A) applies only to covered news sites and lets a publisher subpoena a crawler's identity with no showing of harm; a company that identifies itself and then hits a site 10,000 times a minute still complies.
- The authors cite the Wall Street Journal's 2022 Pulitzer-winning investigation, which analyzed 31,000 financial disclosures from 12,000 senior government employees to show that nearly a quarter traded stock in industries their agencies regulated, as work that requires anonymous crawling.
- The Internet Archive's Wayback Machine has crawled more than one trillion webpages, and the authors argue anonymity matters because subjects change their behavior once they know they are observed.
Hacker News 의견들
This reads like AI lobby work to kill the bill. All you'd have to do is identify yourself as a bot and respect robots.txt, which was the status quo before a bunch of assholes started systematically destroying the internet.
Does the bill even touch human users? Maybe I'm reading it wrong.
You can still ban anyone who ignores robots.txt. Half the sites I crawl don't even serve one, so I back off on 429s and lean on Crawl-Delay. The Chinese trawlerbots rotate smart TV residential proxies and copy someone else's user agent, so UA blocking barely helps.
The internet is global but the fines are US-only. If the scraping comes from Russia, China, India, or France, how does this reduce anything? Feels like a contempt of business model bill.
Knowing the identity is what lets you block them, and you can already go after them for DDoS.
The premise doesn't even hold up. Every browser is a program running on a computer, so is Firefox a bot? An ad blocker? NewPipe? If those aren't the target, what's the line?
Geofencing is spreading precisely because of the bot traffic. A journalist crawling anonymously looks identical to a bad actor, and the publisher whose ad revenue is the trade watches an LLM compress its content and hand it out for free.
I don't think there's any real way to enforce this.
Bots are mostly good. The classic web is just inefficient, otherwise nobody would be complaining about them.
Open web people have absorbed Facebook and Google values. The anime girl gate, cookie dialogs, email walls, Wikipedia doing the same. A friend published a neural net in PHP around 2008 that read CAPTCHAs and got his site hacked by Russians.
Funny that the WSJ piece finds a quarter of federal officials trading in industries they regulate. How many own Nvidia? How many Nvidia employees hold T-bills? Stakeholder capitalism for everyone.
You can already buy a share of a national economy. It's called currency, and cash is a fraction of the economy the way a share is a fraction of a company.
Corporations and governments have swapped places.
It objectively does stop bad bots. Run any website and you can see it.
And site owners don't care about the humans caught in their blocks.
I'd love to sue whoever scrapes my personal site, but if it turns out to be Amazon or Alibaba I'm just screwed. A regular person can't go after them.