What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?

Edit: the article says millions of times per hour? (!?)

The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?

Feels like it would be a good time for freenet and the like to catch on.

I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.

You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.

A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.

Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.

Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!

I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).

I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.

I think they also have to operate in countries that don't mind shady things like this.

TIL:

> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.

WTF. That's just botnets.

Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...

Thank god for residential proxies.

Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet

How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?

just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.

Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.

I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.

What exactly should be illegal here? Scraping websites? AI agents? Not following robots.txt?

"He who has the gold makes the rules" is older than the pyramids.

> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.

Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.

Seriously, dang?

10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.

This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.

I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.

atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.

As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.

Just curious dang, did you warn them before banning?

Im not against the ban perse (single purpose accounts are bad), just curious if they had a chance to change their contribution style.

This is a strong positive sign that poison fountain works.

I wasn't aware of this project. Thanks for the heads up.

Yeah. A fun thing to do is to try and actually read common crawl!

Really makes you think, what we're feeding them...

exactly i took a look at that subreddit and doesnt look like theres any professionals just bunch of anti-AI users who thinks they are smarter

its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective

also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities