Content Independence Day: no AI crawl without compensation(blog.cloudflare.com) |
Content Independence Day: no AI crawl without compensation(blog.cloudflare.com) |
I'd be cautious about relying on just the good will of Cloudflare.
It's unfortunate that we need honeypots and tarpits to trap AI scrapers just so that our hosting bills don't get hosed. It's taking a good chunk of value out of running a site on the Internet.
Definitely don't rewrite your web server more efficiently in Rust instead. /s
For example today some bot from the range 14.224.0.0-14.255.255.255 got crazy and caused a storm of 404s. Dozens per second for hours on end. So I blocked the range like this:
iptables -A INPUT -m iprange --src-range 14.224.0.0-14.255.255.255 -j DROP
That's probably not the best way and might block significant parts of whole countries. But at least it keeps my service alive for now.
What do others here do to protect their servers?
We wouldn't mind if bots simply cloned the repo every week or something. But instead they crawl through the entire reflog. Fucking stupid behavior, and one that has cost us an extra $50/month even with just the 404.
I am advising all my clients away from informational content which is easily remixed by LLMs. And I'm not bothering anymore with targeting informational search queries on my own sites.
I'm doubling down on community and interaction. Finding ways to interact with original content with smaller audiences, rather than produce information for a global search audience.
How is this done, technically? User agent checking? IP range blocking?
Also, what a piece of zero-trust shit the web is becoming thanks to a couple of shit heads who really need to extract monetary value out of everything. Even if this non-solution were to work, the prospect of putting every website behind Cloudsnare is not a good one anyway.
What the web needs right now, to be honest, is machetes. In ample quantity. Tell me who's running that crawler that is bothering you and I will put them to the sword. They won't even need to present a JWK in the header.
Cloudflare to introduce pay-per-crawl for AI bots
Great statement in theory - but in practice, the whole people-as-a-service industry for AI data generation is IMO more damaging to the knowledge ecosystem than open data. e.g. companies like pareto.ai
"Proprietary data for pennies on the dollar" is the late-stage capitalism equivalent of the postdoctoral research trap.
This ends up hurting individuals and small companies that are harmless and cannot afford to pay
Swapping meaning poisons the LLM but makes it really difficult for a preprocessing step to understand the difference between good and bad inputs.
The standard response to a crawler is a 402 Payment Required response, probably as a result of an aggressive bot detection.
So essentially, it's turning a site's entire content into an API: Either sign up for an API key or get blocked.
The question remains though how well they will be able to distinguish bot traffic from humans - also, will they make an exception for search engines?
The idea behind the headers is to allow bots to bypass automatic bot filtering, not blockade all regular traffic. In other words:
- we block bots (the website owner can configure how aggressively we block) - unless they say they're from an AI crawler we've vetted, as attested by the signature headers - in which case we let them pay - and then they get to access the content
(Disclosure: I wrote the web bot auth implementation Cloudflare uses for pay per crawl)
> Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing.
I don't see how it would make sense otherwise, as the requirements for crawlers include applying for a registration with Cloudflare.
Who in their right mind would jump through registration hoops only so they can not access a site? This wouldn't even keep away the crawlers that are operating today.
I agree there has to be some way to distinguish crawlers from regular users, but the only way I can see how this could be done is with bot detection algorithms.
...which are imperfect and will likely flag some legitimate human users as bots. So yes, this will probably leading to web browsing becoming even more unpleasant.
So I'm cautiously optimistic. Well, I suppose pessimistic too: if this works what this will mean is that all contents will end up moving into big player hosting like CF.