Ask HN: Who is running the abusive scrapers? Personally I once had to triple app capacity and double database capacity just for just one scraper. I remember Cloudflare once figured out that Perplexity was one of the bad guys ( https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ ) but has anyone else figured out who else is running large scale and poorly behaved scrapers? |
Don't know, I just block them. These [1] are some of the methods I use with exception to method 7. I need to make time to try out the SSL fingerprint detection. They are crude methods but most of the bots / scrapers are kicked to the curb. The feds shutting down NetNut helped a lot too. I am looking forward to them shutting down more residential proxies.
Mirror [2]
[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
[2] - https://archive.is/kVLJZ