Use cases range from sending 100s of requests per second just to bring a website down to doing a montly request to a municipality's endpoint to get a local dashboard of when trash is picked up. I don't think you can pass a single judgement of automating web requests in general.

> Is it ethical to scrape when a site has explicitly blocked bots

Ethical? ppl get jailed in China for this. "Breaching computer systems" is a felony.

I think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.

It's most definitely unethical.

Algae

fellow contributor on this.

we thought of it the other way round: your automation is the fortress, and every bot-detector trying to fingerprint it is the siege.

also most good burglar names were taken on PyPI

Trojan Horse probably wouldn't make a very good name for a browser.

Reminds me of this old joke: No ethically trained software engineer would ever write a destroyBaghdad function - they’d write a destroyCity function to which you could pass Baghdad as a parameter.

I think this is the point that has become a moral dilemma. Increasingly, in this agentic age tasks like research and web exploration that would be done by humans and even buy things from an website is now done through agents. While I sympathize with the main author about Ddos, but blocking every agent regardless of intent seems cruel. This would allow an MCP doing deep research(almost identical to human behavior) to move forward. But there should definitely be more work done system that focus on agent intent(i.e some agent trying to find the best price for vacation or scraper building alternatives to Travel Advisor).

Why did you need it from a script? Right click -> Save Page As... is still a thing and works quite well in my experience, even with complex React SPAs.

It's not just excess bandwidth. It's cpu cycles, poor site coding, database issues. Like I said, not everything is backed by a huge company that has the money/hardware/engineering resources to absorb the load.

That’s the point. For every legitimate though ultimately unimportant use case like yours there are a thousand others who would scrape the results 86,400 times a day to either try to sell or mirror and plaster with their own ads.

CDN security SWE here. I have functionally zero interest in what individual people do in terms of automation. I automated my gym's class signup forum during COVID so I get the validity of the use case. You want to work around the mitigations to do the same? Go with my blessings.

The killer is that everything that works for individuals trying to get through the day and make the web a bit smoother is immediately used by industrial crawlers strip mining the Internet, and you can't block one without blocking the other. A future where the web is only accessible via device attestation is extremely dystopian but so is an Internet where every drop of content goes into an LLM training set.

Things like this is why all of the worthwhile content is going to drain into balkanized spaces, and we're all the poorer for it.