Web scraping is not ethical if it is done in a way that the site owner disallows.

The problem is that there is no way to defend against it, so the question of "ethics" is largely academic. The only adequate defense is to not put anything you care about on a publicly accessible website.

According to Microsoft it is, because [they assert] anything on the web is in the public domain: "freeware." [0]

Eric Schmidt says it doesn't matter, just do it and let the lawyers clean it up. [1]

0: https://www.forbes.com/sites/barrycollins/2024/07/01/has-mic...

1: https://www.msn.com/en-us/money/news/ex-google-ceo-schmidt-t...