> so that the industry could have a full legitimization from the tech landscape.

I don't think it should have full legitimization. Web scraping always involves putting some of the cost of your business operations onto unconsenting others. In some cases (search engines are the primary example), those others get enough perceived benefit that they're willing to put up with it. But it's still externalizing costs onto the backs of others.

That's why it's a cultural "gray area", and I argue should always remain so.