Ask HN: Why Reddit blocks all automated access but has .json for all URLs? Reddit is aggressively blocking all automated access (see their robots.txt) and uses a lot of heuristics to block crawlers that do not honor it. However, at the same time all Reddit URLs can be made machine readable by adding a .json to the end. Can anyone explain what's the point of that? |
https://arstechnica.com/gadgets/2023/10/reddit-may-block-sea...
It does make one wonder for a site so easy to scrape, why there aren't any popular third-party clients that use scraped data, like FreeTube and NewPipe do with YouTube.
In other words, the current state is the result of hard won experience not syllogistic reasoning.
At least to me because benign crawling is likely to be a tiny subset of all crawling activity that is not done by google, etc.