Ask HN: Fastest Crawl of HN Articles HN links to over 6 million urls in stories and comments. Many domains have expired or content is no longer available. Internet archive has much of the content but throttles requests. What's the fastest way to get the historical content? |
https://github.com/HackerNews/API
I'm not sure what rate limiting policy is in place, but in theory you can start with a request for maxitem and from that point on just GET all items down to zero until you hit some sort of blocker.
As you said the HN api is great and there are at least 2 existing published crawls of it that help a lot.
I might not have a clear picture of what you're looking for, but items of type "story" returned by the HN API do have a URL field, which I believe correspond to submitted links.
You can scrape the text field of comment items, but that takes a bit more work.
There's a dataset containing everything: bigquery-public-data.hacker_news.full
You can write SQL and is super fast. Sample:
SELECT * FROM bigquery-public-data.hacker_news.full LIMIT 1
I don't understand your question. If you have the URL, you just GET it, like any regular URL? Is there something that I'm missing?