New HN data dump available with over 14.5m entries

New HN data dump available with over 14.5m entries(archive.org)

79 points by cdman 8 years ago | 13 comments

minimaxir 8 years ago |

If you're interested in playing with Hacker News data and don't want to download the entire dataset (or don't have the CPU/memory to perform large JOINs on stories/comments), you can use the Google BigQuery HN dataset, which is now up-to-date: https://cloud.google.com/bigquery/public-data/hacker-news (specifically, the .full table, which combines both stories and comments; the dedicated tables are not up-to-date)

venning 8 years ago | |

I see this link mentioned all the time, but every time I try it I can't get it to work.

Specifically, the "GO TO THE HACKER NEWS DATASET" big blue button on that page. It kicks me over to a Google Cloud console link, which spins for a few seconds, and then brings up a "Welcome to BigQuery!" modal. The only thing I can do then is click "Create a Project", which then kicks me over to the generic console with a listing of all APIs.

Am I missing something?

minimaxir 8 years ago | | |

You'll need to create a GCE project before you can use BigQuery (you don't need to provide a credit card if you remain in the free tier)

ers35 8 years ago |

See also: A dump of the stories, comments, and users from the Firebase API as a SQLite database with a full text search index: https://archive.org/details/hackernews-2017-05-18.db

B1FF_PSUVM 8 years ago | |

Can you tell about the time period and (estimate of) % of comments covered, for your DB and the dump posted?

Thanks.

ers35 8 years ago | | |

My DB covers from https://hacker-news.firebaseio.com/v0/item/1.json?print=pret... (10/09/2006 at 6:21pm UTC) to https://hacker-news.firebaseio.com/v0/item/14372035.json?pri... (05/18/2017 at 11:58pm UTC)

The dump posted claims to cover from 1 to 14566367 (06/16/2017 at 3:03am UTC)

natch 8 years ago |

Kudos to archive.org for hosting torrents. It would be helpful to know the size of the download up front. Nice clean web page design; would love to see that one bit of information added.

ers35 8 years ago | |

You can see the size of the download by clicking show all: https://archive.org/download/14566367HackerNewsCommentsAndSt...

natch 8 years ago | | |

Thanks. My point to the web designer stands though, the information should be on the first page. Before seeing your message, I looked and didn't find it; had to determine it by loading up the torrent in a client. Also as sillysaurus3 pointed out the expanded size is useful too.

fit2rule 8 years ago | | |

The answer: 1.6 gigs.