Tell HN: Hacker News BigQuery dataset is updated daily again

51 points by wietsevenema 2 years ago | 11 comments

As you might know, there's a Hacker News BigQuery dataset [1].

I recently noticed that the export was broken (since Nov 2022) and I worked with the team to help fix it. It'll now again run daily.

SELECT timestamp FROM `bigquery-public-data.hacker_news.full` ORDER BY id DESC LIMIT 1 # 2024-06-09 09:12:55 UTC

[1]: https://console.cloud.google.com/marketplace/product/y-combinator/hacker-news

zX41ZdbW 2 years ago |

Daily is too infrequently.

I have this dataset being updated in ClickHouse in real time: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...

I also provide a way to export it or attach it to clickhouse-local and analyze it locally: https://github.com/ClickHouse/ClickHouse/issues/29693#issuec...

tarasglek 2 years ago | |

I love that this dataset is accessible via cors from web frontends. Got me into experiments with clickhouse

Same with github, much better than than throttled apis

dang 2 years ago |

Thanks! I tried emailing my contacts on that team several times and never heard back, so I figured we no longer had contacts on the that team.

izyda 2 years ago | |

Any interest in getting this dataset on Snowflake too?

(I run a Snowflake funded data provider on Snowflake marketplace)

dang 2 years ago | | |

The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API), so anyone is free to do this.

throwaway211 2 years ago |

How big is it?

wietsevenema 2 years ago | |

Closing in on 41 million items