I made a ruby library for accessing hacker news - check it out!

I made a ruby library for accessing hacker news - check it out!(github.com)

10 points by random_guy 15 years ago | 3 comments

It might be fun to scrape and index all HN stories; and perhaps create a way to download links/comments to a local machine automatically . There is a lot of really great information provided via this place.

random_guy 15 years ago | |

The idea is interesting indeed - and easily doable as well: Entry.all gives you all the links on the front page, you'd just have fetch the main page between regular intervals of time (say, once every hour) and then save the entry information to a database.

Could you elaborate on this? If you can provide better "specifications" I'll implement this and add it to the library.

lukehasnoname 15 years ago | | |

I imagine this:

-DOWNLOAD HN pages locally OR to another site

This is to make a permanent copy of the site and its discussions locally, so you'll have the knowledge of the articles and discussion forever.

Either download the linked site itself, or just the link, folowed by comments. Automatically grab comment updates once a day for a week, or whenever discussion statistically tapers off.

In other words, here are the options:

-Submission titles, dates, and links to HN page

This is to provide a searchable database for HN links, and perhaps (if this were done on a separate website) you could implement a "favorites" system for people to login and bookmark those topics of discussion they liked most, ala Server Fault.

Downside: I don't know how long HN articles persist. If they disappear, this is largely useless.

-Linked sites themselves, URL to HN page, and comments

Much more permanent; a complete mirror of source sites and HN discussions, searchable. Similar login/favorites iface could be put in place.

Downside: More information is cached, more complicated scraping (source sites and HN).