Ask HN: Interested in a Wikipedia parsing tool for data mining?

5 points by ajeet 13 years ago | 6 comments

I recently developed a C library that parses Wikipedia, with the goal of being _fast_. For example, I was able to parse and extract text from the entire wikipedia dump (~35 GB uncompressed) in under an hour (on a 5yr old iMac).

It is a work in progress. If there is sufficient interest, then I will clean up the code and put up some documentation.

tarr11 13 years ago |

Have you looked at dbpedia.org? Perhaps your tools would fit in with their project.

ajeet 13 years ago | |

Yes I have, and that project would be a great fit. They already have a scala regex based lib though. My library is likely a faster and lighter alternative.

sareiodata 13 years ago |

Isn't there a complete database with the information in Wikipedia available?

nowarninglabel 13 years ago | |

Yes, but it does take quite some time to get that setup. I remember it took around two days on my server to get the data imported into MySQL. That said, thereafter, searching is a relatively solved problem, so I'd question the value of a C library, though I suppose it'd be useful in a case where you didn't/couldn't have put the data into a database or where you needed to parse new dumps all the time and didn't want to wait.

ajeet 13 years ago | | |

I am curious about your use case. Was it full text search? Did you get the database in wiki format, which you transformed to text?

The goal of my library is to enable quick data mining on wikipedia. Search is just one use case. As an example, you might want to build a content classifier to automatically categorize web pages into wikipedia categories (like politics, sports, etc). To go about doing this, you would need to parse wiki pages and extract features (like n-grams) for a particular category. The C library transforms plain wiki text to a parsed object, that you can use to extract what information you want. The only advantage is that it does this incredibly fast.