How to compare two articles.. hi... trying to figure out what ways are there to compare/determine if two separate articles are the same... curently researching semantic analysis, but figured i'd turn here as well... thoughts/comments... thanks bd |
How to compare two articles.. hi... trying to figure out what ways are there to compare/determine if two separate articles are the same... curently researching semantic analysis, but figured i'd turn here as well... thoughts/comments... thanks bd |
I read a book called Collective Intelligence by Tony Segaran. Its basically machine learning for dummies, very example heavy, all in Python.
He talks about clustering to group like things together in an unsupervised way. The way this works is to build a vector of words from each article and compare these using something known as pearson distance. The vector of words is known as a feature set. Early on you create this vector in a naive way (i.e. eliminate words that don't show up enough and words that show up too much). At the end of the book he talks about feature detection (which I assume is building this vector in a smarter way).
The book really helped me. Pearson correlation is pretty easy to grasp and implement as well.
Good luck.
this of course would/might take into account similar phrases/words, possibly similar titles, similar timeframe of creation, possible a priori knowledge about the author (past works), etc...
thanks