Ask HN: Is python NLTK library still used for word tokenization? |
Ask HN: Is python NLTK library still used for word tokenization? |
Our pipeline uses NLTK to take in a string of text, do word tokenization, lemmatization and stemming, and construct bigrams and trigrams, as part of a large map-reduce job for building text search indices.