Show HN: TextRazor, a scriptable text mining API

adelevie 13 years ago |

This looks great. I'm building http://dokket.aws.af.cm. It's a database filled with documents from the Federal Communications Commission. From day 1, I've been looking for smart ways to make use of the thousands of documents of unstructured text. The customization you offer seems to be the killer feature for me.

I'll write a Ruby api wrapper if you give me an agreed-upon amount of usage when you settle on pricing.

Feel free to email me (HN name @ gmail) if you're interested or just want to follow up for customer development purposes.

Best of luck!

richardofyork 13 years ago | |

I am also building an application that could use this service. So I tested TextRazor and the results were not good on two random articles I processed with the service.

I really want this to work out, because my application needs this kind of technology to be reliable and accurate. I just processed the article at the link below, and the word "Tesla" was not captured as one of the topics. http://www.teslamotors.com/blog/most-peculiar-test-drive-fol...

jsmcgd 13 years ago | | |

It works for me.

richardofyork 13 years ago | | |

I just tried it again and it worked this time. That is interesting: I don't know if they tweaked it a bit or not :), but I am feeling a bit better now, because my application really needs a reliable, accurate service like this.

tcwc 13 years ago | | |

No tweaking, promise :) It's possible you hit an inconsistent server first time around, I'll have a dig on our end. Let me know if you have any other problems - toby@textrazor.com.

steeve 13 years ago |

Cool demo, but how do you guys compare to, say, Alchemy API ?

Also, no pricing => not cool.

tcwc 13 years ago | |

Hey steeve we thought there were a few things missing in the competition. We've built a bunch of extra functionality such as more extensive relation and dependency parsing and contextual entailment generation, and use all that to build much more accurate entity and topic recognition, an area we think the others can be greatly improved on.

We also expose all these results to a Prolog interpreter on our backend and allow you to add custom logic to mashup and extend all of our results, as well as provide a much easier integration experience.

Totally agree with you on the pricing front, we're still finalising the details there. We're aiming to be fully transparent with both the technical and business side of things.

ismaelc 13 years ago | | |

You can add billing/pricing tiers to your API now using Mashape http://www.mashape.com/ (Disclaimer: I work for Mashape. Let me know if you need help!)

PaulHoule 13 years ago | |

Recall is perhaps 300% better than Alchemy API with much better precision.

anilshanbhag 13 years ago |

This is just like stanford parser. http://nlp.stanford.edu:8080/parser/

Why use TextRazor and pay for it ?

mark_l_watson 13 years ago | |

The Stanford NLP tools are very good, and also GPLed, which works for a lot of projects. If the GPL doesn't work for you, the Apache OpenNLP project is also good.

mark_l_watson 13 years ago | | |

BTW, it is not just having software packages to use: it is a ton of work obtaining and preparing training data. That said, Stanford NLP and OpenNLP tools come out of the box with trained models for tagging, entity name recognition, etc. For lots of uses, these pre-trained models will work well for you.

tcwc 13 years ago | |

The Stanford parser is great, but isn't really the same. The Stanford entity recogniser is limited to the standard types of people, places, companies, but we identify and disambiguate into a far richer ontology from wikipedia, and can recognize topic abstractions that aren't explicitly mentioned.

Also we found the Stanford tools (and the other open source NLP tools) were difficult to integrate into "production" apps for various reasons. One big one was performance - we aim to run the full parsing and extraction pipeline on an average news story in a few hundred milliseconds, which can be an order of magnitude faster than the others.

JPKab 13 years ago | | |

How does your offering compare to Calais from Thomson-Reuters?

Edit: To be specific, it looks very similar. What do you have that Calais doesn't?

mark_l_watson 13 years ago | | |

I have been using the free tier (50K API calls per day) of Open Calais for years and have also used it in code examples in three books I have written.

One thing that Open Calais does that I really like is that they attempt to have a single URI uniquely identifying recognized named entities. This is useful because, for example, when it recognizes President Bill Clinton, you get a reference to a unique URI, even if his name, title is different in different processed texts.

Thomson-Reuters bought ClearForest several years ago, thus acquiring Calais. If you are interested in text mining, and if you haven't experimented with Open Calais, then please put that on your TODO list.

gsharma 13 years ago | |

Just tried http://nlp.stanford.edu:8080/parser/ it only allows up to 70 characters for parsing.

mark_l_watson 13 years ago | | |

Try downloading the distribution of code and data and run it locally. Java stuff, maven based, easy to run. Use the example code listed on the installation web page to see how to set it up.

gsharma 13 years ago | | |

Ahh, thanks. I didn't realize it was just a demo. Sounds like a fun weekend project.

thejosh 13 years ago |

No pricing at all on the website is fishy for anyone who wants to use the free plan?

mark_l_watson 13 years ago | |

I offer something similar, and I do have a pricing page: http://kbsportal.com/pricing :-)

tcwc 13 years ago | |

I can understand why you might think that, but we'll always have the free plan. We're still working out all the little pricing details.

abaer 13 years ago |

Really impressive results. I put in some music reviews, and it did an excellent job of identifying artists, Genres, labels, etc. From an API perspective,there's not much out there that competes with this - and nothing with a modern API.

typpo 13 years ago | |

There are a number of other competing APIs. One of the best is OpenCalais, which is owned by Thomson Reuters[1].

You can demo it with some text here: http://viewer.opencalais.com/

[1] http://www.opencalais.com

polskibus 13 years ago |

That's exactly what I need to start working again on my algo trader! Seems to be working well with a sample of financial news extracts. Will definitely look into it further, thanks!

movingahead 13 years ago |

This is very impressive stuff. I ran a news article through the demo, and the entity recognition was very impressive. Waiting for them to reveal more details on pricing.

habosa 13 years ago |

Wow, this seems incredible, signed up immediately. My mind is already spinning with all of the cool apps I could make with this. How hard would it be to allow this functionality offline for paid users? You could have some sort of packaged library which phones home to count requests used, but does the processing offline to take network latency out of the equation. Not sure if that's feasible, but it would be great.

ses 13 years ago |

Being a big Prolog fan, I think this looks like an awesome product. This sort of textual analysis will become more important as time goes on. As the interest in search technologies grows, I think intelligent search (contextual queries, query answering, clustering, recommendations, meta-data extraction etc.) will start to appear in more end-user products. One question... whereabouts in the UK are you based?

tcwc 13 years ago | |

Thanks, great to see the other Prolog fans coming out of the woodwork! Based in London.

petercooper 13 years ago |

Question: What are people actually doing with technology like this right now? (i.e. who are the people who see this and think.. yay, I'll sign up now!)

JPKab 13 years ago | |

It's certainly not going to be ideal for your typical CRUD app. Think about all of the information that is locked inside of unstructured text (MS word docs, pdf's come to mind), and then imagine if you can scan through thousands of documents, find the named entities, and then start connecting them together in queries.

Obvious uses would be any kind of CMS. Investigative journalism is another.

adelevie 13 years ago | |

Me. See my other comment in this thread.

I've recently started exploring the Legal Informatics field. The problems in it are huge and typically involving adding some structure to lots and lots of unstructured text.

Also, Peter, if you do end up reading this, great work on the stuff you do :) Big fan here!

Mikushi 13 years ago | |

Extracting important keywords from thousands of pages for example. On our website we let user enter quite a lot of content, and being able to extract keywords and find patterns between user could be key in naming categories and such.

foobar2k 13 years ago | |

finance

atrilla 13 years ago |

Great work! But I missed the "sentiment analysis" flavour that used to be so popular some years ago with the NLP bunch... In this sense, I did something similar:

http://dtminredis.housing.salle.url.edu:8080/EmoLib/

and

http://nlptools.atrilla.net/web/omsa.php

Drop me a line if I can be of any help!

GotAnyMegadeth 13 years ago |

This looks amazing. Though when I put my name in it came up with: DBPedia types: Person Athlete Agent GolfPlayer owl#Thing

doktrin 13 years ago |

Very cool. This is essentially what I envisioned when I started work on http://www.textalyze.com.

Haven't devoted much time to it over the last couple months, but this provides a lot of inspiration.

wiradikusuma 13 years ago | |

Hi man, I tried my website (http://www.ngajakjalan.com) but the analyzer probably reads my inlined JS and doesn't read the "done" version (I use AngularJS). Maybe you can use something like PhantomJS to extract websites content "as seen by human"?

thehodge 13 years ago |

I don't know if your just being slammed right now but I posted a request for pricing and haven't got anything.. it looks like a nice API to integrate into our system but it really needs clearer pricing..

jsmcgd 13 years ago |

Awesome stuff. It would be great to use something like this for suggesting relevant tags of content. Perhaps a WordPress or Drupal plugin to get the ball rolling.

PaulHoule 13 years ago |

This is the best Wikipedia-backed namex I've seen and I've seen a lot of them, worked on code for one, and even designed one. Awesome!

beebs93 13 years ago |

Wow, nice demo. I agree with the others the lack of pricing is a bit disconcerting.

Btw, anyone else read the title as "Trent Reznor"? I need a coffee...

snake_plissken 13 years ago | |

totes on the Trent Reznor part.

sinzone 13 years ago |

Hi, would love to have this API listed on Mashape

eurodance 13 years ago |

Well done.