Benchmarking the Most Reliable Document Parsing API

Benchmarking the Most Reliable Document Parsing API(tensorlake.ai)

28 points by calavera 195 days ago | 14 comments

serjester 195 days ago |

This is just a company advertisement, not even one that’s well done. They didn’t benchmark any of the real leaders in the space (reducto, extend, etc) and left Gemini out of the first two tests, presumably because it was the best performer (while also being multiple orders of magnitude cheaper).

diptanu 195 days ago | |

Hey! I am the founder of Tensorlake. We benchmarked the models that our customers consider using in enterprises or regulated industries where there is a big need for processing documents for various automation. Benchmarking takes a lot of time so we focussed on the ones that we get asked about.

On Gemini and other VLMs - we excluded these models because they don't do visual grounding - aka they don't provide page layouts, bounding boxes of elements on the pages. This is a table stakes feature for use-cases customers are building with Tensorlake. It wouldn't be possible to build citations without bounding boxes.

On pricing - we are probably the only company offer a pure on-demand pricing without any tiers. With Tensorlake, you can get back markdown from every page, summaries of figures, tables and charts, structured data, page classification, etc - in ONE api call. This means we are running a bunch of different models under the hood. If you add up the token count, and complexity of infrastructure to build a complex pipeline around Gemini, and other OCR/Layout detection model I bet the price you would end up with won't be any cheaper than what we provide :) Plus doing this at scale is very very complex - it requires building a lot of sophisticated infrastructure - another source of cost behind modern Document Ingestion services.

coderintherye 195 days ago | | |

Google's Vertex API for document processing absolutely does bounding boxes. In fact, some of the document processors are just a wrap around Google's product.

ianhawes 195 days ago | | |

I just tested a non-English document and it rendered English text. Does your model not support anything other than English?

hotpaper75 195 days ago | |

Thanks for mentioning them, indeed their post seem to only surface a couple of names in the field and maybe not the most relevant ones.

JLO64 195 days ago | |

Personally I use OpenAI models via the API for transcription of PDF files. Is there a big difference between them and Gemini models?

goldenjm 195 days ago |

This would be more helpful if it included DeepSeek-OCR, PaddleOCR-VL and MinerU 2.5. In general, I've found that OmniDocBench is a reliable benchmark, perhaps surprisingly because it is made by the same team as MinerU. They updated their benchmark table recently: https://github.com/opendatalab/OmniDocBench#end-to-end-evalu.... There are some other models that score above DeepSeek-OCR as well that I'm not as familiar with.

karakanb 195 days ago |

I have been recently looking into extracting a bunch of details from a set of legacy invoice PDFs and had a subpar experience. Gemini was the best among the ones that I tried, but even that missed quite a bit. I'll definitely give this a look.

It seems like such a crowded space and there are many tools doing document extraction, I wonder if there's anything particular pulling more attention into the space?

kissgyorgy 195 days ago |

I just tried it out and docling finished in 20s (with pretty good results) the same document which in Tensorlake is still pending for 10 minutes. I won't even wait for the results.

diptanu 195 days ago | |

There was an unusual traffic spike around that time, if you try now it should be a lot faster. We were calling up but there was not enough GPU capacity at that time.

recursive4 195 days ago |

Curious how it compares to https://github.com/datalab-to/chandra

diptanu 195 days ago | |

We haven’t tested Chandra yet, because it’s very new. Under the hood Tensorlake is very similar to Marker - it’s a pipeline based OCR API, we do layout detection, Text Recognition and Detection, Table Structure Understanding, etc. We then use VLMs to enrich the results. Our models are much bigger than marker, and thus takes a little longer to parse documents. We optimized for accuracy. We will have a faster API soon.