Datalab releases OmniParseBench, an open OCR benchmark with 16,288 tests
Datalab has released OmniParseBench, an open-source benchmark designed to evaluate Optical Character Recognition (OCR) parsers. The dataset consists of 16,288 pass/fail tests derived from 2,937 pages across 2,343 documents, covering 92 languages. Unlike previous benchmarks that rely on subjective metrics or single-bucket categorizations, OmniParseBench uses binary yes/no questions to measure…
Key points
- OmniParseBench contains 16,288 pass/fail tests across 92 languages and 2,937 pages.
- Gemini 3.8 Flash leads the launch leaderboard with a score of 92.9.
- Datalab’s parser ranks second at 92.7, acknowledging a conflict of interest.
The launch leaderboard shows Gemini 3.8 Flash leading with a score of 92.9, followed closely by Datalab’s own "accurate" parser at 92.7. Other notable performers include Claude Sonnet 5.5, which achieved the highest scores in text (93.1) and tables (95.4) but lagged in layout, and Reducto, which led in layout detection at 95.8. Datalab acknowledges a potential conflict of interest, noting that its own parser ranks second and that some tests were built with the help of models like Claude and Gemini. The tool is available under an Apache 2.0 license and can be installed via Python, allowing engineers to audit how well specific parsers handle their own document types.
Model page: Claude Sonnet 5.5 →
The story so far
2 episodes →- Datalab releases OmniParseBench, an open OCR benchmark with 16,288 teststhis story
Datalab Releases OmniParseBench: An Open OCR Benchmark With 16,288 Pass/Fail Tests to Audit OCR Parsers
MarkTechPost · 14 October 2026
Loading the full article…
This text was published by MarkTechPost and written by Asif Razzaq. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic reports small AI models may self-replicate on IoT devices within months · 23 src
- Meta engineer claims Anthropic's Scion copies Mononoke design · 1 src
- Colossal introduces GenoMorph AI to target dodo gene-editing sites · 1 src
- Leash Bio releases Minos, an AI playground for lab-in-the-loop simulations · 1 src
- Stanford economists argue AI should focus on human collaboration over automation · 1 src
Comments
via GitHub Discussions