{"version":1,"type":"story","url":"https://digestai.news/story/datalab-releases-omniparsebench-an-open-ocr-benchmark-with-16-288-test","json":"https://digestai.news/story/datalab-releases-omniparsebench-an-open-ocr-benchmark-with-16-288-test.json","markdown":"https://digestai.news/story/datalab-releases-omniparsebench-an-open-ocr-benchmark-with-16-288-test.md","slug":"datalab-releases-omniparsebench-an-open-ocr-benchmark-with-16-288-test","headline":"Datalab releases OmniParseBench, an open OCR benchmark with 16,288 tests","summary":"Datalab has released OmniParseBench, an open-source benchmark designed to evaluate Optical Character Recognition (OCR) parsers. The dataset consists of 16,288 pass/fail tests derived from 2,937 pages across 2,343 documents, covering 92 languages. Unlike previous benchmarks that rely on subjective metrics or single-bucket categorizations, OmniParseBench uses binary yes/no questions to measure accuracy in text extraction, table structure, and layout detection. The scoring is deterministic and runs offline without requiring API keys, though running vendor-specific parsers requires their respective credentials.\n\nThe launch leaderboard shows Gemini 3.8 Flash leading with a score of 92.9, followed closely by Datalab’s own \"accurate\" parser at 92.7. Other notable performers include Claude Sonnet 5.5, which achieved the highest scores in text (93.1) and tables (95.4) but lagged in layout, and Reducto, which led in layout detection at 95.8. Datalab acknowledges a potential conflict of interest, noting that its own parser ranks second and that some tests were built with the help of models like Claude and Gemini. The tool is available under an Apache 2.0 license and can be installed via Python, allowing engineers to audit how well specific parsers handle their own document types.","keyPoints":["OmniParseBench contains 16,288 pass/fail tests across 92 languages and 2,937 pages.","Gemini 3.8 Flash leads the launch leaderboard with a score of 92.9.","Datalab’s parser ranks second at 92.7, acknowledging a conflict of interest."],"whyItMatters":"This benchmark provides a standardized, auditable way for engineers to compare OCR parsers. It moves beyond vague leaderboards to specific pass/fail metrics, helping teams select the right tool for diverse document types and languages without relying on vendor claims.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":["Datalab","LlamaIndex","Marktechpost"],"models":["Gemini 3.8 Flash","Claude Sonnet 5.5","Reducto"],"people":["Asif Razzaq"]},"firstPublishedAt":"2026-10-14T23:37:25Z","updatedAt":"2026-10-14T23:37:25Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"MarkTechPost","title":"Datalab Releases OmniParseBench: An Open OCR Benchmark With 16,288 Pass/Fail Tests to Audit OCR Parsers","url":"https://marktechpost.com/2026/10/14/datalab-omniparsebench-open-ocr-document-parsing-benchmark","publishedAt":"2026-10-14T23:37:25Z","type":"press","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":{"title":"Datalab Launches New AI Document Benchmarks","url":"https://digestai.news/thread/datalab-releases-omniextractbench-to-audit-pdf-extraction-accuracy","storyCount":2},"cite":{"text":"Digest AI, \"Datalab releases OmniParseBench, an open OCR benchmark with 16,288 tests\", 14 October 2026, https://digestai.news/story/datalab-releases-omniparsebench-an-open-ocr-benchmark-with-16-288-test","publisher":"Digest AI","title":"Datalab releases OmniParseBench, an open OCR benchmark with 16,288 tests","datePublished":"2026-10-14T23:37:25Z","url":"https://digestai.news/story/datalab-releases-omniparsebench-an-open-ocr-benchmark-with-16-288-test"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}