Datalab releases OmniExtractBench to audit PDF extraction accuracy
Datalab launched OmniExtractBench, an open benchmark for structured document extraction from PDFs into JSON. It pools 620 documents from four existing datasets, including regulatory filings and synthetic tests, to measure how well systems fill JSON schemas. A deterministic scorer grades each value with one of six auditable verdicts—matched, misread, unfound, fabricated, invented item, or…
Key points
- OmniExtractBench pools 620 PDFs from four benchmarks, including 88 regulatory filings and 33 multi-page documents
- Scorer assigns six verdicts per value and uses content-based row pairing to avoid table-scoring errors
- Datalab’s accurate mode scores 93.85% accuracy; GPT-5.6-sol and LlamaExtract lag in precision or recall
The benchmark addresses flaws in current leaderboards, which Datalab claims favor vendors or obscure failures like broken harnesses. OmniExtractBench’s scorer uses content-based row pairing (via the Hungarian algorithm) to avoid penalizing tables for a single missed row. It also blocks score inflation by ignoring null fields, a tactic some systems exploit. Datalab tested 10 system configurations, with its own accurate mode leading at 93.85% accuracy, while others like GPT-5.6-sol and LlamaExtract struggled with unfound or fabricated values. The scorer is open-source under Apache 2.0 (PyPI) and the dataset under CC BY 4.0 (Hugging Face). Vendors must provide their own API keys and credits to rerun tests.
Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks
MarkTechPost · 2 October 2026Loading the full article…
This text was published by MarkTechPost and written by Asif Razzaq. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- MIT researchers show neural networks can build own learning rules · 1 src
- Anthropic research suggests small AI models may self-replicate on IoT devices · 6 src
- Anthropic co-founder predicts full RSI possible by 2027 · 1 src
- Sakana AI’s MASS method improves model self-judgment but risks baking errors into training · 1 src
- Microsoft Research finds clearing chat history boosts AI agent performance by 66% · 1 src
Comments
via GitHub Discussions