Benchmarking LLMs for Key-Value Extraction in Noisy OCR Documents
Large language models (LLMs) are increasingly used to pull structured data from documents, but how they fare when the text is corrupted by optical‑character‑recognition (OCR) errors is unclear. A recent arXiv study evaluated five open‑source instruction‑tuned LLMs—Gemma, Mistral, Qwen2.5, LLaMA 3, and DeepSeek—on three document‑extraction benchmarks (FUNSD, CORD, SROIE). The models were tested…
Key points
- Open‑source LLMs Gemma, Mistral, Qwen2.5, LLaMA 3, DeepSeek evaluated on FUNSD, CORD, SROIE
- Performance drops sharply under OCR noise; gaps between models shrink as corruption increases
- Key failure modes include key‑value misalignment, hallucination, numeric corruption, highlighting need for better OCR and structural reasoning
The results show that the LLMs perform well when the input is clean, sometimes rivaling specialized layout‑aware systems. However, as OCR noise increases, performance drops sharply and the differences between models shrink. The study pinpoints key failure modes—misaligned key‑value pairs, hallucinations, and numeric corruption—and stresses that OCR quality, rather than model size, dominates extraction accuracy in realistic settings.
The story so far
2 episodes →- Benchmarking LLMs for Key-Value Extraction in Noisy OCR Documents this story
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.
More in Research
All →- Google Research Unveils R4T: Diffusion Retriever Cuts Query Fan‑Out Latency 12‑20× · 1 src
- Blindspot Benchmark Tests Long-Horizon Safety of Tool-Using LLM Agents · 1 src
- CLEAR framework improves medical LLM accuracy via cross-source evidence adjudication · 1 src
- AI-Driven Strategic Games Require Safety Cases · 1 src
- Legal LLMs' Hallucinations Should Be Evaluated as Warrant Failures · 1 src
Comments
via GitHub Discussions