# arXiv study finds reading LLM judges from first token overstates position bias

Digest AI · Research · published 2026-10-02T04:00:00Z

Canonical: https://digestai.news/story/arxiv-study-finds-reading-llm-judges-from-first-token-overstates-posit

## Summary

A new paper on arXiv examines how reading an LLM judge’s verdict from its first generated token distorts results. Researchers found this method overstates position bias in every tested condition, as judges do not always start with a verdict token. For three Qwen3 judges, this happens in 12% to 49% of cases, while Llama-3.1-8B and Phi-3.5-mini show under 3% non-compliance. When forced to read early, the method flips 89.7% of undecided pairs when responses are swapped, compared to 47.5% after full generation.

The study highlights two key failures: first, the method misrepresents position bias by 42 points while barely affecting judge accuracy. Second, even when judges do start with a verdict, they sometimes begin with a partial letter before reasoning to the final judgment. The authors recommend reporting the rate at which judges lead with a verdict token alongside position-bias figures, as this requires only one forward pass and no labeled data.

## Key points

- Reading LLM judges from the first token overstates position bias in all tested conditions, per arXiv study
- Three Qwen3 judges fail to start with a verdict token in 12% to 49% of cases, while Llama-3.1-8B and Phi-3.5-mini show under 3%
- Forced early reading flips 89.7% of undecided pairs when responses are swapped, compared to 47.5% after full generation

## Why it matters

This finding challenges how AI labs evaluate model fairness, as position bias is a critical metric for judging LLM performance. Overstating bias could mislead developers and auditors into believing models are more neutral than they are.

## Sources

1. [The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating](https://arxiv.org/abs/2610.00054) (arXiv cs.CL, 2026-10-02, primary source)

## Cite

Digest AI, "arXiv study finds reading LLM judges from first token overstates position bias", 2 October 2026, https://digestai.news/story/arxiv-study-finds-reading-llm-judges-from-first-token-overstates-posit

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/arxiv-study-finds-reading-llm-judges-from-first-token-overstates-posit.json