Study Finds Deictic Ambiguity Can Undermine Draft‑Verify‑Revise LLM Pipelines
Researchers examined a common three‑stage LLM orchestration pattern—draft, verify, revise—and discovered that context‑dependent expressions like “previous” can shift meaning between stages, a problem they call a deictic shift. Using a synthetic dataset of ten base examples rendered in three conditions, they varied which stage resolved the expression correctly and how much independent reasoning…
Key points
- Deictic shifts cause “previous” to refer to different items across draft‑verify‑revise stages.
- GPT‑5.2’s accuracy rose from 0.156 to 0.942 with increased reasoning effort.
- Gemini 3 Pro maintained >0.94 accuracy, outperforming GPT‑5.2 at lower cost for 5% of trials.
Six models from three providers (including GPT‑5.2 and Gemini 3 Pro) were evaluated across 21 reasoning‑effort configurations. Balanced accuracy ranged from 0.156 (below chance) to near‑perfect, with GPT‑5.2 improving from 0.156 without reasoning to 0.942 at its highest effort, while Gemini 3 Pro stayed above 0.94 throughout. The study also found that when the meta‑evaluator erred, it relied on surface cues rather than deeper reasoning. The authors advise engineers to make referents explicit in each pipeline stage to avoid costly misinterpretations.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.
More in Research
All →- R2VC Boosts Fact‑Checking Accuracy by 13.74% on FEVER Using Modular Retrieval and Calibration · 1 src
- New GLARE model improves meeting continuation forecasting on MDFB benchmark · 1 src
- Chopthin-Consensus Power Sampling Boosts LLM Reasoning Accuracy Without Retraining · 1 src
- Context-Augmented KG Training Boosts Multi-Hop QA Accuracy on Disease Graphs · 1 src
- Thermodynamical Genetic Algorithm Boosts LLM Artifact Efficiency by 29% on Bin‑Packing · 1 src
Comments
via GitHub Discussions