# Researchers release VagueBench benchmark exposing ambient context illusion in agents

Digest AI · Research · published 2026-11-04T05:00:00Z

Canonical: https://digestai.news/story/researchers-release-vaguebench-benchmark-exposing-ambient-context-illu

## Summary

The paper introduces VagueBench, a new benchmark built on the AI2‑THOR simulation to test how embodied agents handle instructions that lack non‑essential contextual cues. Using a human‑agent co‑generation pipeline, the authors created 1,303 instruction pairs covering entities, procedures, constraints and prior knowledge, and verified that the core task semantics remain unchanged.

Six state‑of‑the‑art embodied agents were evaluated on VagueBench. Results show a relative 19.65% drop in task success rate and a 54.10% increase in empty‑plan rate, indicating strong dependence on auxiliary linguistic signals. The authors also demonstrate that the same ambient context illusion occurs in real‑world robotic systems, where missing contextual details can lead to incorrect decisions and potential safety risks.

The study highlights a previously overlooked robustness gap for agents deployed in open‑world settings, suggesting that future research must address context‑dependency to improve reliability outside controlled environments.

## Key points

- VagueBench contains 1,303 instruction pairs generated via a human‑agent co‑creation framework in AI2‑THOR
- Six leading embodied agents lose 19.65% task success and gain 54.10% empty‑plan rate under ambient context illusion
- The phenomenon also appears in real‑world robots, raising safety concerns

## Why it matters

It shows current embodied agents rely on unnecessary contextual language, limiting their robustness and safety when deployed in real‑world environments.

## Sources

1. [VagueBench: Benchmarking the Robustness of Embodied Agents on Ambient Context Illusions](https://arxiv.org/abs/2611.03765) (arXiv cs.AI, 2026-11-04, primary source)

## Cite

Digest AI, "Researchers release VagueBench benchmark exposing ambient context illusion in agents", 4 November 2026, https://digestai.news/story/researchers-release-vaguebench-benchmark-exposing-ambient-context-illu

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/researchers-release-vaguebench-benchmark-exposing-ambient-context-illu.json