Researchers release VagueBench benchmark exposing ambient context illusion in agents
The paper introduces VagueBench, a new benchmark built on the AI2‑THOR simulation to test how embodied agents handle instructions that lack non‑essential contextual cues. Using a human‑agent co‑generation pipeline, the authors created 1,303 instruction pairs covering entities, procedures, constraints and prior knowledge, and verified that the core task semantics remain unchanged.
Key points
- VagueBench contains 1,303 instruction pairs generated via a human‑agent co‑creation framework in AI2‑THOR
- Six leading embodied agents lose 19.65% task success and gain 54.10% empty‑plan rate under ambient context illusion
- The phenomenon also appears in real‑world robots, raising safety concerns
Six state‑of‑the‑art embodied agents were evaluated on VagueBench. Results show a relative 19.65% drop in task success rate and a 54.10% increase in empty‑plan rate, indicating strong dependence on auxiliary linguistic signals. The authors also demonstrate that the same ambient context illusion occurs in real‑world robotic systems, where missing contextual details can lead to incorrect decisions and potential safety risks.
The study highlights a previously overlooked robustness gap for agents deployed in open‑world settings, suggesting that future research must address context‑dependency to improve reliability outside controlled environments.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic releases FlashProve open-source system for provable AI inference · 7 src
- Gemini and Claude debate AI personality, ending in a three-way tie · 1 src
- Tencent releases WeVisOne-VL, a 2B/4B open vision model for detection, OCR and matting · 1 src
- Lama-dt outperforms baselines on meta-world and mujoco · 1 src
- Researchers release Dory1.0-4B, a 4B-parameter looped model with 256K-token context · 1 src
Comments
via GitHub Discussions