ArXiv introduces realcli benchmark to evaluate human‑agent collaboration for CLI agents
A new arXiv paper presents REALCLI, a paired benchmark designed to assess both task execution and the quality of human‑agent collaboration on realistic command‑line interface (CLI) tasks. The framework defines five dimensions, broken down into 22 sub‑dimensions and 51 interaction scenario templates, allowing systematic measurement of how agents work with users.
Key points
- REALCLI benchmark evaluates both task execution and human‑agent collaboration on CLI tasks.
- Two modes: REALCLI‑EXEC (full specs up front) and REALCLI‑COLLAB (progressive user input).
- Top frontier agent scores 49.2% overall and 37.6% joint collaboration, showing major capability gaps.
REALCLI offers two evaluation modes. REALCLI‑EXEC supplies the full task specification at the start, while REALCLI‑COLLAB uses a scripted user that provides information, corrections, constraint changes, and authorization decisions progressively during execution. When frontier CLI agents were tested, the best configuration achieved an average overall score of only 49.2%, and a joint collaboration score of just 37.6%, revealing a sharp gap between task effectiveness and collaborative performance. The authors argue that current agents struggle with steerability, trust calibration, and safety boundary management, and they position REALCLI as a reproducible foundation for advancing collaborative CLI agents.
The benchmark aims to give researchers a concrete tool for measuring and improving interactive capabilities, which are increasingly critical as CLI agents take on longer‑horizon, user‑driven tasks.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic says small AI may self‑replicate by year‑end after experiment with GLM‑5.2 · 1 src
- Anthropic co-founder Jack Clark says AI could boost US GDP by nearly 50% by 2030 · 13 src
- Hybrid Reasoning Agents improve LLM performance by 10.4% and reduce costs up to 3.7x · 1 src
- Chinese team proves 3D Ghys conjecture using Intern‑S2 AI model · 2 src
- StarSkirmish Bench v0.2 ranks GPT-6 Astra as top StarCraft bot writer · 2 src
Comments
via GitHub Discussions