{"version":1,"type":"story","url":"https://digestai.news/story/arxiv-introduces-realcli-benchmark-to-evaluate-humanagent-collaboratio","json":"https://digestai.news/story/arxiv-introduces-realcli-benchmark-to-evaluate-humanagent-collaboratio.json","markdown":"https://digestai.news/story/arxiv-introduces-realcli-benchmark-to-evaluate-humanagent-collaboratio.md","slug":"arxiv-introduces-realcli-benchmark-to-evaluate-humanagent-collaboratio","headline":"ArXiv introduces realcli benchmark to evaluate human‑agent collaboration for CLI agents","summary":"A new arXiv paper presents REALCLI, a paired benchmark designed to assess both task execution and the quality of human‑agent collaboration on realistic command‑line interface (CLI) tasks. The framework defines five dimensions, broken down into 22 sub‑dimensions and 51 interaction scenario templates, allowing systematic measurement of how agents work with users.\n\nREALCLI offers two evaluation modes. REALCLI‑EXEC supplies the full task specification at the start, while REALCLI‑COLLAB uses a scripted user that provides information, corrections, constraint changes, and authorization decisions progressively during execution. When frontier CLI agents were tested, the best configuration achieved an average overall score of only 49.2%, and a joint collaboration score of just 37.6%, revealing a sharp gap between task effectiveness and collaborative performance. The authors argue that current agents struggle with steerability, trust calibration, and safety boundary management, and they position REALCLI as a reproducible foundation for advancing collaborative CLI agents.\n\nThe benchmark aims to give researchers a concrete tool for measuring and improving interactive capabilities, which are increasingly critical as CLI agents take on longer‑horizon, user‑driven tasks.","keyPoints":["REALCLI benchmark evaluates both task execution and human‑agent collaboration on CLI tasks.","Two modes: REALCLI‑EXEC (full specs up front) and REALCLI‑COLLAB (progressive user input).","Top frontier agent scores 49.2% overall and 37.6% joint collaboration, showing major capability gaps."],"whyItMatters":"The benchmark exposes that current CLI agents excel at completing tasks but falter in interactive collaboration, highlighting risks in steerability, trust and safety, and giving researchers a tool to drive improvements.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":[],"models":[],"people":[]},"firstPublishedAt":"2026-10-19T04:00:00Z","updatedAt":"2026-10-19T04:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"arXiv cs.CL","title":"RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents","url":"https://arxiv.org/abs/2610.22507","publishedAt":"2026-10-19T04:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"ArXiv introduces realcli benchmark to evaluate human‑agent collaboration for CLI agents\", 19 October 2026, https://digestai.news/story/arxiv-introduces-realcli-benchmark-to-evaluate-humanagent-collaboratio","publisher":"Digest AI","title":"ArXiv introduces realcli benchmark to evaluate human‑agent collaboration for CLI agents","datePublished":"2026-10-19T04:00:00Z","url":"https://digestai.news/story/arxiv-introduces-realcli-benchmark-to-evaluate-humanagent-collaboratio"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}