# ArXiv introduces realcli benchmark to evaluate human‑agent collaboration for CLI agents

Digest AI · Research · published 2026-10-19T04:00:00Z

Canonical: https://digestai.news/story/arxiv-introduces-realcli-benchmark-to-evaluate-humanagent-collaboratio

## Summary

A new arXiv paper presents REALCLI, a paired benchmark designed to assess both task execution and the quality of human‑agent collaboration on realistic command‑line interface (CLI) tasks. The framework defines five dimensions, broken down into 22 sub‑dimensions and 51 interaction scenario templates, allowing systematic measurement of how agents work with users.

REALCLI offers two evaluation modes. REALCLI‑EXEC supplies the full task specification at the start, while REALCLI‑COLLAB uses a scripted user that provides information, corrections, constraint changes, and authorization decisions progressively during execution. When frontier CLI agents were tested, the best configuration achieved an average overall score of only 49.2%, and a joint collaboration score of just 37.6%, revealing a sharp gap between task effectiveness and collaborative performance. The authors argue that current agents struggle with steerability, trust calibration, and safety boundary management, and they position REALCLI as a reproducible foundation for advancing collaborative CLI agents.

The benchmark aims to give researchers a concrete tool for measuring and improving interactive capabilities, which are increasingly critical as CLI agents take on longer‑horizon, user‑driven tasks.

## Key points

- REALCLI benchmark evaluates both task execution and human‑agent collaboration on CLI tasks.
- Two modes: REALCLI‑EXEC (full specs up front) and REALCLI‑COLLAB (progressive user input).
- Top frontier agent scores 49.2% overall and 37.6% joint collaboration, showing major capability gaps.

## Why it matters

The benchmark exposes that current CLI agents excel at completing tasks but falter in interactive collaboration, highlighting risks in steerability, trust and safety, and giving researchers a tool to drive improvements.

## Sources

1. [RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents](https://arxiv.org/abs/2610.22507) (arXiv cs.CL, 2026-10-19, primary source)

## Cite

Digest AI, "ArXiv introduces realcli benchmark to evaluate human‑agent collaboration for CLI agents", 19 October 2026, https://digestai.news/story/arxiv-introduces-realcli-benchmark-to-evaluate-humanagent-collaboratio

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/arxiv-introduces-realcli-benchmark-to-evaluate-humanagent-collaboratio.json