Microsoft and CMU host workshop on AI agent evaluation measurement science

The agenda separates evaluation into "horizontal" challenges that apply across agents and "vertical" use‑case‑specific criteria such as healthcare, software development and security. Breakout sessions will address long‑running tasks, multi‑agent collaboration, safety, red‑team testing, and how humans can intervene, understand, and recover from failures. Microsoft Research says the meeting will culminate in public documentation that records unresolved issues, points of consensus and gaps in current methods.

The discussion emphasizes that a single success‑rate score is insufficient; instead, the process, human burden, and controllability must be measured. Ultimately, humans will set the passing line for each application, using the evaluation results as material for responsible decision‑making.