# Researchers propose closed-loop agent evaluation method for deployed AI systems

Digest AI · Research · published 2026-11-06T05:00:00Z

Canonical: https://digestai.news/story/researchers-propose-closed-loop-agent-evaluation-method-for-deployed-a

## Summary

A new paper on arXiv outlines a framework for continuously evaluating deployed AI agents, not just benchmarking them once. The method, called closed-loop evaluation, models performance under biased feedback and incomplete data. It breaks down analysis into four levels—span, trace, conversation, and engagement—and measures outcomes across four dimensions: Execution, Substance, Trajectory, and Results. Conduct is treated as a constraint throughout, with Provenance and Adjudication components ensuring evidence and evaluator reliability.

The approach uses evaluator agents generated by an *Agent Factory*, each equipped with domain knowledge and adapters. Two feedback loops—an inner loop for auditing evidence and rules, and an outer loop for regression checks—drive iterative improvement. The authors demonstrate the method with a business-analysis agent called *SharpInsight* and public conversation data, showing higher reliability in evidence-based scoring (ICC 0.86 vs. 0.81) compared to holistic methods.

## Key points

- Closed-loop evaluation continuously assesses deployed AI agents, not just via one-off benchmarks
- Four analysis levels (span, trace, conversation, engagement) measure Execution, Substance, Trajectory, and Results
- ICC score of 0.86 for evidence-based scoring vs. 0.81 for holistic methods, per the paper

## Why it matters

This method could improve trust in deployed AI agents by shifting from static benchmarks to real-time, self-calibrating evaluation, reducing false positives and guiding iterative improvements.

## Sources

1. [Closed-Loop Agent Evaluation: Layered Measurement and Self-Calibration](https://arxiv.org/abs/2611.07051) (arXiv cs.AI, 2026-11-06, primary source)

## Cite

Digest AI, "Researchers propose closed-loop agent evaluation method for deployed AI systems", 6 November 2026, https://digestai.news/story/researchers-propose-closed-loop-agent-evaluation-method-for-deployed-a

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/researchers-propose-closed-loop-agent-evaluation-method-for-deployed-a.json