{"version":1,"type":"story","url":"https://digestai.news/story/researchers-propose-closed-loop-agent-evaluation-method-for-deployed-a","json":"https://digestai.news/story/researchers-propose-closed-loop-agent-evaluation-method-for-deployed-a.json","markdown":"https://digestai.news/story/researchers-propose-closed-loop-agent-evaluation-method-for-deployed-a.md","slug":"researchers-propose-closed-loop-agent-evaluation-method-for-deployed-a","headline":"Researchers propose closed-loop agent evaluation method for deployed AI systems","summary":"A new paper on arXiv outlines a framework for continuously evaluating deployed AI agents, not just benchmarking them once. The method, called closed-loop evaluation, models performance under biased feedback and incomplete data. It breaks down analysis into four levels—span, trace, conversation, and engagement—and measures outcomes across four dimensions: Execution, Substance, Trajectory, and Results. Conduct is treated as a constraint throughout, with Provenance and Adjudication components ensuring evidence and evaluator reliability.\n\nThe approach uses evaluator agents generated by an *Agent Factory*, each equipped with domain knowledge and adapters. Two feedback loops—an inner loop for auditing evidence and rules, and an outer loop for regression checks—drive iterative improvement. The authors demonstrate the method with a business-analysis agent called *SharpInsight* and public conversation data, showing higher reliability in evidence-based scoring (ICC 0.86 vs. 0.81) compared to holistic methods.","keyPoints":["Closed-loop evaluation continuously assesses deployed AI agents, not just via one-off benchmarks","Four analysis levels (span, trace, conversation, engagement) measure Execution, Substance, Trajectory, and Results","ICC score of 0.86 for evidence-based scoring vs. 0.81 for holistic methods, per the paper"],"whyItMatters":"This method could improve trust in deployed AI agents by shifting from static benchmarks to real-time, self-calibrating evaluation, reducing false positives and guiding iterative improvements.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":[],"models":["SharpInsight"],"people":[]},"firstPublishedAt":"2026-11-06T05:00:00Z","updatedAt":"2026-11-06T05:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"arXiv cs.AI","title":"Closed-Loop Agent Evaluation: Layered Measurement and Self-Calibration","url":"https://arxiv.org/abs/2611.07051","publishedAt":"2026-11-06T05:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Researchers propose closed-loop agent evaluation method for deployed AI systems\", 6 November 2026, https://digestai.news/story/researchers-propose-closed-loop-agent-evaluation-method-for-deployed-a","publisher":"Digest AI","title":"Researchers propose closed-loop agent evaluation method for deployed AI systems","datePublished":"2026-11-06T05:00:00Z","url":"https://digestai.news/story/researchers-propose-closed-loop-agent-evaluation-method-for-deployed-a"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}