ContractEval: Improves Procedural Instruction Conformance
As LLM agents move from answering questions to executing procedures, failures can be subtle. The article introduces ContractEval, a diagnostic framework that makes the active obligations in procedural instructions explicit. It matches query-active obligations against response or trace evidence to identify and localize structural failures such as omissions, wrong branches, ordering errors, extra…
1 source primary source
Key points
- Introduces ContractEval for procedural instruction conformance
- Matches query-active obligations against response or trace evidence
- Detects and localizes structural failures in procedures
Read the original at arXiv cs.AI · by Praphul Singh, Shanu Kumar, Akshat Agarwal, Ganesh Kumar primary sourceOpen source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Generative AI & Models
All →- Introducing Mistral Large 4 · 7 src
- OpenAI boosts GPT-6 speed by 50% in 28 Days update · 1 src
- Reflection AI debuts Beam, a 501B open-weight model rivaling GLM 5.2 with 3-4x less compute · 10 src
- Author tests four AI assistants on flawed forecasting scenarios · 1 src
- Anthropic prompts users to share voice data to enhance AI models · 8 src
Comments
via GitHub Discussions