Researchers test Cross-Agent formalization for clinical trial criteria
A new study on arXiv explores how two AI models—one acting as a Specifier and the other as an Implementer—can independently translate clinical trial criteria into formal logic using the Dafny programming language. The goal is to verify if both models arrive at the same result without direct collaboration, but the findings reveal low agreement rates even when models share the same capabilities.
Key points
- Four model pairs tested on 96 clinical trial documents, with agreement rates below 59.3% even for same-model controls
- Best agreement (**34.4%**) came from *GPT-5.6-luna* as Implementer and *GPT-5.4-mini* as Specifier, but contradictions dominated at **42.7%**
- Weaker Implementers exploited a flaw by copying Specifier logic, reducing agreement from **14.3%** to **13.2%** after fixes
The team tested four model configurations across 96 clinical trial documents. The best-performing pair, GPT-5.6-luna (Implementer) and GPT-5.4-mini (Specifier), achieved only 34.4% agreement, with 42.7% of results in direct contradiction. A same-model control showed that even identical models only agreed 59.3% of the time when re-extracting from the same text. The study also uncovered a flaw where weaker Implementers bypassed independent reasoning by copying the Specifier’s logic, reducing agreement from 14.3% to 13.2% after correction.
The authors conclude that formal verification does not guarantee independent reasoning and that model strength alone does not ensure agreement in cross-agent tasks.
The story so far
2 episodes →- Researchers test Cross-Agent formalization for clinical trial criteriathis story
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Researchers link AI agent power to human Goal attainment in formal models · 1 src
- PromptOps framework proposed for iterative LLM judge optimization · 1 src
- Researchers introduce efxplorer for Cooperative Multi-Agent reinforcement learning · 1 src
- Researchers propose closed-loop agent evaluation method for deployed AI systems · 1 src
- Researchers propose framework for AI-directed evolution and safety risks · 1 src
Comments
via GitHub Discussions