DeepInstructor agentic framework improves idea evaluation using experience graph from 58,607 peer reviews
Researchers introduced DeepInstructor, an agentic AI framework designed to evaluate scientific ideas by reasoning over structured scholarly experience. The system builds an Experience Graph from 58,607 peer reviews and uses a ReAct-based agent to retrieve evidence for traceable evaluation across dimensions like novelty and feasibility. It was tested on a new dataset called DeepInstruct, which…
1 source primary source
Key points
- DeepInstructor constructs an Experience Graph from 58,607 peer reviews
- It uses a ReAct-based agent to retrieve dimension-specific evidence for traceable evaluation
- Experiments show 24.4% improvement in Hit@1 and 29.7% in Hit@2 alignment with human judgments
Read the original at arXiv cs.CL · by Rongcan Pei, Fang Guo, Qinglin Qi, Qi Zhu, Yun Luo, Jianhao Yan, Minjun Zhu, Qiujie Xie, Dehong Zheng, Yue Zhang primary sourceOpen source ↗
Topics · follow one to build your own front page
DeepInstructor
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic research suggests small AI models may self-replicate on IoT devices · 7 src
- Researcher predicts AI engineering acceleration but not superintelligence · 1 src
- MIT researchers show neural networks can build own learning rules · 1 src
- Anthropic co-founder predicts full RSI possible by 2027 · 1 src
- Sakana AI’s MASS method improves model self-judgment but risks baking errors into training · 1 src
Comments
via GitHub Discussions