SAGE system raises grant review agreement to kappa 0.58, beating baseline
Researchers introduced SAGE, a schema‑guided LLM that converts grant rubrics into structured checks and links each judgement to evidence in the application package. In an initial evaluation on 35 nonprofit grant applications, SAGE’s assessments were compared with 105 original competition reviews, yielding a fair ordinal agreement of kappa = 0.29.
Key points
- SAGE evaluated on 35 nonprofit grant applications, achieving kappa 0.29 in initial test.
- In a re‑review of 202 assessments, SAGE reached kappa 0.58, surpassing the baseline kappa 0.33.
- The system links rubric criteria to evidence, producing auditable drafts for expert correction.
After the foundation inspected SAGE’s output, it conducted a criterion‑level re‑review that produced 202 assessments. In this assisted round, SAGE improved to kappa = 0.58 and outperformed a one‑prompt‑per‑criterion baseline, which recorded kappa = 0.33 on the same subset. The system also showed higher rank correlation and lower error rates. A claim‑level audit identified which parts of the structured draft were confirmed, disputed, or left unaddressed, demonstrating SAGE’s ability to generate detailed, evidence‑linked, and auditable drafts for expert correction.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.
More in Research
All →- TatBLiMP benchmarks Tatar linguistic minimal pairs · 1 src
- Researchers propose AI-GRACE framework for operationalizing agentic AI use cases · 1 src
- Embed‑TTT improves rule induction in ARC‑like tasks, authors say · 1 src
- Researchers introduce SpecOpt for agentic molecule specificity optimization · 1 src
- researchers report agent-based hls with rtl refinement speeds chip design 2.6× · 1 src
Comments
via GitHub Discussions