GTA agent and GT Bench improve LLM graph reasoning accuracy
Researchers have introduced Graph Theory Bench (GT Bench), a comprehensive evaluation suite designed to test Large Language Models on complex graph reasoning tasks. The benchmark covers 24 classical graph problems across 44 different task-structure settings, utilizing over 100,000 examples presented in four distinct formats: natural language, structured language, adjacency lists, and adjacency…
Key points
- GT Bench evaluates LLMs on 24 graph problems using over 100,000 examples across four data representations.
- GTA agent improves Phi-4 accuracy from 53.5% to 69.1% on easy graph reasoning tasks.
- Model performance on graph tasks is highly sensitive to input representation format and graph topology.
The study evaluated eight different LLMs, revealing that model accuracy is highly sensitive to the input representation used. The optimal representation varies depending on graph density, size, topology, and the specific model being tested. Even the most advanced reasoning models showed this sensitivity, though it was somewhat attenuated compared to smaller models.
To address these challenges, the authors proposed the Graph Theory Agent (GTA). This system combines a preference-trained representation selector with plan-and-decompose scaffolding around a frozen executor LLM. GTA significantly improved performance, lifting the Phi-4 model from 53.5% to 69.1% on easy tasks and from 33.0% to 41.5% on hard tasks. The agent outperformed eight other prompting and agent baselines and demonstrated transferability to other benchmarks like GraCoRe and NLGraph without requiring retraining.
Coverage and discussion
2sources- Multi-Agent Agentic Graph Learning via Structural SignaturesPrimary source · arXiv cs.AI ·
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- PyTorch tutorial shows how to cut drone data usage by 94% · 1 src
- Article outlines five practices for building robust Python AI libraries · 1 src
- Vector databases explain how AI retrieves similar data at scale · 1 src
- PhAI Labs introduces JEPA-Anything framework for cross-domain world models · 2 src
- Researchers introduce VIMA for finding disease patterns in tissue data · 2 src
Comments
via GitHub Discussions