arXiv study finds PtR outperforms RtP for multi-tool queries
A new paper on arXiv compares two approaches for tool-augmented AI agents: Retrieve-then-Plan (RtP) and Plan-then-Retrieve (PtR). The study uses ToolBench, a dataset with 46,984 unique API endpoints across 50 categories, to test both methods on single-tool and multi-tool tasks.
Key points
- PtR outperforms RtP on multi-tool queries but underperforms on single-tool tasks across all tested conditions
- Study tested both methods on ToolBench with 46,984 APIs and 50 categories, using models like Gemini 2.5 Flash
- PtR’s extra retrieval step doubles inference cost but may be worth it for complex multi-tool scenarios
Researchers found that RtP works better for simple, single-tool queries, while PtR excels on complex, multi-tool scenarios. PtR’s advantage persists across different retrieval budgets (from 1 to 20 results) and registry sizes (ranging from 1,000 to 47,000 APIs). The paper also notes that PtR’s extra discovery step roughly doubles inference costs, but this trade-off benefits multi-tool tasks. The authors highlight a failure mode where PtR’s generated tool stubs may hallucinate non-existent API names, which can hurt performance on simpler queries but help break down complex intents.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- OpenAI offers U.S. university labs free ChatGPT Business seats with Rosalind access · 2 src
- Anthropic reports small AI models may self-replicate on IoT devices within months · 29 src
- Researchers propose LLM-referenced C-test for evaluating supra-intelligence · 8 src
- Researchers study permutation diffusion LLMs for recommendation reranking · 1 src
- Researchers compare Non-Generative verifier to Astra LLM for RAG claim checks · 1 src
Comments
via GitHub Discussions