SLATE benchmark tests long-horizon API selection in e-commerce automation
Researchers introduced SLATE, a new benchmark designed to evaluate how well AI agents can automate complex workflows across thousands of APIs in e-commerce settings. Unlike existing tests, SLATE focuses on long-horizon, multi-step procedures—mapping queries to the correct tools without predefined step-by-step instructions—and uses a deterministic simulator to judge task success based on…
Key points
- SLATE benchmark tests AI agents on long-horizon API selection across ~1,000 e-commerce tools without predefined mappings
- Success is measured by task completion, not subjective LLM scoring, and accepts diverse valid trajectories
- Six open decision models (3B–12B params) achieved only 5% success, highlighting the benchmark’s challenge
Experiments with six open decision models (ranging from 3B to 12B parameters) found that even the top-performing model achieved only a 5% success rate on SLATE’s tasks. The authors argue this demonstrates the benchmark’s difficulty and its potential to drive progress in reliable enterprise tool use. SLATE’s simulator, evaluation framework, and datasets are publicly available for further research.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Tencent releases WeVisOne-VL, a 2B/4B open vision model for detection, OCR and matting · 1 src
- Anthropic releases FlashProve open-source system for provable AI inference · 6 src
- Researchers propose longinus framework for Next-Visit diagnosis prediction · 1 src
- LaplaceDiag uses actuator effects for fault diagnosis · 1 src
- arXiv introduces AI agent for scalable drug discovery workflows · 1 src
Comments
via GitHub Discussions