ALTK-Evolve tops τ²-bench retail leaderboard using GPT-4.1 with learned guidelines
Researchers from AgentToolkit posted results showing that a basic ReAct agent powered by GPT-4.1 reaches 67.5% Pass^4 on the τ²-bench retail domain, topping the October 12, 2026 leaderboard snapshot. The method, called ALTK-Evolve, extracts behavioral guidelines from the agent's own training-task attempts using the same GPT-4.1 model, then appends those guidelines to the system prompt for test…
Key points
- GPT-4.1 agent with learned guidelines scores 67.5% Pass^4 on τ²-bench retail, leading the Oct 12, 2026 leaderboard
- Guidelines are extracted automatically from training runs using the same model, no weight updates or human labels
- Method also lifts GPT-5.5 airline Pass^4 from 55% to 70% on held-out tasks
On held-out retail tasks, Mean Accuracy rises from 78.1% to 82.5% and Pass^4 from 57.5% to 67.5% with the full lossless guideline set (292 guidelines). On airline tasks, GPT-4.1 gains +7.5 accuracy points and +10 Pass^4 points. A compact high-support core (sup3, 14 guidelines) works best for airline reliability. The same approach also improves a GPT-5.5 agent on airline: accuracy +10 points to 82.5% and Pass^4 +15 points to 70.0%. Input token cost per conversation increases from ~$0.084 to ~$0.139 for retail, but conversations become slightly shorter and latency drops. The authors say prompt caching can reclaim most of the added cost.
The story so far
3 episodes →- ALTK-Evolve tops τ²-bench retail leaderboard using GPT-4.1 with learned guidelinesthis story
ALTK-Evolve: How learned memory tops the τ2-bench leaderboard on customer service tasks without a frontier model
Hugging Face · 20 October 2026
Loading the full article…
This text was published by Hugging Face. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
2sources- IBM Research Reports Learned Memory Tops τ2-Bench Service BenchmarkPress · Unite.AI ·
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Ai2 releases FairStress benchmark to evaluate demographic fairness in LLM decisions · 1 src
- Google and Meta release TorchTPU for native PyTorch on TPU · 2 src
- Stanford PhD uses generative AI to draft genetic blueprints for microscopic viruses · 1 src
- OpenAI claims AI-generated proof of quasi‑Riemann hypothesis · 1 src
- Delft University’s MonoRace drone outpaces top human pilots in race · 1 src
Comments
via GitHub Discussions