Taste-Bench shows top AI agents barely exceed 60% accuracy
Researchers from Microsoft and City University of Hong Kong introduced Taste-Bench, a 502‑question benchmark that evaluates AI agents' ability to make good long‑term decisions during multi‑step tasks. The test presents agents with a decision fork—two divergent paths from a shared trajectory—and asks a model to pick the better one, counting a question only if the answer is correct both ways.
Key points
- Taste-Bench contains 502 binary-choice questions evaluating agents' long-term decision quality.
- GPT-5.6 Sol achieved the highest score at 59.7 percent, only slightly above GPT-5.5’s 59.5 percent.
- Teaching a student model (Qwen3.6‑27B) raised accuracy from 30 % to 47.9 % and doubled SWE‑bench Pro success to 33.7 %.
Fourteen models from the Claude, GPT, Grok, DeepSeek, GLM, MiniMax and Mistral families were evaluated. The best performer, GPT‑5.6 Sol, averaged 59.7 percent, closely followed by GPT‑5.5 at 59.5 percent, while pure guessing would yield 25 percent. Accuracy rose to 62.3 percent when the decisive clue appeared early in the trace, but fell to 21 percent when the clue emerged only after extensive later work. A student model distilled from Qwen3.6‑27B improved its own fork‑selection accuracy from 30 percent to 47.9 percent, and when its judgments were added as advice to an agent, success on 41 held‑out SWE‑bench Pro tasks increased from 14.6 percent to 33.7 percent. The benchmark and code are publicly released.
Model page: GPT-5.6 Sol →
The story so far
2 episodes →- Taste-Bench shows top AI agents barely exceed 60% accuracythis story
AI agents have bad taste in long-term decisions and a new benchmark proves it
The Decoder · 18 October 2026
Loading the full article…
This text was published by The Decoder and written by Jonathan Kemper. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic co-founder Jack Clark says AI could boost US GDP by nearly 50% by 2030 · 9 src
- DeepSeek Solves AI Memory Problem with Compression · 1 src
- OpenAI releases 722 AI-generated math papers covering 372 open problems · 2 src
- Analog Devices outlines challenges for physical AI in new report · 1 src
- Study finds Google Gemini 2.5 Flash Image complies with requests to make faces look gay · 1 src
Comments
via GitHub Discussions