LHTB 2.0 benchmarks AI agents over three-hour tasks with 77 new scenarios
Researchers released Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0) on arXiv, expanding a test suite for AI agents that measure sustained progress over long workflows. The update adds 31 extended tasks—bringing the total to 77 across 13 domains—while fixing defects in the original 46. Tasks are generated from real-world artifacts using a parameterized pipeline, with three held out for evaluation and…
Key points
- LHTB 2.0 adds 31 extended tasks to its original 46, totaling 77 across 13 domains with three-hour evaluation windows
- GPT-6-astra scores highest with a dense mean reward of 0.69 and succeeds at 37 tasks with ≥0.95 threshold
- Dense ladder scores average 0.653, showing binary verifiers undercount progress by averaging 0.221 on reconstruction tasks
LHTB 2.0 scores 1,145 runs from 15 frontier models. GPT-6-astra leads with a dense mean reward of 0.69 and succeeds at 37 tasks with a score ≥0.95. GPT-5.6-sol follows with 0.67 and 32 tasks at the same threshold. Binary verifier scores average 0.221 on reconstruction tasks, while dense ladder scores average 0.653, highlighting how outcome-only metrics miss intermediate progress.
Model page: GPT-6 Astra →
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Asset learns to compress speech encoders · 1 src
- Researchers find Qwen3’s reasoning gains may stem from ‘thinking leakage’ · 1 src
- Researchers test AI agents improving LLM training strategies with up to 13.34% gains · 1 src
- Researchers model how AI and humans time new words in text · 1 src
- Researchers propose Next-Concept Modeling for faster text generation · 1 src
Comments
via GitHub Discussions