OpenAI's GPT-6.1-sol outperforms Jev on paused tasks, but lags in real-time
The paper introduces RealtimeBench, a benchmark of 180 tasks spanning six game types that continue running while an agent generates its response. By keeping the environment live, the authors can measure how timing affects performance of large‑language‑model agents.
Key points
- RealtimeBench evaluates agents on 180 continuously running tasks across six game types
- GPT-6.1-sol beats Jev when games are paused, but Jev is faster when games keep running
- TORSI training lifts Jev’s average score from 13.51 to 24.19 and helps three other models improve
Paired tests show that the larger GPT‑6.1‑sol model beats the faster Jev model when the games are paused, but Jev overtakes GPT‑6.1‑sol when the games keep running. To address this gap, the authors propose Two‑Timescale Offline Recursive Self‑Improvement (TORSI). A slow “teacher” analyses failures in paused games, writes reusable instructions, and a fast “student” applies them without further teacher calls during live play. After two TORSI rounds, Jev’s average test score rises from 13.51 to 24.19 on a 0‑100 scale, improving across all six game types, and the same instructions lift three other models without additional tuning.
These results suggest a path toward real‑time intelligence, where deliberation and action are coordinated across different timescales to keep agents effective in continuously changing environments.
Model pages: GPT-6.1 Sol → · Jev →
The story so far
4 episodes →- OpenAI's GPT-6.1-sol outperforms Jev on paused tasks, but lags in real-timethis story
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- OpenAI offers U.S. university labs free ChatGPT Business seats with Rosalind access · 2 src
- Anthropic reports small AI models may self-replicate on IoT devices within months · 29 src
- Researchers propose LLM-referenced C-test for evaluating supra-intelligence · 8 src
- Researchers study permutation diffusion LLMs for recommendation reranking · 1 src
- Researchers release MeddeX: a LLaMA-3 medical report analyzer with Gemini Vision · 1 src
Comments
via GitHub Discussions