Elorian and Scale AI release benchmark showing AI lags in visual reasoning
Elorian and Scale AI introduced Humanity’s Sixth Sense, a benchmark testing how AI models interpret implied details in visual scenes. The test covers 522 questions across 288 images and 234 short video clips (17.6 hours total), divided into categories like physical logic, social understanding, and abstract inference.
Key points
- Benchmark tests AI’s ability to infer implied details in visual scenes, with 522 questions across images and videos
- GPT-6 Astra scores 53.6% pass rate, far below human baseline of 93.1%, while Chinese models lag further
- Common AI failures include missing key details (21% of errors) and misidentifying objects (20%)
Human performance is high—93.1% pass rate—but the best AI model, GPT-6 Astra, scores only 53.6%. Chinese open-weight models like GLM-5.3-flash and Kimi K3 perform worse, with pass rates of 26.8% and 25.5%, respectively. The authors note that AI often misses critical details or misidentifies objects, highlighting gaps in intuitive visual reasoning. The paper argues this challenges assumptions that advanced world models are needed for such tasks.
Model pages: GPT-6 Astra → · Kimi K3 → · Beam →
Import AI 476: Creativity and RSI; worse-than-China open weight models; AI goes to space
Import AI · 12 October 2026
Loading the full article…
This text was published by Import AI and written by Jack Clark. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- AI labs buy data from dead startups for training models · 1 src
- Hugging Face’s ML-intern builds custom models from prompts in days · 1 src
- Coding Agents Can Solve the Task — But How Much Work Did They Waste? · 1 src
- Anthropic research suggests small AI models may self-replicate on IoT devices · 5 src
- Sakana AI's Multi-Layered Review catches 73.43% of core-claim errors · 1 src
Comments
via GitHub Discussions