{"version":1,"type":"story","url":"https://digestai.news/story/elorian-and-scale-ai-release-benchmark-showing-ai-lags-in-visual-reaso","json":"https://digestai.news/story/elorian-and-scale-ai-release-benchmark-showing-ai-lags-in-visual-reaso.json","markdown":"https://digestai.news/story/elorian-and-scale-ai-release-benchmark-showing-ai-lags-in-visual-reaso.md","slug":"elorian-and-scale-ai-release-benchmark-showing-ai-lags-in-visual-reaso","headline":"Elorian and Scale AI release benchmark showing AI lags in visual reasoning","summary":"Elorian and Scale AI introduced **Humanity’s Sixth Sense**, a benchmark testing how AI models interpret implied details in visual scenes. The test covers 522 questions across 288 images and 234 short video clips (17.6 hours total), divided into categories like physical logic, social understanding, and abstract inference.\n\nHuman performance is high—93.1% pass rate—but the best AI model, **GPT-6 Astra**, scores only 53.6%. Chinese open-weight models like **GLM-5.3-flash** and **Kimi K3** perform worse, with pass rates of 26.8% and 25.5%, respectively. The authors note that AI often misses critical details or misidentifies objects, highlighting gaps in intuitive visual reasoning. The paper argues this challenges assumptions that advanced world models are needed for such tasks.","keyPoints":["Benchmark tests AI’s ability to infer implied details in visual scenes, with 522 questions across images and videos","GPT-6 Astra scores 53.6% pass rate, far below human baseline of 93.1%, while Chinese models lag further","Common AI failures include missing key details (21% of errors) and misidentifying objects (20%)"],"whyItMatters":"The benchmark exposes limitations in AI’s ability to reason about visual contexts, which could slow progress in applications like robotics, autonomous systems, or AI-assisted design. It also suggests current models may overperform on tasks requiring less nuanced spatial or social understanding.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":["Elorian","Scale AI","OpenAI","Google DeepMind","NVIDIA","Microsoft Research"],"models":["GPT-6 Astra","GLM-5.3-flash","Kimi K3","Beam","Mistral Large 4","LLaMA 3.x"],"people":["Jack Clark","Robin Sloan"]},"firstPublishedAt":"2026-10-12T12:30:39Z","updatedAt":"2026-10-12T12:30:39Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"Import AI","title":"Import AI 476: Creativity and RSI; worse-than-China open weight models; AI goes to space","url":"https://jack-clark.net/2026/10/12/import-ai-476-creativity-and-rsi-worse-than-china-open-weight-models-ai-goes-to-space","publishedAt":"2026-10-12T12:30:39Z","type":"newsletter","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Elorian and Scale AI release benchmark showing AI lags in visual reasoning\", 12 October 2026, https://digestai.news/story/elorian-and-scale-ai-release-benchmark-showing-ai-lags-in-visual-reaso","publisher":"Digest AI","title":"Elorian and Scale AI release benchmark showing AI lags in visual reasoning","datePublished":"2026-10-12T12:30:39Z","url":"https://digestai.news/story/elorian-and-scale-ai-release-benchmark-showing-ai-lags-in-visual-reaso"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}