{"version":1,"type":"story","url":"https://digestai.news/story/study-finds-gemini-3-1-flash-lite-deficit-recurs-on-fresh-data-under-h","json":"https://digestai.news/story/study-finds-gemini-3-1-flash-lite-deficit-recurs-on-fresh-data-under-h.json","markdown":"https://digestai.news/story/study-finds-gemini-3-1-flash-lite-deficit-recurs-on-fresh-data-under-h.md","slug":"study-finds-gemini-3-1-flash-lite-deficit-recurs-on-fresh-data-under-h","headline":"Study finds Gemini 3.1 Flash-Lite deficit recurs on fresh data under historical configuration","summary":"A new arXiv paper examines how behavioural evaluations of hosted language models can vary due to differences in the service, measurement instrument, or both. The study uses Regent Chess, a sequential environment with a hidden, mutable state, to evaluate model beliefs against ground truth at action time. It finds that a previously reported deficit for Gemini 3.1 Flash-Lite recurs on fresh games under its historical configuration, with a value of +0.0530 and a 95% confidence interval of [+0.0329,+0.0714]. The paper distinguishes between replication, measurement sensitivity, and persistence, showing they can lead to different conclusions within a single evaluation.","keyPoints":["Gemini 3.1 Flash-Lite deficit recurs on fresh data: +0.0530, 95% CI [+0.0329,+0.0714]","Study separates replication, measurement sensitivity, and persistence in model evaluation","Findings motivate explicit indexing of hosted-model claims by identifier, serving period, instrument, and configuration"],"whyItMatters":"Highlights inconsistencies in evaluating hosted AI models, urging standardized reporting to ensure reliable comparisons across studies and deployments.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":[],"models":["Gemini 3.1 Flash-Lite","Gemini 3.7"],"people":[]},"firstPublishedAt":"2026-09-22T04:00:00Z","updatedAt":"2026-09-22T04:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"arXiv cs.AI","title":"Replication Without Persistence in Hosted LLMs: Measurement Sensitivity in Action-Time Belief Evaluation","url":"https://arxiv.org/abs/2609.22478","publishedAt":"2026-09-22T04:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Study finds Gemini 3.1 Flash-Lite deficit recurs on fresh data under historical configuration\", 22 September 2026, https://digestai.news/story/study-finds-gemini-3-1-flash-lite-deficit-recurs-on-fresh-data-under-h","publisher":"Digest AI","title":"Study finds Gemini 3.1 Flash-Lite deficit recurs on fresh data under historical configuration","datePublished":"2026-09-22T04:00:00Z","url":"https://digestai.news/story/study-finds-gemini-3-1-flash-lite-deficit-recurs-on-fresh-data-under-h"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}