Researchers find saturated benchmark transcripts still predict model capabilities

Their experiments show that even when benchmark accuracy is saturated, transcript data still encodes capability signals. Pairwise judging performs best, cutting prediction error by 5.3 times relative to a baseline that uses only the saturated scores. The informative content is spread across transcript sections but is especially concentrated in reasoning steps and tool‑call logs. However, the methods do not always generalise; they tend to underestimate the abilities of the first reasoning models released in early 2025. The study suggests that extending benchmark usefulness will require robust techniques to extract these hidden signals from model transcripts.