# Study finds brain-alignment and Cross-Lingual scores can mislead when probes fail

Digest AI · Research · published 2026-10-06T04:00:00Z

Canonical: https://digestai.news/story/study-finds-brain-alignment-and-cross-lingual-scores-can-mislead-when

## Summary

A new arXiv paper challenges the use of similarity scores as evidence that language models share structure with the human brain or across languages. The authors examine two common settings. In cross-lingual transfer, a probe trained to distinguish grammatical from ungrammatical sentences in one language transfers worse to more distant languages, but the probe itself degrades along the same axis. In four of seventeen languages the probe performs at chance, making 64 of 272 language pairs unreliable. Treating the 272 pairs as independent yields p = 0.0006, while the correct unit of seventeen languages gives p = 0.155. In brain alignment, training a model to match brain responses raises its similarity score from 0.10 to 0.34 against a ceiling of 0.54. However, a model trained on a target with destroyed brain correspondence still scores 0.31, meaning only 0.028 to 0.068 of the increase is brain-specific. A destroyed target alone sits at 0.204 similarity to the real one, a floor tied to target rank divided by sentence count. Steering a language along its own direction improves performance by +6.2 over a random direction in sixteen of seventeen languages, yet shows no variation with language distance.

## Key points

- Brain-alignment score rises from 0.10 to 0.34 but destroyed-target model still scores 0.31
- Statistical significance flips from p=0.0006 to p=0.155 when using languages not pairs as units

## Why it matters

Similarity scores used to claim brain-like or cross-lingual structure can be inflated by methodological artifacts; researchers must check probe reliability and correct statistical units before drawing conclusions.

## Sources

1. [The Score Is Not the Structure: Brain Alignment and Cross-Lingual Transfer](https://arxiv.org/abs/2610.03827) (arXiv cs.CL, 2026-10-06, primary source)

## Cite

Digest AI, "Study finds brain-alignment and Cross-Lingual scores can mislead when probes fail", 6 October 2026, https://digestai.news/story/study-finds-brain-alignment-and-cross-lingual-scores-can-mislead-when

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/study-finds-brain-alignment-and-cross-lingual-scores-can-mislead-when.json