# arXiv study finds language models encode chemistry knowledge before output

Digest AI · Research · published 2026-11-04T05:00:00Z

Canonical: https://digestai.news/story/arxiv-study-finds-language-models-encode-chemistry-knowledge-before-ou

## Summary

A new paper on arXiv examines how five open-source language models process chemical functional groups. Researchers tested models on 92 molecules and 20 functional groups, finding that even without chemistry training, they organize functional groups by chemical family and encode chemical differences in consistent ways. For example, models place thiols (R–SH) near alcohols (R–OH) in their internal representations, showing structured knowledge before generating text.

The base model correctly named functional groups only 58.5% of the time, but its errors were mostly failures to respond rather than incorrect answers. Fine-tuning for chemistry shifted the models’ focus from prompt wording to functional-group structure. The study warns that relying solely on model outputs to evaluate chemical knowledge may underestimate the underlying information they encode.

## Key points

- Five open-source models organize chemical functional groups by family in internal representations, even without chemistry training
- Base model correctly identifies functional groups 58.5% of the time, but errors stem from silence, not wrong answers
- Chemistry fine-tuning reorganizes model focus from prompt wording to functional-group structure

## Why it matters

This research reveals that language models may hold deeper chemical knowledge than their outputs suggest, challenging how we evaluate AI agents for scientific tasks.

## Sources

1. [Functional-Group Knowledge Is Structured in Language Models Before It Is Spoken](https://arxiv.org/abs/2611.03684) (arXiv cs.AI, 2026-11-04, primary source)

## Cite

Digest AI, "arXiv study finds language models encode chemistry knowledge before output", 4 November 2026, https://digestai.news/story/arxiv-study-finds-language-models-encode-chemistry-knowledge-before-ou

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/arxiv-study-finds-language-models-encode-chemistry-knowledge-before-ou.json