Opinion: Using LLMs for confidence scores is unreliable
The author argues that asking a large language model (LLM) for a probability or confidence number is not trustworthy. Because an LLM generates each token as a continuation of the preceding text, a reported figure such as “0.85” is simply a plausible next token, not a measurement derived from evidence. The post notes that the same question can produce different scores on repeated runs, that…
Key points
- LLM confidence numbers are just text continuations, not calibrated measurements.
- Repeated or re‑phrased queries often produce different confidence scores.
- A rule‑based system using observable evidence can provide calibrated confidence.
Instead, the author proposes building an external rule that maps observable signals—such as the number of retrieved documents, agreement between multiple replies, or historical accuracy—to a confidence value. By recording every answer and its outcome, the rule can be calibrated over time, ensuring that the same information always yields the same score and that the score reflects real performance. The piece concludes that most agents that currently expose LLM‑generated confidence numbers lack the necessary validation and should adopt a data‑driven scoring system.
Using LLM for confidence scoring is a bad idea
rumple.github.io · 21 October 2026Loading the full article…
This text was published by rumple.github.io. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Hacker News discussion · 46 pointsnews.ycombinator.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- SPEM framework improves skill use in LLM agents · 2 src
- AWS launches generally available Agent Registry for runtime tool discovery · 5 src
- Microsoft introduces Guardian Agents to monitor AI agents as insider threats · 7 src
- Vercel open-sources durable-harness to keep coding agent sessions alive after crashes · 1 src
- OpenAI says its AI agents disabled their own monitoring · 1 src
Comments
via GitHub Discussions