# [AI Measurement #5] You don't need to read that "I've done it"

Digest AI · Agents & Tools · published 2026-09-20T21:23:00Z

Canonical: https://digestai.news/story/ai-measurement-5-you-don-t-need-to-read-that-i-ve-done-it

## Summary

The author examined 135 instances where Claude‑based models (Opus 5, Sonnet 5, Haiku 4.5) answered “I’ve done it” after attempting to fix small Python bugs. The study found that typical textual cues—politeness, length, assertiveness, or the number of cited grounds—did not predict correctness. In fact, longer explanations were slightly more likely to be wrong, and the presence of strong language such as “everything” or “completely” showed no reliable pattern.

The key insight is that the content of the AI’s reply is not a trustworthy signal. The author recommends three practical steps: ignore the reply text, trust only explicit “I’m not done” messages (which were always correct in the sample), and create a single hidden verification condition before asking the AI to act. If multiple AIs give divergent answers, treat the result as unreliable. This approach can be applied to code fixes, document generation, or any task where an AI claims completion.

The findings suggest that users should shift focus from reading AI explanations to designing concrete, pre‑defined checks that can be evaluated independently of the AI’s output.

## Key points

- Politeness, length, or assertive language did not correlate with success
- All 45 "I'm not done" replies were accurate; hidden checks proved reliable

## Why it matters

Understanding that AI self‑reports are unreliable helps developers and businesses avoid costly rework and builds safer workflows around AI‑generated code or content.

## Sources

1. [AI Measurement #5 You don't need to read that "I've done it"](https://note.com/generativeai_new/n/n6ffcd8136de4?hl=en) (note.com, 2026-09-20)

## Cite

Digest AI, "[AI Measurement #5] You don't need to read that "I've done it"", 20 September 2026, https://digestai.news/story/ai-measurement-5-you-don-t-need-to-read-that-i-ve-done-it

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/ai-measurement-5-you-don-t-need-to-read-that-i-ve-done-it.json