Opinion: ChatGPT hallucinates factual answers, author explains why it seems confident

The first time I thought, "This is a problem," was when I asked about the business hours of a local supermarket.

ChatGPT answered as if it had seen it for itself. What time it opened and closed, and when it was closed. It was written in such natural language that there seemed to be no room for doubt.

But when I actually went there, the business hours were completely different.

I wish it had just said, "I don't know."

What felt strange wasn't the mistake itself, but rather its attitude.

A human would say, "I think it's from 10 o'clock, but I'm not sure." But ChatGPT didn't seem to hesitate at all. It answered incorrectly, boldly and with full confidence.

Why wouldn't it just say, "I don't know"?

I don't think there's a reason for it to lie. Yet, it didn't look like it was making a mistake on purpose either. This sense of discomfort bothered me, so I decided to look into how AI works.

What I found out: AI doesn't actually "know" things

As I researched, I realized that my own assumptions were fundamentally off.

I had always thought of ChatGPT as a "knowledgeable person who remembers many things." I imagined it had a large dictionary in its head and would look up and tell me the answer from there.

But in reality, that wasn't the case.

What an AI like ChatGPT is doing is predicting "what word would naturally come next" to construct sentences. It learns from vast amounts of text and acquires patterns like, "In this flow, these words should follow next." That accumulation is what creates those natural sentences.

In other words, it is not "looking up" answers, but "creating" plausible-sounding sentences.

When you think about it that way, that confident attitude makes sense. For an AI, answering with the correct business hours and creating plausible-sounding business hours are the same thing. Both are just the task of "constructing natural sentences." That's why it couldn't possibly hesitate.

These plausible but factually incorrect answers are called "hallucinations." Knowing that this isn't a bug but something that can happen based on how it works made me strangely satisfied.

It wasn't a "tool that makes mistakes," but "that kind of tool"

Since understanding this mechanism, my way of using ChatGPT has changed.

Until then, every time it didn't answer well, I thought, "This is useless." But I was just using the tool incorrectly. It's like trying to drive a nail with a screwdriver and saying, "This tool is useless."

To put it simply, it's like this.

What it is bad at is answering factual questions. Business hours, prices, the latest news, names of people, and numbers. It makes mistakes on these quite casually. If you ask about them, always do so with the premise that you will verify the information yourself.

What it is good at is creating or refining text. "Make this content into a polite message," "summarize this long sentence into a short one," "create a meal plan based on these conditions." It is very good at tasks like these where there is no single correct answer.

Having it create meal plans is exactly where it is convenient. There is no "single correct answer" for a meal plan. That is why it works well with AI, which is good at creating plausible suggestions.

The benefit of having doubts

If I had just thought "AI is actually not very useful" and left it at that back then, instead of wondering "why does it make mistakes?", I probably wouldn't be using it even now.

Just because I know how it works doesn't mean the AI has become smarter. It still makes mistakes quite casually. But I have come to understand what to delegate and what to verify myself. That alone has made it a much more reliable tool.

Whether you keep your distance from things you don't understand by thinking "it's scary because I don't really get it," or take a step closer by wondering "why is that?" My takeaway from this experience was that the difference between those two is bigger than I thought.

I usually write about ways to use ChatGPT for housework and work during spare time. Please feel free to take a look at my other articles as well.