Why Chatbots Make Things Up, and What Fixes Help

Ask a chatbot for the history of a small-town newspaper and it may hand you a founding year, a founder's name and a colorful quote from the first editorial. All three can be invented. They read fine. They are formatted like facts. And the model will often defend them if you push back. This is hallucination, and anyone who repeats what a chatbot says needs to understand where it comes from.

Why it happens

Recall lesson 5: a language model picks plausible next tokens. A plausible-sounding founding year and the real one look much the same to that process. For famous facts, the training data repeated the true version thousands of times, so the true version wins. For obscure facts (a county commissioner, a 1962 bond vote, a small band's first album) the model saw little or nothing, and it fills the gap with something that fits the shape.

Recent research from inside two AI companies adds detail.

• The incentive problem. In September 2025 OpenAI published a paper arguing that standard training and testing reward guessing. If a benchmark counts only right answers, a model that guesses on every question beats one that says "I don't know" when unsure. Like a student on a multiple choice test with no penalty for wrong answers. The authors proposed scoring that punishes confident errors more than honest uncertainty.
• The misfire. In March 2025 Anthropic published research tracing the internal circuits of its Claude model. It found a default "can't answer" pathway that switches off when the model recognizes a familiar name. Hallucinations can happen when a name feels familiar but the model knows little else about it. The brake comes off, and there's nothing real behind it.

Hallucinations cluster in predictable places. Learn these and you'll catch most of them:

Grounding: put the sources in front of it

The best fix so far is to stop asking the model to remember and start asking it to read. Retrieval-augmented generation, or RAG, was named in a 2020 paper led by Patrick Lewis at Facebook AI Research. The idea:

When a chatbot shows "Searched the web" and lists sources, it is doing a version of this, often called grounding. So does any tool that lets you upload a PDF and ask questions about it.

Grounding helps a lot. It doesn't make answers trustworthy on its own. The search can pull the wrong documents (an old version of the budget, a different town with the same name). The model can misread a table, merge two sources, or add a "fact" from memory that isn't in any of them. It can cite a real page that doesn't support the sentence attached to it. So open the source and find the sentence.

Reasoning models: thinking before answering

Starting with OpenAI's o1 in September 2024, companies trained models to write out a long internal chain of steps before giving a final answer, using reinforcement learning to reward chains that end at right answers. OpenAI reported o1 averaged about 74 percent on the 2024 AIME, a hard high school math competition, with one attempt per problem. GPT-4o averaged about 12 percent. In January 2025 the Chinese lab DeepSeek published R1 with open weights, reporting that reasoning ability could be grown with reinforcement learning alone, without people writing out example reasoning. Today most big model families have a "thinking" mode or setting.

Reasoning helps most on problems with checkable steps: math, code, logic puzzles, multi-step planning. It costs time and money, since the model may produce thousands of hidden tokens before answering. It does not give the model facts it never learned. A reasoning model asked about an obscure local official can reason very carefully toward an invented answer. And the written-out reasoning a model shows you isn't always a faithful account of how it got there; Anthropic's circuit research found a model describing the schoolbook carrying method for addition while internally doing something else.

Tool use: hand the job to the right tool

The other big fix is to stop making the model do everything itself. With tool use, the model can decide mid-answer to call a calculator, run a short program, search the web, or query a database, then read the result and carry on. Ask a model with a code tool for the average of 400 numbers and it writes a few lines of code and runs them, instead of predicting digits. In November 2024 Anthropic released the Model Context Protocol, an open standard for wiring tools and data into AI apps, which other major companies have since adopted.