Why AI makes things up, and how to reduce it

Confident error is the default, not a glitch

The word hallucination describes a model producing content that is fluent, specific and false: a citation that does not exist, a legal clause that was never written, a product feature that was never shipped. It is tempting to see this as a bug that will be patched. It is more accurate to see it as the natural behaviour of a system trained to continue text plausibly, only partly tamed by later training.

Recall the mechanics. At every step the model produces a probability distribution over the next token and picks one. There is no separate step where it consults a store of facts and checks whether the sentence it is building is true. If the training data contained many academic papers with references, then after a claim the model will find "as shown by" followed by an author name and a year highly plausible. Whether that specific author wrote that specific paper is not something the objective ever asked about. The model is completing a pattern, and the pattern of a reference is far more common in its training than any particular real reference.

The confidence is also structural. Text in the training data rarely says "I am not sure, but". Encyclopaedias, manuals and articles state things. A model trained to imitate that text inherits the register of certainty regardless of whether it has anything to be certain about. Later training stages push models towards admitting uncertainty, but the underlying tendency towards fluent assertion is always there beneath the surface.

When it is more and less likely

Hallucination is not evenly distributed. Understanding where it clusters is the difference between a practitioner who gets burned and one who plans around it.

Risk rises sharply when you ask for specifics that are rare in training data. Exact quotations, precise figures, obscure names, URLs, case numbers, version details and references are all places where the pattern is common but the particular fact is scarce. The model knows what a citation looks like far better than it knows any given citation. Risk rises again when the honest answer would be "there is no such thing". Asked for the fifth chapter of a four-chapter book, or a policy your organisation never wrote, a model that has been rewarded for helpfulness will often produce something instead of nothing.

Risk also rises with distance from the prompt. A long generated answer drifts because each token conditions on previously generated tokens, including any early mistakes. And it rises when the question sits in a gap between things the model has seen often, where two plausible patterns blend into a confident hybrid that matches neither.

Risk falls when the relevant information is inside the context window, when the task is transformation rather than recall, and when the format leaves less room for invention. Summarising a document you supplied is far safer than answering from memory about that document's topic. Choosing between options you listed is safer than generating options. Extracting fields that are present in the text is safer than inferring fields that are not.

Three families of mitigation

Every practical defence falls into one of three families, and reliable workflows use all three.

Grounding means putting the truth inside the context window. Supply the document, the data, the definitions. Ask the model to work from what it has been given, not from what it remembers. Retrieval systems, covered in the next module, automate this by finding relevant passages and inserting them before the model answers. Grounding reduces invention because plausible continuation of a provided text is much closer to the truth than plausible continuation of a general pattern.

Verification means checking outputs against something other than the model's own confidence. That can be a human reading critical claims, a second pass where the model is asked to find the evidence for each statement it made, a tool that checks whether cited sources exist, or a test suite that runs generated code. The principle is that verification must reach outside the generation loop; asking the model "are you sure" is weak, because the same machinery that produced the error will often produce the reassurance.

Constraint means shaping the task so that invention has less room. Give the model permission and explicit instruction to say "not found" or "I do not know". Ask for quotes with locations instead of paraphrases. Restrict outputs to a fixed set of options. Ask for structured output with fields that must be filled from the source. Lower the temperature for factual work. Each of these narrows the space of plausible continuations towards ones that can be checked.

Applying this tomorrow

Before your next factual task with a model, run a short triage. Ask what kind of claim you need: is it a transformation of material you can supply, or a recall of something you cannot? If you can supply it, do so, and tell the model to stay within it. If you cannot, treat every specific detail in the output as a lead to be checked, not a fact to be used.

Then decide who or what verifies. For low-stakes drafting, a quick read may do. For anything that will be sent, published, relied upon or acted on, name the check explicitly: a person, a lookup, a test. Make "I could not find this in the material" an acceptable and welcome answer, and notice how often a well-constrained model gives it.