Pre-training, fine-tuning and retrieval (RAG) explained

Three different places knowledge can live

When a model answers correctly, the knowledge behind that answer came from one of three places, and they behave very differently. Confusing them is the single most common source of muddled AI strategy in organisations.

The first is pre-training. This is the enormous initial phase where the model learns next-token prediction over a vast general corpus. Everything it absorbs here is stored in its weights, distributed across the knobs described earlier. This knowledge is broad, comes for free with the model, and is frozen at the point training stopped. It is also fuzzy: the model has a sense of what is typically true rather than a record of what is specifically true, which is exactly the property that produces hallucination.

The second is fine-tuning. Here an already-trained model is trained further on a smaller, targeted dataset so its weights shift towards a desired behaviour. This is how general models are made to follow instructions, adopt a tone, produce a particular format or handle a specialist style of task. Fine-tuning changes the weights, so its effects are permanent for that version of the model and apply to every request.

The third is retrieval. Nothing about the model changes at all. Instead, at the moment of a request, a system searches a store of documents, usually with the embedding techniques from an earlier module, and inserts the most relevant passages into the context window alongside the question. The model then answers from what it has just been shown. The common name for this pattern is retrieval-augmented generation, or RAG.

The trade-offs, honestly

Each approach has a profile of strengths and costs, and the interesting decisions live in the tension between them.

Pre-trained knowledge is free and broad but you cannot update it, you cannot audit where a particular belief came from, and you cannot remove things. If your need is general language competence, common sense and wide background, this is what you are buying when you choose a model at all.

Fine-tuning is the most powerful way to change how a model behaves. It can make outputs shorter, more consistent, better formatted, more aligned with a house style, or more reliable on a narrow task with a distinctive shape. Its costs are real: it needs a curated dataset of examples, it takes engineering effort to run and evaluate, it must be redone whenever the base model changes, and it can quietly degrade abilities you did not intend to touch. It is also a poor way to teach facts. Training on a few hundred examples that mention a product does not reliably install accurate recall of that product; it tends to install the style of talking about it, with the facts themselves still fuzzy.

Retrieval is the workhorse for knowledge that is specific, changing or sensitive. Update a document and the next answer reflects it. Restrict which documents a user can retrieve and you have access control. Show the retrieved passages alongside the answer and you have a citation trail a human can check. Its costs are different: the quality of answers is capped by the quality of retrieval, so poor chunking, a weak embedding model or a messy document store all surface as poor answers. It also adds latency and complexity, and it only helps with knowledge that can be written down and found.

Choosing, and combining

A practical decision sequence goes like this. Start by asking whether the base model already does the job acceptably with a good prompt. Surprisingly often it does, and prompt engineering is cheap, reversible and needs no data pipeline. If it fails, ask why.

If it fails because it does not know something specific, recent or private, reach for retrieval. Build a store of the relevant material, get the search working well, and feed the passages in. Judge it by whether the right passages are being found, because that is where most failures originate.

If it fails because of how it behaves, meaning tone, format, consistency, verbosity, or handling of a narrow repetitive task, consider fine-tuning, but only after prompting with good examples has been tried and only when you can assemble a clean dataset that shows the behaviour you want. Fine-tuning is a commitment; make sure the behaviour is stable enough to be worth committing to.

In mature systems the three combine. The pre-trained model supplies language and reasoning ability. Retrieval supplies the specific, current and permissioned facts. Fine-tuning, if used, supplies reliable behaviour in using those facts, such as always citing, always declining out-of-scope questions, or always producing a particular structure.

Applying this tomorrow

Next time someone proposes making a model "know" something about your organisation, translate the request into the three-way question. Is this general competence the model should already have? Is it a specific fact, document or record that changes and must be auditable? Or is it a behaviour you want repeated reliably? The first is a model choice, the second is retrieval, the third is prompting and possibly fine-tuning.