Large language models are remarkably capable out of the box, but they do not know your products, your policies or last week's price list. There are two common ways to close that gap: retrieval-augmented generation (RAG) and fine-tuning. They are often discussed as alternatives, yet they solve different problems.
What each approach actually does
RAG leaves the model unchanged. When a question arrives, the system searches your documents or database, retrieves the most relevant passages and places them in the prompt. The model then answers using that context — ideally citing where the information came from.
Fine-tuning changes the model itself. You train it further on examples of the inputs and outputs you want, so it learns a style, a format or a narrow skill.
A useful rule of thumb: RAG changes what the model knows at answer time; fine-tuning changes how the model behaves.
When RAG is the better choice
- Your knowledge changes often. Updating a search index takes minutes; retraining a model takes days.
- Answers must be traceable. Retrieved passages give you citations that users and auditors can verify.
- Access control matters. You can filter retrieved documents by user permissions. A fine-tuned model cannot "forget" data for one user.
- You have documents, not labeled examples. RAG works with the manuals, FAQs and tickets you already have.
When fine-tuning earns its place
- You need a consistent output format — for example strict JSON for an extraction pipeline.
- The task is narrow and repetitive, such as classifying support tickets into 40 categories.
- Latency and cost matter at scale. A small fine-tuned model can replace a large general model for one well-defined job.
- Domain language is unusual — code-mixed speech, legal shorthand or internal jargon.
The combination most teams end up with
In practice, mature systems often use both: retrieval supplies fresh, permissioned facts, while a lightly fine-tuned model follows the house style and output format reliably. But the order matters. We recommend starting with RAG because it is faster to build, easier to debug and gives you something fine-tuning cannot: an evaluation dataset built from real user questions.
Measure before you decide
Whatever you choose, build an evaluation set first. Collect 100–300 real questions with reference answers and score every change against them:
| Metric | What it tells you |
|---|---|
| Answer accuracy | Is the response factually correct? |
| Groundedness | Is every claim supported by retrieved context? |
| Retrieval recall | Did search find the passage that contains the answer? |
| Refusal quality | Does the system say "I don't know" when it should? |
| Cost & latency | Can it run at your volume and budget? |
If retrieval recall is low, fine-tuning the generator will not help — fix chunking, embeddings and search first. If recall is high but answers are still inconsistent in format or tone, that is when fine-tuning pays off.
A practical starting plan
- Index your top documents and ship a RAG prototype to a small group of internal users.
- Log every question and rate the answers to grow your evaluation set.
- Improve retrieval until recall is consistently high.
- Fine-tune only for the specific behaviours that prompting cannot fix.
Choosing well is less about model hype and more about disciplined measurement. If you would like help designing an evaluation set or a first prototype, our Generative AI & LLM team can get you from idea to a measured pilot in a few weeks.