Prompting, RAG or fine-tuning: how to adapt a language model to your task

There are three main ways to adapt a language model to a task: a well-written prompt with instructions and examples, RAG — inserting relevant passages from your knowledge base into the request — and fine-tuning on your own examples. Almost always start with prompts, supply company knowledge through RAG, and move to fine-tuning only to lock in a special format, style or behaviour you cannot get otherwise.

Here is what each approach solves, what it costs and how to choose.

How do prompting, RAG and fine-tuning differ?

The approaches solve different problems. A prompt tells the model what to do and how. RAG gives it knowledge it lacks. Fine-tuning changes its behaviour at the level of its weights.

Approach	What it solves	Setup cost	Updating knowledge	When to choose
Prompt engineering	Instructions, format, tone, examples	Minimal	Edit the text	Always first
RAG	Answers from your documents and data	Medium	Add documents	Current company facts are needed
Fine-tuning	Consistent style, format, narrow skills	High	Retrain	Prompts and RAG cannot produce the behaviour

When is a good prompt enough?

A good prompt is enough for most tasks: classifying requests, extracting data, generating templated text, answering in a set tone. Modern models follow detailed instructions well, especially with a few examples of correct answers.

What a good prompt includes:

1. Role and context: who the user is and why the answer is needed.
2. A clear task and criteria for a good result.
3. The answer format — ideally a strict schema for structured output.
4. A few examples of inputs and correct outputs.
5. What to do in hard cases: decline, ask, or leave a field empty.

A prompt is code: keep it in the repository, version it and test it against a set of examples. See LLM evaluation.

Photo: Unsplash

When do you need RAG?

RAG is needed when the model must answer from your data: documentation, policies, a knowledge base, a catalogue. Before each answer the system finds suitable passages and passes them to the model with the question. Knowledge is updated by adding documents, with no training, and answers can cite sources.

• a support assistant over the knowledge base and ticket history;
• search over internal documents and policies;
• answers about the product catalogue with current prices and stock;
• an employee assistant for contracts and procedures.

How to build it: RAG: an AI assistant for your own documents and vector databases.

When is fine-tuning genuinely needed?

Fine-tuning is needed for consistent behaviour that instructions cannot achieve: a strict, specific format across huge request volumes, a particular style, narrow classification into hundreds of categories, or transferring a large model's quality to a small, cheap one. It is poorly suited to adding knowledge: facts are absorbed unreliably and go stale.

• the same complex format is needed across millions of requests and the prompt is too long and costly;
• a large model's behaviour must move to a small one to cut cost and latency;
• a narrow domain task where the base model lacks quality even with examples;
• you have thousands of high-quality labelled examples and a process to refresh them.

Before fine-tuning, assemble a test set and measure what prompting and RAG achieve — otherwise you cannot tell whether training helped.

How do you choose?

Move from simple to complex and measure quality at every step. Most products stop at prompting and RAG; fine-tuning remains an optimisation for mature, high-volume scenarios.

1. Start with the best available model and a detailed prompt.
2. Build a test set and measure quality.
3. If knowledge is missing, add RAG.
4. If format or behaviour is unstable, improve the prompt and examples.
5. If volume is high and cost must fall, consider fine-tuning a smaller model.

On cost, see AI feature cost, and on general principles, how to integrate an LLM into a product.

In short

• A prompt explains the task, RAG supplies knowledge, fine-tuning changes the model's behaviour.
• Always start with a good prompt and a test set.
• Supply company knowledge through RAG — easy to update and verify against sources.
• Fine-tuning pays off for consistent format, narrow tasks and cutting cost at scale.
• Measure quality at each step, or the choice is made blind.