Almost every AI project I've scoped in the last two years starts with the same question from the client: "Should we fine-tune a model for this?" It's the wrong first question. The right first question is: what specifically is the base model getting wrong, and why?
How is prompt engineering different from fine-tuning? Prompt engineering changes the instructions and examples you give a model at the moment of each request; the model itself is untouched. Fine-tuning changes the model's weights by training it on your own examples, so the behaviour is built in. Prompting is fast, cheap and reversible. Fine-tuning is slower and costs more, and it only pays off once you know exactly which behaviour prompting cannot deliver. If the real problem is missing knowledge rather than behaviour, neither is the answer: use retrieval. And before changing anything, set up an evaluation so you can tell whether the change helped.
Start by diagnosing the failure, not picking a technique
There are really only three categories of failure, and each points to a completely different fix:
| Failure mode | What it looks like | Right fix |
|---|---|---|
| Missing knowledge | Model doesn't know your product, docs, or recent events | RAG, not fine-tuning |
| Wrong format or tone | Right answer, wrong structure/voice | Prompt engineering + few-shot examples |
| Wrong behavior pattern | Model reasons incorrectly even with the right context | Fine-tuning |
Most "we need to fine-tune" requests I get turn out to be missing-knowledge problems. You cannot fine-tune your way out of a model not having seen your internal pricing sheet from last week. That's a retrieval problem wearing a fine-tuning costume.
When fine-tuning actually earns its cost
Fine-tuning is expensive in ways that don't show up on the invoice: data curation time, evaluation infrastructure, and the ongoing cost of re-tuning every time the base model improves. It's worth it when:
- You need consistent behavior across thousands of edge cases that few-shot prompting can't reliably cover
- Latency or cost requires a smaller model to match a larger model's behavior on a narrow task
- The task requires a response format so specific and repetitive that it's cheaper to bake in than to prompt every time
# A narrow, well-scoped fine-tuning target beats a broad one every time
training_examples = [
{"input": support_ticket, "output": structured_triage_json}
for support_ticket, structured_triage_json in labeled_tickets
]
# 500 tightly-scoped examples outperformed 5,000 loosely-scoped ones, in practice
The framework, in order
- Can better retrieval fix this? Try it first. It's reversible, cheap, and iterates in minutes.
- Can a better prompt fix this? Structured outputs, few-shot examples, and explicit reasoning steps solve more than people expect.
- Only then: does the remaining gap justify fine-tuning? If yes, scope the smallest possible training set that closes that specific gap - not a general-purpose retrain.
I've seen six-figure fine-tuning projects get replaced by a better system prompt and a re-ranked retrieval pipeline. I've also seen teams prompt-engineer for six months to solve a problem that a 300-example LoRA fine-tune closed in a week. The framework isn't "fine-tuning bad" - it's "diagnose before you prescribe."