Fine-Tuning vs. Prompt Engineering: A Practical Decision Framework
Everyone asks "should we fine-tune?" before asking the question that actually matters. Here is the framework we use with clients before writing a single line of training code.
Almost every AI project I've scoped in the last two years starts with the same question from the client: "Should we fine-tune a model for this?" It's the wrong first question. The right first question is: what specifically is the base model getting wrong, and why?
Start by diagnosing the failure, not picking a technique
There are really only three categories of failure, and each points to a completely different fix:
| Failure mode | What it looks like | Right fix |
|---|---|---|
| Missing knowledge | Model doesn't know your product, docs, or recent events | RAG, not fine-tuning |
| Wrong format or tone | Right answer, wrong structure/voice | Prompt engineering + few-shot examples |
| Wrong behavior pattern | Model reasons incorrectly even with the right context | Fine-tuning |
Most "we need to fine-tune" requests I get turn out to be missing-knowledge problems. You cannot fine-tune your way out of a model not having seen your internal pricing sheet from last week. That's a retrieval problem wearing a fine-tuning costume.
When fine-tuning actually earns its cost
Fine-tuning is expensive in ways that don't show up on the invoice: data curation time, evaluation infrastructure, and the ongoing cost of re-tuning every time the base model improves. It's worth it when:
- You need consistent behavior across thousands of edge cases that few-shot prompting can't reliably cover
- Latency or cost requires a smaller model to match a larger model's behavior on a narrow task
- The task requires a response format so specific and repetitive that it's cheaper to bake in than to prompt every time
# A narrow, well-scoped fine-tuning target beats a broad one every time
training_examples = [
{"input": support_ticket, "output": structured_triage_json}
for support_ticket, structured_triage_json in labeled_tickets
]
# 500 tightly-scoped examples outperformed 5,000 loosely-scoped ones, in practice
The framework, in order
- Can better retrieval fix this? Try it first. It's reversible, cheap, and iterates in minutes.
- Can a better prompt fix this? Structured outputs, few-shot examples, and explicit reasoning steps solve more than people expect.
- Only then: does the remaining gap justify fine-tuning? If yes, scope the smallest possible training set that closes that specific gap — not a general-purpose retrain.
I've seen six-figure fine-tuning projects get replaced by a better system prompt and a re-ranked retrieval pipeline. I've also seen teams prompt-engineer for six months to solve a problem that a 300-example LoRA fine-tune closed in a week. The framework isn't "fine-tuning bad" — it's "diagnose before you prescribe."