Fine-Tuning vs RAG vs Prompting

Three adaptation strategies, three tradeoffs

When you need an LLM to behave for a specific domain, you have three adaptation strategies, and the senior skill is knowing which one (or which combination) fits, because they trade cost, freshness, and quality differently. Getting this wrong is expensive: teams routinely fine-tune for knowledge that changes weekly, then rebuild the model every time the data moves.

The decision framework

• Prompting (including few-shot) changes behavior. Put instructions, format rules, and a few examples in the context. Zero training cost, instant to change, but limited by context window and it does not add knowledge the model never had. Use it for tone, output format, and task framing. Always start here.
• RAG adds fresh, private knowledge. Retrieve relevant data at query time and ground the answer. This is the right tool whenever the knowledge changes or is private or is large, because you update an index, not a model. Facts stay current by re-indexing. Use it for "answer over our docs / our data / today's numbers."
• Fine-tuning changes style, format adherence, and latency. Train the weights (usually with adapters) on many examples so the model internalizes a behavior you cannot reliably prompt for, or so a smaller/cheaper model matches a bigger one on your task. It bakes knowledge in as of training time, so it goes stale. Use it for consistent structure, a specialized tone, a narrow classification, or to distill a big model into a cheap one, not for facts that change.

The one-line heuristic: prompting for behavior, RAG for knowledge, fine-tuning for style/format/latency. They compose: a strong system often fine-tunes a small model for format and cost, then RAG-grounds it for facts.

PEFT and LoRA change the economics

Full fine-tuning updates all weights, which is expensive and produces a whole new multi-gigabyte model per task. LoRA (a PEFT method) freezes the base model and trains tiny low-rank adapter matrices, a few megabytes, that adjust behavior. This is transformative operationally: you host one base model and swap or multiplex many small adapters on top (adapter-per-tenant or adapter-per-task) on the same GPU, instead of hosting a separate full model each. Full fine-tuning is rarely justified now; LoRA gives most of the benefit at a fraction of the cost and storage. Interview nuance: when asked "how would you fine-tune," naming LoRA/PEFT and adapter multiplexing signals you understand production economics, not just the concept.

The data flywheel and freshness

Capture production traces (inputs, chosen outputs, human corrections, thumbs), curate them, and use them to distill a smaller cheaper model or to improve the next adapter. Real usage becomes training data, so quality and cost improve over time. This flywheel is the durable moat.

RAG index updates keep facts current continuously; fine-tuning requires periodic re-tuning to refresh baked-in knowledge, which is why you do not fine-tune for volatile facts. Whatever you train, you version the model and adapters, gate promotion behind eval, and keep rollback ready.

Recap: prompting for behavior, RAG for fresh/private knowledge, fine-tuning (via LoRA adapters, rarely full) for style/format/latency; they compose; drive continuous improvement with a data flywheel; and never fine-tune for knowledge that changes when RAG keeps it fresh.