Fine-Tuning vs RAG: Which One Does Your App Actually Need?

Teams waste months building the wrong solution. Here's how to choose correctly in under 10 minutes.

The Core Question

Ask yourself: Is this a knowledge problem or a behavior problem?

• Knowledge problem: The model doesn't know the facts it needs (your docs, your products, recent events). Use RAG.
• Behavior problem: The model knows the facts but isn't responding the right way (format, tone, style, domain vocabulary). Use fine-tuning.

Most teams need RAG. Very few need fine-tuning.

When RAG is the Right Choice

Use RAG when:

• Your knowledge base changes frequently (updated docs, new products, recent data)
• You need source citations
• You have more than ~50 documents worth of context
• You need to add knowledge without touching the model

A basic RAG setup:

from openai import OpenAI
import numpy as np

client = OpenAI()

def embed(text: str) -> list[float]:
    return client.embeddings.create(
        input=text,
        model="text-embedding-3-small"
    ).data[0].embedding

def retrieve(query: str, chunks: list[dict], k: int = 5) -> list[str]:
    query_vec = embed(query)
    scores = [(np.dot(query_vec, c["embedding"]), c["text"]) for c in chunks]
    scores.sort(reverse=True)
    return [text for _, text in scores[:k]]

def answer(question: str, chunks: list[dict]) -> str:
    context = "\n\n".join(retrieve(question, chunks))
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": f"Answer using this context:\n{context}"},
            {"role": "user", "content": question}
        ]
    )
    return response.choices[0].message.content

When Fine-Tuning is the Right Choice

Use fine-tuning when:

• You need consistent format/style that system prompts can't achieve reliably
• You're making thousands of API calls and want a smaller, cheaper model
• You have a narrow domain where you want expert-level language (medical, legal, code)
• Prompt engineering has hit its ceiling

What fine-tuning is NOT good for: teaching the model new facts. Fine-tuned knowledge is baked in at training time -- it goes stale. Use fine-tuning to teach behavior, not information.

The Cost Comparison

At low volume (<10k requests/day), RAG almost always wins on cost.

Decision Flowchart

Does the model need current/changing information?
  YES -> Use RAG

Does the model know the facts but format them wrong?
  YES -> Try system prompt first; if still wrong, fine-tune

Do you have >1000 labeled examples of correct responses?
  NO -> Don't fine-tune yet (not enough data)

Are you spending >$5k/month on LLM API calls?
  YES -> Fine-tuning ROI worth calculating

Start with RAG, Always

Unless you have a clear behavior problem with hundreds of labeled examples, start with RAG. It's faster, cheaper to update, and easier to debug. Fine-tune later if you hit a wall.

The teams that ship fastest prototype with a large model plus RAG, then optimize with fine-tuning once they understand the failure modes.