AI Engineer Portal

Why this matters

The most common architectural question on AI teams is some version of: "should we fine-tune the model or use RAG?" Both techniques can improve output quality, but they solve different problems. Teams that conflate them either spend weeks fine-tuning for a knowledge problem (should have used RAG) or build a retrieval system for a formatting problem (should have fine-tuned).

As an AI engineer, being able to diagnose which technique fits a given use case — and articulate the tradeoffs clearly — is a high-signal competency. This lesson gives you a decision framework you can apply to real product requirements, not just academic comparisons.

Core concepts

What each technique actually does

RAG (Retrieval-Augmented Generation) solves the knowledge problem. The base model does not know your product's documentation, your company's proprietary data, or events after its training cutoff. RAG injects relevant knowledge at inference time by retrieving context from an index and stuffing it into the prompt. The model itself does not change — you are changing what it sees.

Fine-tuning solves the behavior problem. The model knows how to write prose, but it does not write in your brand's voice. It can answer questions, but it formats answers inconsistently. It handles code, but not in your internal framework. Fine-tuning adjusts the model's weights so it produces the right style, format, or domain behavior without needing a lengthy prompt to explain it.

The cleaner framing: RAG gives the model knowledge it does not have. Fine-tuning changes how the model behaves with the knowledge it already has.

When to fine-tune

Fine-tuning pays off when:

Consistent format or style at scale. If every response must follow a strict structure (JSON output with specific keys, a particular writing style, a domain-specific tone), fine-tuning teaches this once instead of repeating instructions in every prompt. Prompt instructions drift; fine-tuning generalizes.
Specialized domain with stable knowledge. Medical coding, legal contract review, proprietary API syntax — domains where the terminology and patterns are fixed and the model consistently makes predictable errors on base inference.
Latency and cost optimization. A fine-tuned smaller model (7B–13B) can match a larger base model on a narrow task at a fraction of the cost and latency. If your use case is well-defined and high-volume, this tradeoff is often worth the fine-tuning investment.
Reducing prompt length. If you need 500 tokens of instruction in every system prompt to get consistent behavior, fine-tuning can internalize those instructions and shrink the prompt. At scale, this is significant.

When RAG is better

RAG is the right choice when:

Knowledge changes frequently. Product documentation, pricing, policies, recent events — anything that updates faster than you want to retrain a model. RAG lets you update the index without touching the model.
Transparency and citations matter. RAG outputs can cite specific source chunks. Users and auditors can verify claims. Fine-tuned models produce confident outputs with no traceable provenance.
You need to iterate quickly. Fine-tuning requires dataset curation, training runs, evaluation, and deployment — a cycle measured in days to weeks. Updating a RAG index takes minutes. If requirements are still evolving, RAG keeps you unblocked.
The base model already knows how to behave. If the only gap is missing knowledge, not behavioral drift, RAG is simpler and more maintainable.

The hybrid approach

The most effective production systems often combine both. Fine-tune for format and style consistency; use RAG for dynamic knowledge injection.

Example: a customer support assistant that must always structure responses as {"issue_summary": str, "resolution_steps": list[str], "escalate": bool}. Fine-tune for this output format. Then use RAG to inject the relevant knowledge base articles about the specific product issue. The fine-tuned model knows how to format; RAG gives it the knowledge to format accurately.

Dataset curation for fine-tuning (the hard part)

The fine-tuning process is simple. Dataset curation is where most projects fail.

A good fine-tuning dataset for an instruction-following use case:

500–2000 examples for most narrow tasks (more for broader behavior)
Each example: a prompt and the ideal completion
Balanced coverage of the full input distribution, not just easy cases
No examples where the base model already performs well — these waste training data

from pydantic import BaseModel, Field
from typing import Literal
import json
from pathlib import Path


class FineTuningExample(BaseModel):
    prompt: str = Field(min_length=10)
    completion: str = Field(min_length=5)
    category: str
    difficulty: Literal["easy", "medium", "hard"]

    def to_openai_format(self) -> dict:
        return {
            "messages": [
                {"role": "user", "content": self.prompt},
                {"role": "assistant", "content": self.completion},
            ]
        }


def validate_dataset(path: Path) -> dict:
    examples = []
    errors = []

    for i, line in enumerate(path.read_text().splitlines()):
        try:
            raw = json.loads(line)
            ex = FineTuningExample.model_validate(raw)
            examples.append(ex)
        except Exception as e:
            errors.append({"line": i, "error": str(e)})

    categories = {}
    for ex in examples:
        categories[ex.category] = categories.get(ex.category, 0) + 1

    return {
        "total": len(examples),
        "errors": len(errors),
        "by_category": categories,
        "avg_prompt_length": sum(len(e.prompt) for e in examples) / len(examples) if examples else 0,
    }

Curation anti-patterns: filtering to only high-quality outputs (the model cannot learn from edge cases), including examples where the base model is already correct (wastes compute), and treating curation as a one-time step (you need a second curation pass after the first training run to find failure modes).

LoRA/QLoRA for parameter-efficient fine-tuning

Training all parameters of a 7B model requires 80GB+ of GPU memory. LoRA (Low-Rank Adaptation) makes fine-tuning accessible by freezing the base model weights and training only small adapter matrices inserted into each attention layer. These adapters represent the delta from base behavior to target behavior.

QLoRA adds 4-bit quantization to the frozen base model, reducing memory requirements to a single consumer GPU (24GB VRAM for a 7B model). The quality loss from quantization on the frozen weights is minimal for most tasks.

Application-level understanding: you do not need to implement LoRA yourself. Libraries like peft handle this. What you need to know:

LoRA adapters are small (tens of MB) and can be switched at runtime — useful if you need different behaviors for different customers
The r (rank) hyperparameter controls capacity: higher rank learns more complex behavior but needs more data and compute
LoRA fine-tunes do not replace the base model; they sit on top. If the base model lacks a capability entirely, LoRA cannot add it

# Conceptual: what LoRA looks like with the peft library
# (requires: pip install peft transformers)
from peft import get_peft_model, LoraConfig, TaskType
from transformers import AutoModelForCausalLM

def create_lora_model(base_model_name: str, r: int = 8) -> object:
    """Wrap a base model with LoRA adapters for fine-tuning."""
    model = AutoModelForCausalLM.from_pretrained(base_model_name)

    config = LoraConfig(
        task_type=TaskType.CAUSAL_LM,
        r=r,                       # Rank: higher = more capacity
        lora_alpha=32,             # Scaling factor
        target_modules=["q_proj", "v_proj"],  # Which layers to adapt
        lora_dropout=0.05,
        bias="none",
    )

    return get_peft_model(model, config)

Decision tree: fine-tune or RAG?

from dataclasses import dataclass
from typing import Literal

Recommendation = Literal["rag", "fine_tune", "hybrid", "prompt_engineering_first"]


@dataclass
class UseCaseCriteria:
    knowledge_changes_frequently: bool
    need_citations_or_traceability: bool
    problem_is_formatting_or_style: bool
    domain_has_stable_specialized_vocabulary: bool
    response_volume_is_high: bool  # >100k calls/day
    iteration_speed_matters: bool
    base_model_format_already_correct: bool


def recommend_approach(criteria: UseCaseCriteria) -> tuple[Recommendation, str]:
    # RAG signals
    if criteria.knowledge_changes_frequently and not criteria.problem_is_formatting_or_style:
        return "rag", "Knowledge changes frequently — RAG lets you update without retraining."

    if criteria.need_citations_or_traceability:
        return "rag", "Citations required — RAG provides traceable source chunks."

    if criteria.iteration_speed_matters and not criteria.response_volume_is_high:
        return "rag", "Fast iteration required — RAG index updates are immediate."

    # Fine-tuning signals
    if criteria.problem_is_formatting_or_style and criteria.response_volume_is_high:
        return "fine_tune", "Consistent format at high volume — fine-tuning internalizes the pattern cheaply."

    if criteria.domain_has_stable_specialized_vocabulary and not criteria.knowledge_changes_frequently:
        return "fine_tune", "Stable specialized domain — fine-tuning improves domain accuracy."

    # Hybrid
    if criteria.problem_is_formatting_or_style and criteria.knowledge_changes_frequently:
        return "hybrid", "Fine-tune for format consistency; use RAG for dynamic knowledge injection."

    # Default: try prompt engineering first
    return "prompt_engineering_first", (
        "Neither signal is strong. Start with a well-engineered prompt; "
        "invest in fine-tuning or RAG only after measuring baseline performance."
    )


# Example usage
criteria = UseCaseCriteria(
    knowledge_changes_frequently=False,
    need_citations_or_traceability=False,
    problem_is_formatting_or_style=True,
    domain_has_stable_specialized_vocabulary=True,
    response_volume_is_high=True,
    iteration_speed_matters=False,
    base_model_format_already_correct=False,
)
rec, reason = recommend_approach(criteria)
print(f"Recommendation: {rec}")
print(f"Reasoning: {reason}")
# Recommendation: fine_tune
# Reasoning: Consistent format at high volume — fine-tuning internalizes the pattern cheaply.

Common mistakes

Fine-tuning for a knowledge gap. If the model does not know your company's products because that information postdates training, fine-tuning on that knowledge teaches it during training but the knowledge becomes stale immediately. Use RAG.

Using RAG for a style problem. If every output needs to follow a strict JSON schema, adding retrieval does not help. The retrieval step brings in content; the model still formats it inconsistently without instructions or fine-tuning.

Underinvesting in dataset curation. 500 carefully curated examples outperform 5,000 scraped examples in fine-tuning. The single highest-leverage investment before starting a fine-tuning project is 2–3 days of dataset review and cleaning.

Forgetting the eval loop. Fine-tuning without a golden dataset to measure before and after is guesswork. Define your quality metric and run it on both the base model and fine-tuned model before declaring success.

Skipping the prompt engineering baseline. Before spending time on RAG infrastructure or fine-tuning, always measure what a well-engineered prompt achieves on the base model. Many teams discover the gap is smaller than expected, or that the problem is different from what they assumed.

Try it yourself

Implement the recommend_approach function from above and extend it with two more decision branches: one for cases where latency is critical (under 200ms response time) and one for cases where the use case involves multiple languages. For the latency case, consider whether a fine-tuned smaller model or a RAG-augmented larger model would be faster end-to-end. Write out the reasoning as a docstring for each branch, not just the return value.

Your personal operating system for career transition.

Fine-tuning vs RAG: decision framework