What is the difference between fine-tuning and prompt engineering?

Prompt engineering guides a model at runtime with instructions; fine-tuning trains a new model checkpoint on your examples to bake in behaviour permanently - far more expensive but more consistent.

AILLMFine-tuning

Overview

Both techniques customise LLM behaviour, but at different stages and costs. Prompt engineering changes what you send to the model at inference time. Fine-tuning changes the model's weights by training on your examples, producing a new model that behaves differently by default. Most production systems start with prompt engineering and only reach for fine-tuning when they have abundant labelled examples and a clear quality gap.

Key Points

•**Prompt engineering**: free, fast to iterate, but behaviour lives in the prompt and can "drift" if the prompt is modified or the model is updated.
•**Fine-tuning**: trains a new model on hundreds to thousands of (input, output) pairs. The behaviour is baked in - shorter prompts, lower latency, higher consistency.
•Fine-tuning is NOT a way to add new knowledge - use RAG for that. Fine-tuning teaches style, format, and tone.
•Fine-tuning requires a supervised dataset: hundreds of high-quality example (prompt, ideal completion) pairs.
•Cost: OpenAI fine-tuning costs ~$8/M tokens for training + higher inference cost. Often better to first try a strong system prompt.
•**RAG vs fine-tuning**: RAG for "what the model knows"; fine-tuning for "how the model behaves".

Examples

Decision tree: which approach to use

text

Problem                           → Approach
──────────────────────────────────────────────────────────
Need domain knowledge (docs/data) → RAG
Need specific output format        → Prompt engineering first
Need consistent tone/persona       → Prompt engineering (system prompt)
Prompt engineering isn't reliable  → Fine-tuning
Need low latency (shorter prompts) → Fine-tuning
Labelled examples < 100            → Prompt engineering (few-shot)
Labelled examples > 500            → Fine-tuning is viable
Knowledge changes frequently       → RAG (no re-training needed)

Fine-tuning training data format (JSONL)

json

// Each line in the training file is a complete conversation example
{"messages": [
  {"role": "system",    "content": "You are a customer support agent for Acme Corp. Be concise and friendly."},
  {"role": "user",      "content": "How do I reset my password?"},
  {"role": "assistant", "content": "To reset your password: go to Settings → Security → Reset Password. You'll receive an email within 2 minutes. Let me know if you need help!"}
]}

{"messages": [
  {"role": "system",    "content": "You are a customer support agent for Acme Corp. Be concise and friendly."},
  {"role": "user",      "content": "Can I get a refund?"},
  {"role": "assistant", "content": "Yes! We offer full refunds within 30 days of purchase. To start the process, visit acme.com/refunds or reply here with your order number."}
]}

You need at least 10 examples to fine-tune, but 100–500 high-quality examples produce meaningfully better results.

Uploading and running a fine-tuning job (OpenAI)

typescript

import OpenAI from 'openai';
const client = new OpenAI();

// 1. Upload training data
const file = await client.files.create({
  file: fs.createReadStream('training.jsonl'),
  purpose: 'fine-tune',
});

// 2. Start fine-tuning job
const job = await client.fineTuning.jobs.create({
  training_file: file.id,
  model: 'gpt-4o-mini-2024-07-18',
});

console.log('Job ID:', job.id);  // ft-job-abc123

// 3. Poll until complete (takes 15min–several hours)
const result = await client.fineTuning.jobs.retrieve(job.id);
console.log('Fine-tuned model:', result.fine_tuned_model);
// ft:gpt-4o-mini-2024-07-18:myorg::abc123

// 4. Use the fine-tuned model
const response = await client.chat.completions.create({
  model: result.fine_tuned_model,
  messages: [{ role: 'user', content: 'How do I reset my password?' }],
});

Pro Tip

Before investing in fine-tuning, run an A/B test: measure whether a well-crafted system prompt with 5–10 few-shot examples already meets your quality bar. In practice, strong prompt engineering closes 80–90% of quality gaps at near-zero cost. Fine-tune only when the last 10–20% of quality matters and you have the data to support it.