Skip to content

What Is Fine-Tuning in AI? LoRA, QLoRA and When to Fine-Tune vs Use RAG

Fine-tuning further trains a pre-trained AI model on your own examples so it follows a format, style or task better. How it works, full fine-tuning vs LoRA and QLoRA, how much GPU memory you need, and when RAG or prompting is the better choice.

5 min read

What Is Fine-Tuning? — cover image from TensorCore.si

Short answer

Fine-tuning means taking a model that has already been pre-trained on a huge general dataset and training it a little more on a smaller, specific dataset so it gets better at a particular task, style or format. Modern methods such as LoRA and QLoRA train only a tiny set of extra weights, so a 7B or 8B model can be fine-tuned on a single 16 to 24 GB GPU. Fine-tuning is best for teaching behaviour and format; for adding up-to-date facts, RAG is usually the better tool.

How fine-tuning fits into model training

Most language models go through several stages:

1. Pre-training: the model learns language and general knowledge by predicting the next token over trillions of tokens. This is the expensive part (see AI training vs inference). 2. Supervised fine-tuning (SFT): the model is trained on examples of prompts and ideal responses, which turns a raw text predictor into a helpful assistant. 3. Preference tuning: methods such as RLHF (reinforcement learning from human feedback) or DPO (direct preference optimization) teach the model which of two answers people prefer.

When companies and developers talk about fine-tuning, they usually mean doing step 2, and sometimes step 3, again on their own data, starting from an existing open model or through a provider's fine-tuning API.

What fine-tuning is good for

  • Consistent format: always returning valid JSON, a specific report layout or a fixed tone.
  • Domain style and terminology: legal drafting, medical coding conventions, a brand voice.
  • Narrow tasks: classification, extraction, routing or tagging, where a small fine-tuned model can match a much larger general model.
  • Shorter prompts: behaviour you would otherwise describe in a long system prompt gets built into the model, saving tokens.
  • Smaller, cheaper models: fine-tuning a small model on outputs from a large one (distillation) cuts serving costs.

What fine-tuning is not good for

  • Adding fresh facts: knowledge learned by fine-tuning is hard to update and the model may still hallucinate details. Use RAG for documents, prices and anything that changes.
  • Citing sources: a fine-tuned model cannot point to where a fact came from.
  • Fixing a weak base model: fine-tuning sharpens existing abilities more than it creates new ones.

Fine-tuning vs RAG vs prompt engineering

Prompt engineeringRAGFine-tuning
What changesThe instructionsThe information suppliedThe model's weights
Best forQuick behaviour changesUp-to-date or private knowledgeStyle, format, narrow tasks
Setup effortMinutesDays (index, embeddings, retrieval)Days to weeks (data, training, evaluation)
Needs GPUsNoFor embeddings and serving onlyYes, for training
UpdatingEdit the promptRe-index documentsRetrain
Cites sourcesNoYesNo

A practical order is: try better prompts first, add RAG if the model lacks knowledge, and fine-tune when you need consistent behaviour that prompting cannot deliver. Many production systems combine RAG and fine-tuning.

Full fine-tuning vs parameter-efficient fine-tuning

Full fine-tuning

All of the model's weights are updated. This gives the most control but needs the same memory as training: about 16 bytes per parameter with mixed precision and the Adam optimizer, so a 7B model needs over 100 GB plus activations. That means several data center GPUs, and you must store a full new copy of the model for every fine-tune.

LoRA (Low-Rank Adaptation)

LoRA freezes the original weights and adds small trainable matrices to selected layers. Instead of updating a large weight matrix directly, it learns two thin matrices whose product represents the change. The LoRA paper reported cutting the number of trainable parameters for GPT-3 by 10,000 times and GPU memory needs by about 3 times, with quality on par with full fine-tuning. The resulting "adapter" is often only tens or hundreds of megabytes and can be swapped on and off the base model.

QLoRA

QLoRA goes further by loading the frozen base model in 4-bit precision and training LoRA adapters on top. Its authors fine-tuned a 65-billion-parameter model on a single 48 GB GPU. QLoRA is the most common way hobbyists and small teams fine-tune models today. For more on 4-bit formats, see LLM quantization.

How much GPU memory do you need?

Rough guide for fine-tuning language models (exact needs depend on sequence length, batch size and settings):

Model sizeQLoRALoRA (16-bit base)Full fine-tuning
1B to 3B6 to 8 GB12 to 16 GB24 to 48 GB
7B to 8B12 to 16 GB24 GB+100 GB+ (multi-GPU)
13B to 14B16 to 24 GB40 GB+200 GB+
70B48 GB+160 GB+Over 1 TB (cluster)

A 24 GB card such as the RTX 4090 or RTX 5090 covers most hobby fine-tuning; see the best GPU for local LLMs. For bigger jobs, renting an H100 or A100 by the hour is usually cheaper than buying; check today's rates in our GPU cloud price tracker.

The fine-tuning process, step by step

1. Define the task and success metric. What exactly should improve, and how will you measure it? 2. Collect examples. Hundreds to a few thousand high-quality prompt-response pairs often beat a larger, messy dataset. 3. Format the data in the chat template your base model expects. 4. Choose a base model that already does reasonably well on the task. 5. Train with a library such as Hugging Face TRL and PEFT, Unsloth or Axolotl, or with a provider's fine-tuning API. 6. Evaluate on held-out examples the model never saw, and compare against the base model with a good prompt. 7. Deploy by merging the adapter or loading it at inference time, and keep monitoring.

Risks to watch

  • Overfitting: the model memorises training examples and generalises poorly.
  • Catastrophic forgetting: it gets worse at things it used to do well.
  • Safety regression: research has shown fine-tuning can weaken a model's safety training, even with harmless data, so re-test safety behaviour.
  • Licences: check the base model's licence allows your use.

Frequently asked questions

How many examples do I need to fine-tune?

For format and style, a few hundred good examples can be enough. Complex tasks may need thousands. Quality and consistency matter more than volume.

Can I fine-tune ChatGPT or Claude?

OpenAI offers fine-tuning for several of its models through its API, and some providers offer fine-tuning on their cloud platforms. Open-weight models such as Llama, Mistral, Qwen and Gemma can be fine-tuned on your own hardware.

Is fine-tuning the same as training from scratch?

No. Training from scratch builds all of a model's knowledge from random weights and costs millions of dollars for large models. Fine-tuning starts from a capable model and adjusts it, usually in hours.

Does fine-tuning make a model smarter?

It makes a model better at the specific task you trained it on. It rarely improves general reasoning and can sometimes reduce performance on unrelated tasks.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *