LoRA (Low-Rank Adaptation)

What does LoRA mean in AI?

LoRA stands for low-rank adaptation. It was introduced by Edward Hu and colleagues at Microsoft in a 2021 paper. When people talk about "a LoRA" or "LoRAs," especially in image generation communities, they mean the small trained add-on file that adapts a base model to a style, subject, or task.

Not to be confused with LoRa (Long Range), the low-power wireless radio technology used in Internet of Things devices. Same letters, unrelated field.

How LoRA works

The analogy: rather than rewriting a whole textbook to teach a new specialty, you add a short set of margin notes that adjust how it is read. The book stays the same; the notes change the result.

The precise version: full fine-tuning updates every parameter in the model. LoRA keeps the pretrained weights frozen and injects pairs of small, trainable low-rank matrices into the transformer layers. Only those matrices are trained. Afterward they can be merged into the original weights, so, unlike some other adapter methods, LoRA adds no extra delay when the model runs.

How much more efficient is LoRA?

In the original paper, Hu et al. reported that, compared with fully fine-tuning GPT-3 175B with the Adam optimizer, LoRA cut the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times, while performing on par with or better than full fine-tuning in model quality on the tasks they tested.

A later method, QLoRA, combined LoRA with 4-bit quantization and reported fine-tuning a 65-billion-parameter model on a single 48GB GPU.

A real example of LoRA

A company wants an open-weights model to write product descriptions in its house style. Instead of fully retraining the model, it trains a LoRA on a few thousand of its own approved descriptions. The base model stays untouched, and the company can keep separate LoRAs for different brands and load whichever one a task needs.

Why LoRA matters and when to use it

LoRA made customizing large models practical on modest hardware and budgets. Use it when prompting alone cannot lock in a behavior, format, or style, and you have good examples to train on. If the problem is missing knowledge, such as your current policies or prices, RAG is usually the better fit.

Related terms

• Fine-tuning: the broader process LoRA makes cheaper.
• Quantization: often combined with LoRA to fit training on smaller hardware.
• Open weights: you need access to a model's weights to train a LoRA for it.
• Foundation model: the base model a LoRA adapts.

Common misconceptions about LoRA

• A LoRA is a separate model. It is an add-on that only works with the base model it was trained for.
• LoRA teaches a model new facts reliably. It is best at shaping style and behavior; for changing facts, retrieval is usually more dependable.
• LoRA always matches full fine-tuning. The paper found parity on the tasks tested, but results depend on the task, data, and settings.

LoRA in practice

Most small businesses never need to train a LoRA; good prompts and retrieval handle most needs. Start with our prompting guide, and if you are weighing customization options, see how RAG compares with fine-tuning.