LoRA

AI concepts
About 1 min read

An efficient fine-tuning (PEFT) technique that drastically reduces training parameters and hardware requirements by freezing the existing weights of a large model and adding only trainable low-rank matrices.

Also known as
Low-Rank AdaptationLoRA

Detailed explanation

LoRA (Low-Rank Adaptation) is a technique for optimizing large language models (LLMs) or image generation models for specific purposes by training only very small 'adapter' matrices instead of modifying the entire model. By freezing the existing weights and inserting and training two small matrices using matrix decomposition principles, it achieves performance comparable to full fine-tuning while training less than 0.1% to 1% of the total parameters. The resulting adapter file is lightweight—typically tens of megabytes—making it easy to share and manage, and its strength lies in the fact that merging it with the base model during deployment introduces zero inference latency.

Why it matters in tool selection

LoRA is a key technology for the 'democratization of AI', enabling the customization and training of large models on personal PCs (consumer-grade GPUs) without the need for high-end servers. It is highly cost-effective for operations, as you can swap multiple LoRA adapters on a single base model to perform various tasks (translation, summarization, mimicking specific styles, etc.).

What to check when selecting and configuring

  • Rank (r) setting: A larger value improves performance but increases memory usage. A value between 8 and 64 is typically recommended.
  • Target Modules: While mainly applied to attention layers, recent research shows that including MLP layers yields better performance.
  • Alpha (α) value: A scaling factor for training stability, typically standardized to be set to twice the r value (r*2).
  • Merging capability: To prevent performance degradation during real-time inference, check if the weights can be merged with the base model after training is complete.

Use Cases

Typical examples include creating a 'consultation specialized LoRA' by training a model on a specific company's internal document style based on the Llama-3 model, or downloading and using LoRA files trained on a art style or character for Stable Diffusion models from sharing sites (such as Civitai).

Comparison of Variant Techniques

QLoRA

A technique that combines LoRA with 4-bit quantization to drastically reduce memory usage even further.

DoRA

A recent technique that decomposes weights into magnitude and direction to improve training performance closer to full fine-tuning.

Related terms

Fine-tuning