AirLLM

AirLLM is an open-source Python inference library designed to run large-scale open-source language models even on low-end computers. Instead of loading the entire model into GPU memory, it loads only one layer at a time and streams the necessary parts from the disk, significantly reducing memory usage to allow running Llama 70B-class models on a single 4GB GPU. Various model families, such as Qwen, Llama, DeepSeek, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, and InternLM, can be loaded using only their Hugging Face repository IDs, and text generation inference is performed through the AutoModel interface in a manner similar to Transformers. It also supports 4-bit and 8-bit block-wise compression, prefetching, CPU inference, unsharded models, and running on Apple Silicon-based macOS. Its primary targets are AI researchers, ML engineers, open-source developers, and individual users with limited VRAM environments who want to study large models or experiment locally. However, execution speed, disk space, and dependencies on CUDA, PyTorch, and specific models may vary depending on the usage environment.