Overview
AirLLM is a Python library for running very large language models on hardware that couldn't normally hold them. The README says a 70B model runs on a single 4GB GPU, and reports measured runs of much larger models in a few gigabytes of VRAM. It works by streaming model layers through the GPU one at a time. Recent releases add FP8 models and memory-efficient training.
Key capabilities
- Layer streaming: run models far larger than your GPU memory.
- AutoModel: load most popular models by Hugging Face repo ID.
- Optional compression: 4-bit and 8-bit block-wise quantization for up to 3x faster inference.
- macOS and CPU support.
- Training: stream frozen weights and keep adapters on the GPU (added 2026).
Tradeoffs
- Much slower than in-memory inference; not suited to real-time chat.
- Needs plenty of disk space for split model layers.
- Memory figures are the project's own measurements.
Ideal for
- Trying large open models on a consumer GPU
- Offline batch inference
- Research on limited hardware
Not ideal for
- Low-latency production serving
- Interactive assistants
Getting started
- Install with
pip install airllm. - Load a model by Hugging Face repo ID with
AutoModel.from_pretrained(...). AirLLM splits it into layers on first load, so allow time and disk space. - Run generation as you would with a normal transformers model, and expect it to be slower than in-memory inference.
- If speed matters more than exact quality, try the 4-bit or 8-bit compression option.
Technical notes
- Install:
pip install airllm - Language: Python
- License: Apache-2.0
- Source: lyogavin/airllm
- GitHub stars: 42,223 (as of 2026-11-12)