Transformers documentation
Quanto
Get started
Base classes
Models
Preprocessors
Inference
Pipeline API
Generate API
Optimization
Chat with models
Serving
Training
Hardware
Quantization
OverviewQuantization conceptsChoose a method OptimumContribute a quantization method
Common tasks
Quantize while loading
bitsandbytesEETQFBGEMM FP8Fine-grained FP8Four Over SixFP-QuantGPTQHIGGSHQQMetalMXFP4NVFP4QuantoSINQtorchao
Load pre-quantized checkpoints
Ecosystem integrations
Resources
API
You are viewing main version, which requires installation from source. If you'd like
regular pip install, checkout the latest stable version (v5.23.0).
Quanto
Quanto is a PyTorch quantization library from Optimum. It replaces linear layers with quantized versions built on PyTorch, so it works on any device PyTorch supports and with any model that has torch.nn.Linear layers, including speech and vision models. Use it to quantize weights to 2, 4, or 8 bits on hardware other methods don’t support, like MPS or CPU.
The table below summarizes what Quanto supports in Transformers.
| Config | QuantoConfig |
quant_method | quanto |
| Workflow | Quantize while loading |
| Bits | 2, 4, 8 |
| Hardware | CUDA, MPS, XPU, CPU |
| Save | ❌ |
| Fine-tune | ✅ |
| Auto-compile | ❌ |
Install Quanto
Install optimum-quanto and Accelerate.
pip install -U optimum-quanto accelerate
Quantize a model with Quanto
Create a QuantoConfig and pass it to from_pretrained(). weights sets the weight dtype and accepts "int8" (the default), "float8", "int4", or "int2".
from transformers import AutoModelForCausalLM, QuantoConfig
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-8B",
quantization_config=QuantoConfig(weights="int4"),
device_map="auto",
)Limitations
- Transformers quantizes only the weights. Setting
activationsin QuantoConfig raises aValueError. Use the Quanto library directly for activation quantization, calibration, or quantization-aware training. - Transformers can’t save Quanto models. The Quanto library has its own way to save and reload quantized models.
- Quantizing while loading can’t offload to the CPU or disk. A
device_mapthat places part of the model on"cpu"or"disk"raises aValueError.
Resources
- Quanto: a PyTorch quantization backend for Optimum blog post
- optimum-quanto repository