Transformers documentation
Metal
Metal
Metal quantization packs weights into uint32 tensors with an affine scale and bias for each group, and runs a fused dequantize and matmul kernel on the Apple silicon GPU. The kernels come from MLX and load from the Hub (kernels-community/mlx-quantization-metal-kernels) through the kernels library. Use it to run 2-, 4-, or 8-bit models on a Mac with MPS.
The table below summarizes what Metal supports in Transformers.
| Config | MetalConfig |
quant_method | metal |
| Workflow | Quantize while loading |
| Bits | 2, 4, 8 |
| Hardware | Apple silicon (MPS) |
| Save | ✅ |
| Fine-tune | ❌ |
| Auto-compile | ❌ |
Install Metal
Install Transformers with the kernels extra. The Metal kernels download from the Hub the first time a quantized layer runs.
pip install -U "transformers[kernels]"Quantize a model with Metal
Create a MetalConfig and pass it to from_pretrained(). Set device_map="mps" so the model runs on the Apple silicon GPU. The example below quantizes every linear layer to 4 bits in groups of 64 weights.
from transformers import AutoModelForCausalLM, MetalConfig
quantization_config = MetalConfig(bits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-0.6B",
quantization_config=quantization_config,
device_map="mps",
)A checkpoint saved with Metal quantization stores the packed weights, scales, and biases. Loading it with from_pretrained() reads them directly without quantizing again.
Configure Metal
These MetalConfig parameters change how the weights are quantized.
| Parameter | Default | Effect |
|---|---|---|
bits | 4 | Number of bits per weight. Supports 2, 4, and 8. |
group_size | 64 | Number of weights that share a scale and bias. Must be positive. |
modules_to_not_convert | None | Modules to leave unquantized. |
Load a Metal checkpoint without Apple silicon
If you load a Metal checkpoint on a machine without MPS, Transformers logs a warning and dequantizes the weights back to floating point, so the model runs as a regular model on the CPU or a CUDA GPU. To dequantize on any machine, set dequantize=True. The example below loads the model from Quantize a model with Metal after it was saved with model.save_pretrained("Qwen3-0.6B-metal").
from transformers import AutoModelForCausalLM, MetalConfig
model = AutoModelForCausalLM.from_pretrained(
"Qwen3-0.6B-metal",
quantization_config=MetalConfig(dequantize=True),
device_map="cpu",
)dequantize only applies when loading. It isn’t saved in the checkpoint’s config.
Limitations
- Quantizing a model needs an MPS device. Quantizing without one raises a
RuntimeError. - Loading without the kernels package installed raises an
ImportError. - Quantizing a model with
"cpu"or"disk"in itsdevice_mapraises aValueError.
Resources
- MLX repository
- kernels-community/mlx-quantization-metal-kernels on the Hub