Transformers documentation

Metal

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.23.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Metal

Metal quantization packs weights into uint32 tensors with an affine scale and bias for each group, and runs a fused dequantize and matmul kernel on the Apple silicon GPU. The kernels come from MLX and load from the Hub (kernels-community/mlx-quantization-metal-kernels) through the kernels library. Use it to run 2-, 4-, or 8-bit models on a Mac with MPS.

The table below summarizes what Metal supports in Transformers.

ConfigMetalConfig
quant_methodmetal
WorkflowQuantize while loading
Bits2, 4, 8
HardwareApple silicon (MPS)
Save✅
Fine-tune❌
Auto-compile❌

Install Metal

Install Transformers with the kernels extra. The Metal kernels download from the Hub the first time a quantized layer runs.

pip install -U "transformers[kernels]"

Quantize a model with Metal

Create a MetalConfig and pass it to from_pretrained(). Set device_map="mps" so the model runs on the Apple silicon GPU. The example below quantizes every linear layer to 4 bits in groups of 64 weights.

from transformers import AutoModelForCausalLM, MetalConfig

quantization_config = MetalConfig(bits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    quantization_config=quantization_config,
    device_map="mps",
)

A checkpoint saved with Metal quantization stores the packed weights, scales, and biases. Loading it with from_pretrained() reads them directly without quantizing again.

Configure Metal

These MetalConfig parameters change how the weights are quantized.

ParameterDefaultEffect
bits4Number of bits per weight. Supports 2, 4, and 8.
group_size64Number of weights that share a scale and bias. Must be positive.
modules_to_not_convertNoneModules to leave unquantized.

Load a Metal checkpoint without Apple silicon

If you load a Metal checkpoint on a machine without MPS, Transformers logs a warning and dequantizes the weights back to floating point, so the model runs as a regular model on the CPU or a CUDA GPU. To dequantize on any machine, set dequantize=True. The example below loads the model from Quantize a model with Metal after it was saved with model.save_pretrained("Qwen3-0.6B-metal").

from transformers import AutoModelForCausalLM, MetalConfig

model = AutoModelForCausalLM.from_pretrained(
    "Qwen3-0.6B-metal",
    quantization_config=MetalConfig(dequantize=True),
    device_map="cpu",
)

dequantize only applies when loading. It isn’t saved in the checkpoint’s config.

Limitations

  • Quantizing a model needs an MPS device. Quantizing without one raises a RuntimeError.
  • Loading without the kernels package installed raises an ImportError.
  • Quantizing a model with "cpu" or "disk" in its device_map raises a ValueError.

Resources

Update on GitHub