Transformers documentation

MXFP4

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.24.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

MXFP4

MXFP4 is the 4-bit floating-point format from the Open Compute Project Microscaling (MX) specification. It stores E2M1 values in blocks that share one power-of-two (E8M0) scale. In Transformers, MXFP4 quantizes the mixture-of-experts layers of GPT-OSS models, so use it to run GPT-OSS with less memory.

The table below summarizes what MXFP4 supports in Transformers.

ConfigMxfp4Config
quant_methodmxfp4
WorkflowQuantize while loading, load pre-quantized
Bits4
HardwareCUDA (compute capability 7.5+), XPU, CPU
Save✅
Fine-tune❌
Auto-compile❌

Install MXFP4

Transformers downloads the MXFP4 Triton kernels from the kernels-community/gpt-oss-triton-kernels repository on the Hub. Install the kernels library, Accelerate, and Triton. CUDA needs Triton 3.4.0 or later, and XPU and CPU need Triton 3.5.0 or later.

pip install -U "transformers[kernels]" accelerate "triton>=3.4.0"

Quantize a model with MXFP4

Create a Mxfp4Config and pass it to from_pretrained() to quantize a bf16 GPT-OSS checkpoint while it loads. MXFP4 quantizes only the expert weights. The attention layers, router, embeddings, and lm_head keep their original precision.

from transformers import AutoModelForCausalLM, Mxfp4Config

quantization_config = Mxfp4Config()
model = AutoModelForCausalLM.from_pretrained(
    "unsloth/gpt-oss-20b-BF16",
    quantization_config=quantization_config,
    device_map="cuda",
)

If the hardware or packages MXFP4 needs aren’t available, set dequantize=True to load the model in bf16 instead.

Load a pre-quantized checkpoint

The openai/gpt-oss-20b and openai/gpt-oss-120b checkpoints are stored in MXFP4, and their config sets quant_method to mxfp4. Load them without a Mxfp4Config.

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", device_map="cuda")

If your hardware or packages can’t run MXFP4, Transformers logs a warning and dequantizes a pre-quantized checkpoint to bf16 instead of raising an error.

Choose a kernel backend

By default, MXFP4 runs the expert layers with the Triton kernels on CUDA and XPU. Pass use_kernels=True to from_pretrained() to run them with the MegaBlocks kernel from the Hub instead.

model = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", device_map="cpu", use_kernels=True)

The backend you get depends on the device.

Deviceuse_kernels=Falseuse_kernels=True
CUDAMXFP4 with Triton kernelsDequantizes to bf16
XPUMXFP4 with Triton kernelsMXFP4 with MegaBlocks
CPUDequantizes to bf16MXFP4 with MegaBlocks

Limitations

  • MXFP4 quantizes only GPT-OSS expert layers. Other models load without any quantized layers, and Transformers logs a warning.
  • MXFP4 needs a CUDA GPU with compute capability 7.5 or later, an XPU, or a CPU. Quantizing a model on unsupported hardware or without Triton or kernels raises a ValueError. Quantizing a model on a machine with another accelerator, like MPS, raises a RuntimeError.
  • Quantizing a model while it loads can’t offload to disk. A device_map that includes "disk" raises a ValueError.
  • MXFP4 models can’t be fine-tuned. Load the model with dequantize=True to train it in bf16.

Resources

Update on GitHub