What is NVFP4? NVIDIA's 4-bit format for running large models on Blackwell

NVFP4 is NVIDIA's 4-bit number format for Blackwell GPUs: how it works, how much memory it saves, what it does to accuracy and how to run it.

Two liquid-cooled NVIDIA GB200 boards with copper cold plates at COMPUTEX 2024, Blackwell hardware with native FP4 support
Photo: Eindrücke von der COMPUTEX 2024 ( 极客湾Geekerwan) 22 by 极客湾Geekerwan, Wikimedia Commons, CC BY 3.0

NVFP4 is a 4-bit floating point format that NVIDIA built into its Blackwell GPUs. Each weight takes 4 bits, and every block of 16 weights shares one small scale factor, so a model needs about 3.5 times less memory than in FP16 and about 1.8 times less than in FP8. NVIDIA reports that DeepSeek-R1 loses 1% or less accuracy on key tasks when moved from FP8 to NVFP4. The catch: it runs natively only on Blackwell hardware, and your inference software has to support it.

Which format for whom

  • You have a Blackwell GPU and want the biggest model or the most users per card → NVFP4. It roughly halves memory against FP8 and uses the fastest Tensor Core path.
  • You need the safest accuracy and the model fits anyway → FP8. FP8 is mature in vLLM, SGLang and TensorRT-LLM, and runs on Hopper and Blackwell.
  • You run on older GPUs (Ampere, Ada, Hopper) → INT4 methods such as AWQ or GPTQ, or GGUF. These give similar memory savings but without native FP4 Tensor Core math.
  • You fine-tune or research model quality → BF16. Quantize only the copy you serve.
  • You want to serve 70B to 400B models in 4-bit for a team → an 8-GPU Blackwell server. Our server in Sofia has eight RTX PRO 6000 Blackwell Server Edition cards with 768 GB of GPU memory in total. It rents from $1.80 per GPU-hour, or about $10,510 a month for the whole machine. It is rented into early 2027 and we take reservations for the period after. See the server.

How NVFP4 works

A 4-bit number on its own can hold only a handful of values. NVFP4 uses the E2M1 layout: 1 sign bit, 2 exponent bits and 1 mantissa bit. That is far too coarse to describe the weights of a model directly, so the format adds scaling at two levels:

  1. A block scale. Every 16 consecutive values share one scale factor stored in FP8 (E4M3). The scale stretches the tiny 4-bit range to fit the real values in that block.
  2. A tensor scale. A second scale in FP32 covers the whole tensor, so the FP8 block scales stay within their range.

With one 8-bit scale per 16 values, the cost works out to about 4.5 bits per value, according to NVIDIA.

The older open format MXFP4 uses blocks of 32 values and a scale that can only be a power of two (E8M0). NVFP4 uses smaller blocks and a scale with a fractional part. Both choices let the scale follow the data more closely, which is where the accuracy gain comes from.

How much memory it saves

Weights are only part of the picture, but they are the largest part. A rough count for a 70 billion parameter model, weights only:

Format Bits per weight Weights of a 70B model
BF16 / FP16 16 about 140 GB
FP8 8 about 70 GB
NVFP4 about 4.5 about 39 GB

On top of the weights you need room for the KV cache, which grows with context length and the number of users at once. That is why a 4-bit model is not just "smaller": the memory it frees goes to longer contexts and more parallel requests. On a single 96 GB RTX PRO 6000, a 70B model in NVFP4 leaves more than half the card for the cache. In FP8 the same model leaves about a quarter. For the full method, see how much VRAM an LLM needs.

What it does to accuracy

Any 4-bit format loses some precision. The question is how much. NVIDIA published one detailed comparison: DeepSeek-R1-0528 quantized after training (post-training quantization) from FP8 to NVFP4. The scores moved by one point or less on most tests:

Benchmark FP8 NVFP4
MMLU-PRO 85 84
GPQA Diamond 81 80
LiveCodeBench 77 76
Math-500 98 98
AIME 2024 89 91

These are NVIDIA's own numbers on one model. Smaller models usually suffer more from 4-bit quantization than very large ones, so test your own model on your own tasks before you switch. A short evaluation set of real prompts tells you more than any public benchmark.

Which GPUs run NVFP4

NVFP4 needs the fifth-generation Tensor Cores of the Blackwell architecture. That covers the data center parts (B200, B300, GB200, GB300) and the RTX PRO 6000 Blackwell. NVIDIA lists the RTX PRO 6000 Blackwell Server Edition at 4 PFLOPS of FP4 Tensor Core performance, against 2 PFLOPS in FP8 and 1 PFLOP in FP16/BF16.

Hopper (H100, H200) and older cards have no FP4 Tensor Cores. Software can still load a 4-bit model on them, but the math does not run at FP4 speed. For a side-by-side of the cards, see B300 vs H200 vs RTX PRO 6000.

How to run a model in NVFP4

There are two paths.

Download a ready checkpoint. NVIDIA publishes NVFP4 versions of popular models on Hugging Face, for example Llama-3.3-70B-Instruct-FP4 and DeepSeek-R1-0528-FP4. NVIDIA's NIM catalog also lists NVFP4 profiles built for the RTX PRO 6000 Blackwell.

Quantize your own model. NVIDIA's TensorRT Model Optimizer and the open source LLM Compressor can produce NVFP4 weights from a BF16 model. You need a small calibration dataset, ideally prompts close to your real use.

Then serve it with TensorRT-LLM, vLLM or SGLang. Support for NVFP4 arrived at different times in each framework and differs between GPU generations, so use a recent version and check its release notes for your card. Our comparison of vLLM, Ollama and SGLang covers which server suits which job.

When NVFP4 is not worth it

  • The model already fits in FP8 with room for the cache you need, and accuracy matters more than throughput.
  • Your framework or model architecture does not support NVFP4 yet on your GPU.
  • You run on Hopper or older hardware, where you get the memory saving but not the speed.

In those cases FP8 stays the simpler choice. For private model hosting on dedicated hardware in the EU, see private LLM hosting.

Questions

Is NVFP4 the same as INT4? No. INT4 stores whole numbers with a scale. NVFP4 stores tiny floating point numbers (E2M1) with an FP8 scale for every 16 values, and Blackwell Tensor Cores compute with it directly.

Is NVFP4 the same as MXFP4? No. MXFP4 is an open format with blocks of 32 and power-of-two scales. NVFP4 uses blocks of 16 and FP8 scales, which NVIDIA says gives better accuracy.

Can I use NVFP4 for training? NVIDIA has published research on training with NVFP4, but for most teams today it is an inference format. Train or fine-tune in BF16, then quantize the copy you serve.

Does the RTX PRO 6000 support NVFP4? Yes. It is a Blackwell GPU with fifth-generation Tensor Cores and FP4 support, rated at 4 PFLOPS of FP4.

Sources

SHARE