Four Blackwell Formats, One Model: Quantizing Qwen3-32B on RTX PRO 6000

For a long time, dropping a model from high-bit to low-bit precision came with a real cost. You saved memory, but the quality usually took a hit and the speedup was small or missing. So low precision felt like a trade-off you accepted, not a win.

NVIDIA's Blackwell chips change that. Native 8-bit floating point isn't new: Hopper (H100) and Ada (L40S) GPUs already had FP8 tensor cores. What Blackwell adds is native FP4 (and FP6) math and hardware support for microscaling (MX) block formats, so 4-bit floating point runs natively instead of being faked in software. The lineup goes from the RTX PRO 6000 Blackwell Server Edition up to the data-center B200 / GB200 systems. This matters because on older hardware, quantization was mostly a way to make a model small enough to fit in memory — any speedup was a bonus, and usually a small one. On Blackwell, the low-bit formats run on dedicated hardware, so moving to FP8 or FP4 gives you real, measurable speed, not just less memory use.

All the tests in this post ran on an RTX PRO 6000 Blackwell Server Edition. The bigger B200 cards are built to speed up the same formats we test here, so we'd guess the order we find holds up on that hardware too — but we haven't measured it.

This makes picking a format an important choice, and no longer an obvious one. FP8, MXFP8, NVFP4, and MXFP4 all shrink the model by about the same amount as before, but on Blackwell each one behaves very differently once you actually serve real traffic. Pick the wrong one and you leave a big chunk of the GPU unused.

What this article does. We take one model — Qwen3-32B — and quantize it into four Blackwell floating-point formats using AWQ: FP8, MXFP8, NVFP4 and MXFP4. We left out FP6 / MXFP6 and FP8 E5M2 (E5M2 is mainly used for gradients in training; inference checkpoints use E4M3). Then we measure two things that matter in production:

• Quality: Wikitext perplexity and GSM8K accuracy, to see what each format costs you.
• Serving performance: output throughput and time-to-first-token across a range of concurrency levels, to see what each format gives you under real load.

The result is a fair, side-by-side comparison of BF16, FP8-AWQ, MXFP8-AWQ, NVFP4-AWQ, and MXFP4-AWQ on the same hardware — boiled down to three charts and one table you can act on. The rest of this post walks through both sides, starting with a short look at how the quantization works. For a deeper theoretical and hands-on introduction, see our earlier post, The vLLM Quantization Complete Guide + Benchmarks. It covers the well-known algorithms, including AWQ, GPTQ, BitsAndBytes, TorchAO, Marlin-AWQ, Marlin-GPTQ, BitBLAS and ModelOPT.

How the quantization works

All variants start from the same BF16 weights and are quantized to a smaller format. Fewer bits means less memory and faster math, but coarser numbers. Two ideas do the heavy lifting to keep quality high: AWQ decides which weights to protect, and a scale factor decides how each number gets squeezed into the smaller type. We'll build up both, then look at each format in turn.

AWQ: protect the weights that matter

Naively rounding every weight to a low-bit format hurts, because some weights carry an outsized share of the model's behaviour. AWQ (Activation-aware Weight Quantization) protects them by rescaling each weight channel before rounding.

The key observation is that a weight's importance isn't about the weight's own magnitude — it's about how large the activations flowing through it are. A channel multiplied by large, frequent activations matters far more than one multiplied by near-zero inputs. So AWQ works like this:

1. Calibration pass. Run a few hundred real text samples (the calibration dataset) through the BF16 model and record, for each linear layer, the average magnitude of the input activations per channel.
2. Search a scale for every channel. AWQ builds a per-channel scale from those activation magnitudes (roughly activation_magnitude ^ α). It tries a grid of α values, quantizes the layer with each, and keeps the one with the lowest output error versus the original layer. Channels with larger activations end up scaled up more.
3. Scale before rounding. AWQ multiplies each weight channel by its scale (and divides the corresponding activations down to compensate, so the math still produces the same result). Scaling the important weights up means that when they're rounded to the coarse grid, the relative rounding error on them shrinks.
4. Quantize. Now round the (rescaled) weights into the target low-bit format.

Because AWQ only reads activation statistics — never gradients — it's fast and needs little data.

From BF16 to FP8: the mechanics

A floating-point number is stored as three parts: a sign bit, some exponent bits (how big the number is), and some mantissa bits (the precision within that magnitude). BF16 uses 1 sign + 8 exponent + 7 mantissa bits. FP8 (E4M3) keeps 1 sign + 4 exponent + 3 mantissa bits — fewer exponent bits (so its largest value is only 448) and far fewer mantissa bits, so it can only represent a coarse grid of values.

The problem is that a whole weight tensor spans a wide range of magnitudes, but FP8's grid is narrow. If you cast directly, small weights collapse to zero and large ones saturate. The fix is a scale factor s, shared across a group of weights:

w_fp8      = round(w_bf16 / s)
w_restored = w_fp8 × s

You divide by s to bring the group into FP8's representable range, round to the nearest FP8 value, and multiply back by s at compute time to recover the original scale. The scale is chosen from the group's largest absolute value so nothing saturates.

For example, take four BF16 weights: [0.021, -0.008, 0.052, 0.011].

1. Pick the scale. The largest magnitude is 0.052, and FP8 E4M3's largest value is 448, so s = 0.052 / 448 ≈ 0.000116.
2. Divide by s. This stretches the weights onto FP8's range.
3. Round to the nearest E4M3 value. Near these magnitudes the representable values are 8 apart (64–128) or 16 apart (128–256).
4. Multiply back by s. This recovers weights on the original scale.

The restored weights land within a few percent of the originals. The scale is what lets 3 mantissa bits track weights of very different sizes.

FP8-AWQ uses block scaling: one scale per 128×128 block of weights, with activations quantized dynamically in groups of 128. It roughly halves the model versus BF16 (32.04 GiB vs 61.03 GiB in vLLM, about 1.9×) with almost no quality loss, because 8 bits still cover a wide dynamic range.

Microscaling: MXFP8 and MXFP4

The weakness of one scale for a big group is that a single outlier forces a large scale, which wastes precision on all the small values in that group. Microscaling (MX) fixes this by chopping the weights into tiny blocks (typically 32 values) where each block gets its own shared scale (a power-of-two, stored compactly as an 8-bit exponent, E8M0). Finer blocks adapt to local magnitudes, so precision stays high everywhere.

• MXFP8-AWQ — the microscaling flavour of FP8. Same 8-bit E4M3 elements as plain FP8, but with per-block (32-value) scales instead of 128×128 blocks. That extra locality keeps quality essentially at BF16 levels while still being an 8-bit format.
• MXFP4-AWQ — microscaling applied to a 4-bit float (E2M1: 1 sign, 2 exponent, 1 mantissa — only ~8 distinct magnitudes). Each 32-value block shares one power-of-two scale. This shrinks the weights about 3.3× (18.59 GiB vs 61.03 GiB), but with only a single mantissa bit per weight it takes a real quality hit. In our run it also did not use Blackwell's FP4 tensor cores: it ran on vLLM's Marlin weight-only kernel with BF16 activations (see the serving flags in the setup), so its speed here reflects that fallback kernel, not the MXFP4 format.

NVFP4: two-level scaling for 4-bit

NVFP4-AWQ is NVIDIA's 4-bit float, engineered to make 4 bits actually usable. It uses the same 4-bit E2M1 elements as MXFP4, but with two refinements:

1. Smaller blocks (16 values) rather than 32 — even finer locality.
2. Two levels of scaling. Each 16-value block gets a higher-precision FP8 scale (not just a power-of-two), and there's an additional global per-tensor FP32 scale on top. The higher-precision block scale means the tiny 4-bit numbers land much closer to their true values than MXFP4's coarse power-of-two scale allows.

The payoff is large as NVFP4 is about 3.1× smaller than BF16 (19.55 GiB vs 61.03 GiB; the scale factors add bits and some layers stay in BF16), and on Blackwell the native 4-bit tensor cores make it the fastest format we tested — for only a modest quality drop. It's the format Blackwell was really built to accelerate.

The pattern across all four is FP8 formats trade almost nothing for ~1.9× smaller weights; FP4 formats push to ~3.1–3.3× smaller and lean on smarter scaling (NVFP4's two-level scheme especially) to limit the quality cost.

Activations are quantized too. It's not only the weights: in these checkpoints, FP8 and MXFP8 also quantize the activations to 8 bits, and NVFP4 quantizes them to 4 bits. MXFP4 is the exception — because of the fallback kernel, its activations stayed in BF16 in our run. This affects both speed and quality.

All the Qwen3-32B quantized checkpoints were produced with LLM Compressor — Neural Magic / vLLM's toolkit for creating vLLM-ready quantized checkpoints. A single recipe-driven script handles the calibration pass and emits each target format. The exact script used for Qwen3-32B is available here: run_quant_32b.sh.

We have compressed Qwen3-32B into all four formats and published the checkpoints on Hugging Face:

• Qwen3-32B-FP8-AWQ-wikitext2
• Qwen3-32B-MXFP8-AWQ-wikitext2
• Qwen3-32B-NVFP4-AWQ-wikitext2
• Qwen3-32B-MXFP4-AWQ-wikitext2

Setup

• Model: Qwen3-32B
• Hardware: RTX PRO 6000 Blackwell Server Edition
• Software: vLLM 0.24.0, lm-eval 0.4.12, compressed-tensors 0.17.0
• Accuracy benchmarks: Wikitext perplexity (word PPL, WikiText-2 test split) and GSM8K (strict / flex) on the first 100 of the 1,319 test questions
• Serving benchmark: ShareGPT V3, 1000 prompts, all sent at once (--request-rate inf), single run per concurrency level
• Sampling: temperature 0.6, top-p 0.95, top-k 20 (Qwen3 defaults), EOS stopping on
• Client / server: vllm bench serve / vllm serve (vLLM)
• Max model length: 4096
• GPU memory utilization: 0.90
• KV cache capacity (tokens): BF16 93,312 · FP8 212,864 · MXFP8 208,192 · NVFP4 264,272 · MXFP4 267,200
• Prefix caching: on. One server handled the whole concurrency sweep with the same seeded prompts, so by the end the prefix cache hit rate was 0% for BF16, ~8% for FP8 and MXFP8, and ~14% for NVFP4 and MXFP4. This gives the quantized models a small extra boost.
• Quantization: calibration with 512 samples × 2048 tokens from the WikiText-2 train split, sharded across two GPUs (it ran out of memory on one). Perplexity is also measured on WikiText-2 (test split), so the same kind of text may make perplexity look a bit better than it would elsewhere.

Quantization variants compared: BF16 (baseline), FP8-AWQ, MXFP8-AWQ, NVFP4-AWQ, MXFP4-AWQ.

Serving flags. Two of the variants needed an extra environment flag on vllm serve to run correctly on this hardware; the other two ran with defaults:

We used the Marlin fallback for MXFP4 because, when we ran these experiments, vLLM's native MXFP4 path on Blackwell (the FlashInfer kernel) wasn't stable enough to run this checkpoint on vLLM 0.24.0.

Quality barely changes

ΔPPL is computed from the unrounded perplexities, so it can differ from subtracting the rounded values by 0.0001.

Observations:

• FP8-AWQ and MXFP8-AWQ are nearly lossless on perplexity (+0.03 and +0.06 PPL).
• The 4-bit variants (NVFP4, MXFP4) show a larger PPL movement (~+0.4), but it is still small in absolute terms. MXFP4's number isn't directly comparable with NVFP4's, since NVFP4 also quantized the activations and this MXFP4 run did not.
• GSM8K does not degrade in this run, but it was measured on only the first 100 questions, where the standard error is about ±3 points. Differences of a few points at that sample size are not statistically meaningful, so we treat the spread from 0.88 to 0.94 as noise rather than genuine "better reasoning."

The takeaway: quality is essentially preserved across the board. That frees us to choose a format based almost entirely on serving performance.

Quality vs speed: the hero chart

This single chart tells the whole story — how much quality you pay for how much throughput you gain. The x-axis is Wikitext ΔPPL versus BF16 (further left is better quality); the y-axis is peak output throughput (higher is better), taken at each variant's own best concurrency level, so this is peak against peak. The ideal position is top-left: fast and accurate.

Observations:

• FP8-AWQ and MXFP8-AWQ cluster in the top-left quality zone. Both sit within +0.06 PPL of BF16 while already delivering 1.63× and 1.73× the peak throughput (2,076 and 2,211 tok/s vs BF16's 1,276). They are the "almost-free" upgrades — you effectively pay nothing in quality for a two-thirds throughput boost.
• NVFP4-AWQ sits farthest up the chart by a wide margin, reaching 3,079 tok/s at no-limit, 2.41× BF16's peak of 1,276 tok/s at concurrency 420. The horizontal jump to +0.41 PPL looks large on this axis, but in absolute terms perplexity only moves from 10.21 to 10.61 — a modest cost for the single biggest throughput gain in the field.
• MXFP4-AWQ is the clear outlier and the weakest tradeoff in this test. It lands to the right of NVFP4 (worst quality, +0.44 PPL) yet only manages 1,715 tok/s — below FP8 and MXFP8 and roughly 1,360 tok/s short of NVFP4. It ran on the Marlin weight-only kernel with BF16 math, so it got the 4-bit memory savings but not the FP4 tensor-core speedup.

Throughput is where the story gets interesting

NVFP4-AWQ is the clear serving-performance winner, peaking at 2.41× BF16's peak output throughput.

Observations:

• NVFP4 is the clear throughput winner at every point on the curve. It opens at 2,004 tok/s at concurrency 60 — already above where BF16 ever reaches — and climbs to 3,078.91 output tok/s at no-limit. Its lead over the next-best variant ranges from 608 to 1,020 tok/s, a gap that never closes as load increases.
• MXFP8 and FP8 form a tight middle band, tracking within ~150 tok/s of each other across the whole curve. MXFP8 stays slightly ahead almost everywhere (e.g. 2,211 vs 2,076 at no-limit), making it the better of the two FP8-class formats for raw throughput.
• MXFP4 badly underperforms NVFP4 despite both being 4-bit formats. It is nearly flat — around 1,400–1,500 tok/s for most of the curve — and finishes at just 1,715 tok/s, closer to BF16 than to NVFP4. This is because MXFP4 ran on the Marlin weight-only kernel, which keeps activations in BF16 and does the math in BF16, so it never used Blackwell's FP4 tensor cores.
• BF16 saturates early and low, plateauing around 1.2K–1.3K output tok/s from concurrency 240 onward. Adding more concurrent requests buys it almost nothing because it runs out of KV cache memory: with 61 GiB of BF16 weights on the card, it has room for only ~93,000 tokens of KV cache, versus ~208,000–267,000 for the quantized models. Much of the quantized models' win at high load comes from the memory they free up, not only from faster math.

Latency matters more than peak tok/s

Peak throughput alone can be misleading — responsiveness under load matters just as much. On a log-scale time-to-first-token (TTFT) view, the separation is stark.

At concurrency 360:

Each point is a single run, and 360 is a high point for FP8 and MXFP8 (FP8 goes 1,574 → 1,828 → 1,624 tok/s at concurrency 300, 360 and 420; MXFP8 does the same), so read their throughput here as slightly flattering.

At concurrency 360, NVFP4 gives 2.18× BF16 throughput while cutting mean TTFT by roughly 7.6×.

Observations:

• BF16 hits the TTFT cliff first and hardest. Its mean TTFT explodes from ~1.7 s at concurrency 240 to 8.6 s at 360 and over 38 s at no-limit. This is the KV cache filling up: the BF16 server log shows KV cache usage hitting 99.9% with requests waiting in the queue, so new requests wait seconds just to emit their first token.
• NVFP4 pushes that cliff furthest out. It holds the lowest TTFT at every concurrency level — 1,132 ms at 360 versus BF16's 8,605 ms (a 7.6× reduction) — and even at no-limit stays at 7.4 s while BF16 is at 38 s, a 5.2× gap. With far more KV cache space and faster 4-bit math, more requests run at once and fewer wait in the queue, so latency degrades gracefully instead of collapsing.
• FP8 and MXFP8 track closely just above NVFP4, both around 2.0–2.1 s at concurrency 360 (~4× better than BF16). MXFP8 is marginally faster than FP8 at most points, mirroring its small throughput edge.
• MXFP4 is the worst of the quantized group, at 3,650 ms at concurrency 360 — better than BF16 but noticeably behind the FP8 pair and more than 3× slower than NVFP4. At lower load (concurrency 120, 180 and 240) its TTFT is actually worse than BF16's. That reflects the Marlin fallback kernel it ran on, not the MXFP4 format itself.

Because the y-axis is logarithmic, the vertical gaps represent multiplicative differences: the spread between NVFP4 and BF16 at high load is nearly an order of magnitude. Under heavy load, quantized variants keep TTFT low deeper into the concurrency curve, and NVFP4 keeps it lowest. Time per output token is more mixed: NVFP4 has the lowest TPOT everywhere, but at no-limit FP8 (374 ms), MXFP8 (368 ms) and MXFP4 (584 ms) all have higher mean TPOT than BF16 (264 ms).

Verdict

• NVFP4-AWQ is the overall winner. It delivers the highest throughput (3,078.91 tok/s, 2.41× BF16's peak) and the lowest latency across the load curve, for only a modest +0.41 PPL. If you are optimizing a serving deployment on Blackwell, this is the default choice.
• FP8-AWQ is the safe pick. Nearly lossless quality (+0.03 PPL) with a solid 1.63× throughput gain — the right call when you want the smallest quality change rather than maximum tokens/sec.
• MXFP8-AWQ is the balanced middle. Slightly better throughput than FP8 with still-tiny quality cost; a reasonable default when you want a bit more speed without moving to 4-bit.
• MXFP4-AWQ is hard to judge from this run. It ran on the Marlin weight-only fallback kernel with BF16 activations, and in that setup it delivered only 1.34× throughput, well behind NVFP4. These numbers measure that kernel, not MXFP4 on Blackwell's FP4 tensor cores.

If you keep one table from this whole post, keep this one:

If we had to ship one format today for a Blackwell serving workload, it would be NVFP4-AWQ, with FP8-AWQ as the conservative fallback.

Conclusion

The experiment set out to answer a simple question: on Blackwell, what do you actually give up when you quantize Qwen3-32B, and what do you get back? The answer is lopsided in the best way — you give up almost nothing on quality and gain a lot on serving performance.

Across every variant, Wikitext perplexity moved by at most +0.44 and GSM8K held steady (even drifting slightly upward, which we treat as noise rather than improvement). Meanwhile, peak output throughput ranged from 1.34× to 2.41× BF16, and time-to-first-token under load improved by up to ~8.4× (NVFP4 vs BF16 at concurrency 420). Quantization on Blackwell is not a quality-vs-size compromise; it is a near-free serving upgrade.

Appendix

The complete per-metric tables (TPOT, ITL, total throughput, request throughput, and median/p99 percentiles across every concurrency level) are available in RESULTS.md, and the full accuracy results (Wikitext and GSM8K) are in the results folder on Hugging Face. In those files the unquantized baseline is labelled fp16; vLLM served it in BF16 (dtype=torch.bfloat16 in qwen3-32b-fp16_server.log).

Further reading: for a detailed comparison of quantization techniques — AWQ, GPTQ, Marlin-AWQ, Marlin-GPTQ, GGUF, BitsAndBytes, and torchao — see The vLLM Quantization Complete Guide + Benchmarks.