Mixed Precision Quantization: Half a 4-Bit File Is Not 4-Bit

What Does Mixed Precision Quantization Do to the Gigabytes?

It moves the big tolerant part of the model down to 4 bits and leaves the small fragile part alone, and on a Mixture of Experts model that is most of the weight for almost none of the damage. GLM-5.3-Flash goes from 328 GB as Z.ai published it to 194.7 GB, with 97% of its parameters at 4 bits. Qwen3.8-Flash-Next goes from 360 GB to 133 GB. Neither number is what “4-bit” would predict, in either direction, and the gap is the whole subject.

One thing first, because it decides whether the rest of this page is for you. NVFP4 is a Blackwell format. A 3090, a 4090 and an A100 cannot execute it. If that is your hardware, skip to “No Blackwell?” below, because the same idea has a version that runs on yours.

The Number Everybody Skipped Is Inside NVIDIA’s Own File Listing

nvidia/Qwen3.8-Flash-Next-NVFP4 went up on 2 September 2026 and had 1,129 downloads when we read it. It is labelled FP4. It is 132.73 GB.

We pulled the repository’s file listing and its dtype index and added the bytes up by hand. Here is where they go.

The 4-bit part of a 4-bit model is 51.2% of the download. The other 48.8% is the material NVIDIA refused to put in four bits, and 40.5 points of that is a single lookup table.

The four rows sum to 132.64 GB against an actual 132.68 GB of weight files. A 41 megabyte residual across 132 gigabytes is norms and router tensors, which is as close as this arithmetic gets to closing.

Their Compression Claim Checks Out, and Then One of Them Does Not

NVIDIA’s card says the checkpoint is “approximately 2.7x smaller than the BF16 source checkpoint (about a 63% reduction in disk size).” The BF16 source, Qwen/Qwen3.8-Flash-Next, is 360.02 GB. Dividing gives 2.712x and 63.1%. That is a vendor stating its own compression accurately to three significant figures, which is rarer than it should be.

Their other repository does the opposite. nvidia/Qwen3.8-2.4T-A95B-NVFP4 says its optimization reduces disk size “by approximately 2.5x”. The base weights are 4,892.39 GB and the NVFP4 build is 1,444.56 GB. That is 3.387x, a 70.5% reduction, and NVIDIA is underselling its own file by nearly a third. We are reporting the arithmetic, not correcting anybody: both figures come from the same publisher’s own listings, and the smaller claim is the one that is wrong.

The Plan Is Not a Number. It Is a List of 50 Things and a List of 292.

Open hf_quant_config.json in the Flash-Next repository and there is no bits-per-weight figure anywhere. The algorithm field reads MIXED_PRECISION. Under it sit two lists.

Quantized, 50 entries: the routed expert stacks in all 48 layers at NVFP4 with a group size of 16, the n-gram embedding at FP8, and the MTP expert block at FP8.

Excluded, 292 entries: every shared expert, every routing gate, every hyper-connection, the 36 linear attention blocks, the 12 full attention blocks, lm_head, embed_tokens, and model.visual*, which is the entire 27-layer vision tower.

Qwen’s own config says this model has 48 layers and 512 experts, of which 10 fire per token. So the thing being crushed to four bits is the 512-way expert bank that dominates the parameter count, and the things left untouched are the parts every token passes through. A quant name describes a request. This is a floor plan.

That distinction has a cheap version people already meet: two publishers ship a file called Q4_K_M and they are four percent apart. Same failure, one axis down.

GLM-5.3-Flash: 3% of the Weights, 10% of the File

The clean example is LibertAIDAI/GLM-5.3-Flash-NVFP4, up since 26 August with 36,081 downloads. There is a second build, orcarouter/GLM-5.3-Flash-Uncensored-NVFP4, which is an abliterated derivative of the same base. Its repository is gated, so we could not read its card, but Hugging Face still serves the file table and the dtype index, and those are what the arithmetic needs.

For the OrcaRouter build:

Protecting 3% of the parameters costs 10% of the file. If every weight had gone to NVFP4 at the same 4.5 effective bits, the checkpoint would be 180.7 GB. It is 194.7 GB. The BF16 island costs 13.9 gigabytes, which is what a design decision looks like when you can put a price on it.

The base checkpoint Z.ai publishes is not BF16, which matters for reading any compression claim about this model. zai-org/GLM-5.3-Flash is FP8 at 328.37 GB. So the trip from the published weights to the NVFP4 build is 8 bits down to a mixed 4, a factor of 1.73, not the factor of 4 the label suggests.

The download is 194.7 GB, and the weights occupy essentially that once resident, before a single token of context. Written the other way, in the binary units a GPU vendor uses, 194.7 GB is 181.3 GiB, which is the same file and not a saving. Z.ai has published no NVFP4 build of its own and neither has NVIDIA, so this is the most downloaded conversion rather than an official one: LibertAIDAI/GLM-5.3-Flash-NVFP4, 36,081 downloads, with a parameter count matching the base checkpoint to the digit. The full memory breakdown for this model, including what the KV cache does at a million tokens, sits on its own page.

One 320B Model, Five Published Builds, All Called 4-Bit or 2-Bit

Every row below is GLM-5.3-Flash. Every one is somebody’s mixed precision recipe. The four measured rows are already in our speed corpus, each attributed to the person who ran it.

Ninety-eight gigabytes to one hundred and ninety, and every one of them is a four-bit-class build of the same weights. @ViC305’s description of his own recipe could be NVIDIA’s config file with the nouns changed: trellis quantization on all 288 routed experts, with attention, the shared expert, embeddings, lm_head and vision left at source precision.

Two people converged on the same floor plan using two different quantizers on two different runtimes in the same week. That is the part worth noticing.

What It Costs in Quality, With Every Number Attributed

Nobody outside the labs has run an independent evaluation of these checkpoints, so what follows is each quantizer marking their own homework. That is worth saying once and then getting on with.

NVIDIA’s own table for the Flash-Next build puts NVFP4 against their FP8 baseline across nine suites, and the NVFP4 column wins five of them: SciCode 18.8 against 16.3, AA-LCR 74.1 against 71.9, MMMU Pro 78.3 against 77.1, HLE 35.4 against 34.7, IFBench 81.0 against 80.5. It loses the other four narrowly, the widest being GPQA Diamond at 91.5 against 92.0. Measured with reasoning_effort=xhigh on B200 and B300.

@MiaAI_lab published a divergence rather than a benchmark, which is the more useful shape. EXL3 4bpw at 176 GB scored 0.0246 KL divergence against the FP8 checkpoint’s 0.0206 at 328 GB, as recorded on 28 August 2026. Half the bytes, and the distance from the original distribution grows by four thousandths. His repository now frames the same comparison as roughly 1.00x KLD, so read the pair as a snapshot of one build on one day rather than a standing measurement. Either way the point survives: the gap is small enough that it is not what should decide your build.

@TeksEdge reported perplexity of 2.788 rising to 2.894 on the OrcaRouter build, labelled “+3.8%, author test”. That is a secondary report of a gated repository’s own claim, we could not open the card to confirm it, and it should be read as the quantizer’s number rather than a measurement. The file sizes in his post are correct and they are in binary units: 306 down to 177 GiB against listings of 305.8 and 177.2 GiB, which are the same files we quote elsewhere as 328.4 and 190.2 GB. Read every size in this article as decimal GB unless it says GiB.

No Blackwell? The Same Idea Ships as a GGUF, and Somebody Measured It

NVFP4 needs a Blackwell tensor core. The cheapest box that has one and enough memory for these models is a DGX Spark, and two of them are what the measured rows above are running on. An Ada or Ampere card gets nothing from an NVFP4 file at all.

The idea travels anyway, because GGUF has always allowed per-tensor formats and almost nobody used the freedom deliberately. Ivan Fioravanti’s build of Qwen3.8-Flash-Next is the clearest example we have found, and you can read the recipe without opening the repository, because it is in the filename:

Qwen3.8-Flash-Next-Q4KImatrixExperts-MXFP4Down-BF16Emb-BF16Control-Q8GDN-Q8QSA-Q8Shared-Q8Out.gguf

A hundred and two characters, four formats, eight named tensor roles. Routed experts at imatrix-calibrated Q4_K, the down projections at MXFP4 because 640-wide rows cannot take a 256-block type, the DeltaNet and sparse-attention projections and shared experts and output at Q8_0, embeddings and routers and norms at BF16. The 51-billion-value n-gram table sits in a separate 32.00 GB file and is read from disk on demand, never resident.

His measured figures, on an M3 Ultra he owns and states: about 75 GB resident, 45.4 to 45.6 tok/s decode between 4K and 16K, 42.2 tok/s at the full 262K window, and 55 to 56 with the MTP block doing speculative decoding.

His quality harness is worth reading closely, because it does not say what the announcement said. He scored the new imatrix Q4_K recipe at 0.0431 target-logprob MAE with 96.6% top-1 agreement against an exact BF16 reference. His previous Q4_0-routed release scored 0.0430 and 96.5%. In his own words the new recipe “buys back the down-projection precision budget as MXFP4 while holding overall drift flat”, which is a more honest description than an improvement, and it is his own description of his own file.

A second builder went further in the same direction. Baekpica’s mixed-quant conversion moves the whole n-gram table out to an SSD-backed sidecar and reassigns the freed residency to the compute backbone. Their BF16 extraction of that table is 102,400,786,432 bytes. Our own catalogue has carried 102.40 GB for it since the Flash-Next launch, derived independently. Those agree to the byte, which is the sort of check that makes a number safe to build on.

llama.cpp 0.4.0 Landed Yesterday, and One Line of It Belongs Here

Version 0.4.0 was published on 4 September 2026 with initial Qwen3.8-Flash-Next support, Nemotron-3-Puzzle support, video input options and ggml 0.23.0. The entry that matters for memory is --lazy-mode, and it is easy to over-read.

It saves host RAM, not VRAM. The pull request author’s own summary: models carrying PLE and n-gram embeddings do not need the whole table in RAM, so it is read lazily through mmap. Set to auto it engages only on tensors above 4 GiB, which is a deliberate guard, because on a small model the read latency is large next to the token.

He published the cost on Gemma 4 E4B at Q4_K_M, where the lazy tensor is 39% of the file. Peak resident set falls from 7.37 GB to 6.16 GB. Generation falls from 105.8 tok/s to somewhere between 94.2 and 97.3, depending on whether the madvise hint is set. A 16% memory saving for 8 to 10.7% of the speed, measured by the person who wrote the patch.

That is the same insight as NVIDIA’s FP8 embedding table, arriving in a different project in the same fortnight, and pointing the other way: NVIDIA kept the table on the device and paid for it in bits, llama.cpp keeps it off the device and pays for it in latency. @0xBakeer measured the second trade at about 6% throughput on a DGX Spark, which is why our calculator models that one tensor and refuses to model offload in general.

Our Own Calculator Cannot Give You the 195 GB Answer

This is the part we would rather not write, so here it is plainly.

Our engine carries one bits-per-weight figure for an entire model. Ask it for GLM-5.3-Flash at Q4_K_M and it multiplies 321.3 billion parameters by 4.85 bits and returns 181.2 GiB, which is 194.6 GB. That happens to land within a tenth of a percent of the real NVFP4 file, and the agreement is a coincidence: a Q4_K_M rung and a mixed NVFP4 plan arrive at the same total by different routes. Change the plan and the coincidence evaporates. The engine has no way to say “experts at 4.5, attention at 16”, so it cannot produce @danpacary’s 108.4 or @ViC305’s 97.7, and there is no NVFP4 rung on the ladder at all.

Two pieces of the machinery already exist. observedSizes lets a real published file overrule our arithmetic, which is how the Nemotron 3.5 Lightning page stopped telling 24 GB owners a 25.3 GB file would fit. And the Flash-Next entry already carries a separable block for exactly one tensor, the n-gram table, with its measured 6% cost. What is missing is the general case: a tensor-role table, and quant identifiers for the formats that do not appear on a GGUF ladder.

We have written that up for the developer side rather than quietly shipping a guess. Until it lands, the honest instruction is the one at the top of this section: for a mixed precision build, read the file size off the repository listing, then use our calculator for the KV cache and the runtime overhead on top of it. Those two parts it does correctly.

What To Do With This

Blackwell, with a memory pool above 200 GB. The NVFP4 builds are the best size-to-quality trade published for these two models, they run on vLLM and SGLang today, and the vendor evaluations are attached to the cards. Read the parameter breakdown before you plan capacity, because half of a Flash-Next download is not four bits and your pool has to hold all of it.

One DGX Spark, and this is the fit worth knowing. NVIDIA’s own Qwen3.8-Flash-Next NVFP4 build is 132.68 GB of weights, which is 123.6 GiB, against a Spark’s 128 GB pool. Holding all of it would leave nothing to work with. It runs anyway, and the reason is the table above. Tony Deangelo (@Tech2Wild) serves the checkpoint byte for byte on a single Spark by leaving the 47.68 GiB n-gram table on NVMe and reading the 16 rows each token needs on demand, so about 76 GiB stays resident and the rest of the pool becomes KV cache. His measurements on the box, 5 September 2026: a 995,129 token KV pool, 21.7 tok/s on prose, 32.5 median across 40 prompts, and a p99 step latency within 10% of p50 at six streams, so the on-demand reads do not show up as a tail. The 40 percent of the download that is not four bits is also the 40 percent you never have to load. Our own catalogue has carried that table as separable since the Flash-Next launch, at about a 6 percent throughput cost.

Two DGX Sparks for GLM-5.3-Flash. At 194.7 GB the NVFP4 build is 181.3 GiB, so one Spark cannot hold it whatever the label says, and two leave room for a 262K window. @Tech2Wild measured 43.4 tok/s on that pair. The alternative on a single box is the trellis or 2-bit build, which gives up more.

Anything older. The mixed precision idea is not locked to NVFP4. A GGUF with experts at Q4_K and everything else higher is buildable today with tools that already exist, and it is what the fastest community builds of these models are doing whether or not their filenames admit it. Start from the flags that decide where each tensor lives.

Then there is the habit, and it applies on any hardware. Stop reading the quant name as a description of the file. On a Mixture of Experts model it describes one part of the file, and on the checkpoints in this article that part is between half and nine tenths of it.

FAQ

What is mixed precision quantization?

Storing different parts of one model at different numeric precisions in the same checkpoint, chosen by what each tensor does rather than by one setting for the whole file. On a Mixture of Experts model the usual split is routed experts at 4 bits, with attention, shared experts, routers, embeddings and any vision tower left at 8 or 16 bits. NVIDIA’s config file for Qwen3.8-Flash-Next lists 50 tensor groups it quantized and 292 it excluded.

Does mixed precision quantization need special hardware?

NVFP4 does. It is a Blackwell-generation format, so a B200, a B300, a GB10 in a DGX Spark or an RTX PRO Blackwell card can execute it and an RTX 3090, a 4090 or an A100 cannot. Mixed precision as an idea needs nothing special: GGUF has supported per-tensor formats for years, and community builds of Qwen3.8-Flash-Next already ship four or five different formats inside one file.

How much VRAM does GLM-5.3-Flash need at NVFP4?

The most downloaded NVFP4 build is 194.7 GB for the weights alone, before the KV cache. An abliterated variant of the same model is 190.2 GB. Z.ai’s own FP8 checkpoint is 328.4 GB. Those are file listings rather than our arithmetic, which for a mixed precision build is the number to trust.

Why is a 4-bit checkpoint not four times smaller?

Three reasons, and they stack. Some tensors were deliberately left at higher precision. NVFP4 carries an FP8 scale for every 16 values, so it costs about 4.5 bits per weight rather than 4. And the base checkpoint may not have been 16-bit to begin with: GLM-5.3-Flash ships as FP8, so its 4-bit build is a 1.73x saving, not a 4x one.

Does mixed precision quantization hurt quality?

On the published evaluations, very little, and every one of those evaluations was run by the person who built the file. NVIDIA’s own table has NVFP4 beating its FP8 baseline on five of nine suites for Flash-Next. @MiaAI_lab measured 0.0246 KL divergence for a 176 GB EXL3 build against 0.0206 for the 328 GB FP8 original. As of 5 September 2026 we have found no independent evaluation of these NVFP4 checkpoints, so treat the numbers as vendor and author claims.

Can your calculator size a mixed precision build?

Not yet, and this article says so on purpose. The engine models one bits-per-weight figure for a whole model, so it cannot express experts at 4 bits with attention at 16, and it has no NVFP4 setting. For a mixed build, take the size from the repository’s file listing and use the calculator for the KV cache and runtime overhead on top.

What does llama.cpp’s new lazy mode save?

Host RAM, not VRAM. --lazy-mode reads very large embedding tables from disk on demand instead of pulling them into memory at load. On the author’s own Gemma 4 E4B test, peak resident set fell from 7.37 GB to 6.16 GB and generation fell from 105.8 tok/s to between 94.2 and 97.3. Set to auto it only engages on tensors above 4 GiB.