What does llama.cpp 1.0 change on a real GPU?
llama.cpp 1.0 arrived on 13 November 2026 with two new things for people who run models at home: an FP8 cache type and an expert cache for MoE models that can now hand some of its work to the CPU. We ran it against the build from 28 October on two cards, an NVIDIA L4 and a 96 GB RTX PRO 6000 Blackwell, with a dense model and a large MoE, three runs per cell. The short version for a reader about to upgrade:
- Dense models get no speed from the upgrade. Qwen 3.8 27B read and wrote within 1.3% on the L4 and within a few percent on the Blackwell.
- Qwen 3.8 Flash Next read prompts about 22% faster at every length we tried, in the same VRAM.
- The FP8 cache saved 0.3 GB on the 27B and wrote more slowly than q8_0 at long prompts on both cards. On Flash Next it does not start at all.
- A q8_0 cache on Flash Next saved 3.2 GB and cut writing nearly in half at 236K tokens. This is the setting that cost the most in the whole run.
- The expert cache ran Flash Next in 36.3 GB of VRAM at 72% of its fully resident writing speed.
Every figure below is ours, measured on 14 and 16 November. Where we explain why a number came out the way it did, we say what fits the data; we did not profile the kernels. The method and its limits are at the end.
Is llama.cpp 1.0 faster?
It depends on the model, and the split is clean.
Dense, Qwen 3.8 27B UD-IQ4_XS. On the L4 we measured on 14 November, 1.0 read and wrote within 1.3% of the 28 October build at every prompt length. On the Blackwell, with a q8_0 cache, the gap was a little larger and mostly in 1.0's favour: reading +9.1%, +3.1% and -1.6% at about 24K, 48K and 96K tokens, writing +4.3%, +2.8% and +0.6%. Nobody upgrading for speed on a dense model will feel this.
MoE, Qwen 3.8 Flash Next UD-Q3_K_XL. Here 1.0 read the prompt 22.4%, 22.7% and 22.0% faster than the October build at about 24K, 118K and 236K tokens, with an F16 cache. Writing moved less, +1.8% to +4.7%. Both builds held 70.0 GB of VRAM after loading. At 236K tokens that is the difference between waiting 86.2 s for the first word and waiting 70.7 s.
F16 cache, median of 3 runs per bar. Writing speed moved +1.8% to +4.7%. VRAM after load was 70.0 GB on both builds.
Why only the MoE, and only reading?
The gain is flat across lengths, about 22% at 24K and at 236K. A change in attention would grow or shrink with the prompt. A flat gain points at the part of each layer that does not depend on length, which in this model is the expert layers, and reading is where hundreds of tokens pass through those layers at once. The dense 27B has no expert layers, and it did not move. That is our reading of the numbers, not a profile. In practice: if you run a large MoE on llama.cpp and your prompts are long, 1.0 is worth installing for this alone.
The speed-up 1.0 users already had: MTP
The biggest single gain in our L4 session came from a flag that also works in the October build. Qwen 3.8 27B ships a multi-token prediction (MTP) head, and llama.cpp can use it to draft tokens that the full model then checks. On 1.0 with MTP drafting up to 3 tokens a step, the L4 wrote 27.8 tok/s at about 24K tokens and 19.91 at about 96K, against 14.82 and 10.23 without it. That is 1.88x and 1.95x.
Readers asked what acceptance rate sat behind the 19.91. llama-server logs it per request: the three 96K runs accepted 73 of 102, 75 of 105 and 76 of 108 drafted tokens, about 71%. At 24K and 48K it was lower, 61% to 67%. The flag is --spec-type draft-mtp, and our llama.cpp performance flags page has where it sits among the other settings. We did not repeat MTP on the Blackwell.
The FP8 cache: what it saves and what it costs
1.0 adds f8_e4m3 as a cache type, next to q8_0, q4_0 and F16. It is an 8-bit format like q8_0, which stores 8.5 bits a value once its scales are counted, so the saving against q8_0 was always going to be small. Measured, it was: on Qwen 3.8 27B with a 131,072-token window, VRAM after load was 19.2 GB with FP8 against 19.5 GB with q8_0. That is 0.3 GB on a 96 GB card. Against F16 the saving is real, 3.8 GB, but q8_0 already gives you almost all of it.
What it cost depends on the card:
| Prompt | L4, FP8 against q8_0 (writing) | Blackwell, FP8 against q8_0 (writing) | Blackwell, FP8 against q8_0 (reading) |
|---|---|---|---|
| about 24K tokens | 9.4% slower | 0.3% slower | 3.8% slower |
| about 48K tokens | 13.5% slower | 4.1% slower | 0.1% slower |
| about 96K tokens | 19.9% slower | 9.5% slower | 3.1% slower |
Blackwell cuts the penalty to half or less, but the penalty is still there and still grows with the prompt. At 96K tokens the Blackwell wrote 46.71 tok/s with FP8 against 51.62 with q8_0.
llama.cpp 1.0, 131,072-token window, median of 3 runs. VRAM is after load, as nvidia-smi reported it.
The chart also shows the cache choice that matters more than FP8. F16 wrote 19.3% faster than q8_0 at 96K tokens and cost 3.5 GB more. On a card with room to spare, that trade is better than either 8-bit cache. On a card that is tight, q8_0 is the 8-bit cache to pick: FP8 saves 0.3 GB and pays for it in writing speed.
Why FP8 is slower, and why the gap grows with the prompt
Two readers asked under our L4 post whether the slowdown was the GPU running out of compute, and whether it grew with the prompt. Turning tok/s into milliseconds per token answers both. On the L4, FP8 added 7.0, 12.2 and 24.4 ms to every token at 24K, 48K and 96K tokens: about 0.25 ms per 1,000 tokens of prompt, nearly constant. On the Blackwell it added 0.05, 0.67 and 2.04 ms, about 0.02 ms per 1,000 tokens at 96K.
A cost that scales with the number of cached tokens, while reading barely moves, fits a conversion step paid every time attention reads the cache. Blackwell pays it about ten times faster, which is why the penalty shrinks there but does not vanish. We did not profile the kernel, so treat that as the explanation that fits the numbers.
The utilization question has a plain answer on the Blackwell, where we logged it: both caches kept the GPU 98% to 99% busy at its 450 W limit. FP8 is not leaving the card idle. It spends the same full effort on more work per token.
One more difference from the L4 run: there, FP8 and q8_0 wrote the same greedy reply at all three lengths. On the Blackwell the replies differed at all three. We noted that and did not judge quality.
Qwen 3.8 Flash Next: no FP8, and q8_0 is a trap
We planned to repeat the FP8 test on the MoE. llama.cpp 1.0 refused to start the server with this line in its log:
FP8 cache is not supported for architecture 'qwen4exp'
So on Flash Next the choice is F16 or q8_0, and the q8_0 numbers are the surprise of this run:
Median of 3 runs per bar. q8_0 used 3.2 GB less VRAM (66.7 GB against 70.0 GB after load). Prompt reading stayed within 3%.
q8_0 wrote 9.0% slower at 24K tokens, 32.0% slower at 118K and 47.7% slower at 236K. In exchange it freed 3.2 GB.
Why it hurts this model more than the 27B
Writing one token means reading the weights that token uses plus the whole cache. The dense 27B reads all of its weights for every token. Flash Next has 180B parameters but activates 6B per token, so its weight read is small and the cache read is a much larger share of each token's time. Any extra cost on the cache side shows up almost undiluted. That fits what we measured: q8_0 cost the 27B 16.1% at 96K tokens, and cost Flash Next 32.0% at 118K. It also suggests the same warning for other MoE models with few active parameters, though we only measured this one.
Flash Next keeps only part of its context in a full attention cache, so the cache is small to begin with and there is little to save. If you use this model with long prompts and the F16 cache fits, keep F16. Our Qwen 3.8 Flash Next VRAM page sizes that cache for your context.
The expert cache: Flash Next on half the VRAM
The expert cache is the part of 1.0 that changes what a smaller card can attempt. A MoE model only uses a few experts for each token. With -cmoe all experts stay in system RAM; --moe-cache-mib keeps the ones in use on the card, and the cache decides which to keep. The cache itself was already in the 28 October build. What 1.0 adds is --moe-cache-cpu-share, which lets the CPU run some of the experts the cache misses.
We gave Flash Next a 24 GiB cache (--moe-cache-mib 24576), every expert in system RAM, an F16 cache, the same prompt of about 24K tokens and replies of up to 512 tokens:
| Setup on llama.cpp 1.0 | VRAM after load | Reading | Writing | GPU busy |
|---|---|---|---|---|
| Fully on the card | 70.0 GB | 3,522.7 tok/s | 114.67 tok/s | 97% |
| Experts in RAM, 24 GiB cache, CPU share 0 | 36.3 GB | 456.9 tok/s | 82.11 tok/s | 59% |
| Experts in RAM, no cache | 10.6 GB | 169.7 tok/s | 7.36 tok/s | 41% |
llama.cpp 1.0, about 24K-token prompt, median of 3 runs. The cache writes 11 times as fast as no cache.
With the cache, Flash Next kept 72% of its fully resident writing speed on 52% of the VRAM, 33.6 GB less. Against no cache at all it wrote 11 times as fast and read 2.7 times as fast.
Why writing survives and reading does not
Writing handles one token at a time, and consecutive tokens tend to reuse the same experts, so most of what each token needs is already in the 24 GiB on the card. Reading pushes thousands of tokens through every layer at once, and together they touch most of the experts, far more than 24 GiB holds, so the card waits for experts to arrive from system RAM. The GPU busy column fits that: 59% with the cache against 97% fully resident. Reading at 456.9 tok/s is 13% of the resident figure, so a 24K-token prompt that the full model reads in about 7 s takes close to a minute. For chat with short prompts the cache is a large win. For long documents the wait at the start is the cost.
What it means for your hardware
The experts live in system RAM, so the machine needs room for the 90 GB of model files on top of what the system uses. The card needs the cache plus the rest of the model, 36.3 GB in our setup with a 131,072-token window and an F16 cache. A smaller cache or a shorter window lowers that, at a cost in speed we did not measure. To check what fits fully in VRAM, without any of this, use the calculator.
The 28 October build, given the same 24 GiB cache, wrote 85.7 tok/s and read 444.6 in the same 36.3 GB. If you already used the expert cache before 1.0, the upgrade does not change it.
We also ran 1.0 with --moe-cache-cpu-share 0.5. We are not reporting that row as a result: our machine gave the run 30 CPU cores while llama.cpp started 96 threads, and the two setups that lean on the CPU (the CPU share and no cache) ran with too many threads for the cores they had. On your own machine, set -t to your real core count before you judge the CPU share.
Questions readers asked after our L4 run
Did the FP8 slowdown come from the GPU running out of compute?
On the Blackwell both caches kept the GPU 98% to 99% busy at its 450 W limit. FP8 does more work per token rather than leaving the card idle, which fits a conversion cost in attention. Details in the FP8 section above.
Did the extra time per token grow with the prompt?
Yes, on both cards, and almost linearly: about 0.25 ms per 1,000 tokens on the L4 and about 0.02 ms on the Blackwell.
Does FP8 at least save memory at long context?
Against q8_0, very little: 0.3 GB on Qwen 3.8 27B with a 131,072-token window. The 10.23 tok/s baseline in our L4 post was q8_0, not F16. Against F16, FP8 saved 3.8 GB, but q8_0 saves nearly as much and writes faster.
What acceptance rate did MTP reach on the 19.91 tok/s run?
About 71% across the three 96K-token runs, with up to 3 drafted tokens a step.
What to set after upgrading to llama.cpp 1.0
- Upgrade if you run a MoE with long prompts. The 22% faster reading on Flash Next came with no flag at all.
- Leave the FP8 cache off. On our two cards it saved 0.3 GB at most against q8_0 and wrote more slowly. On Flash Next it does not start.
- On Qwen 3.8 Flash Next, use the F16 cache if it fits. q8_0 frees 3.2 GB and costs nearly half the writing speed at 236K tokens.
- On a dense model with room to spare, try F16 before any 8-bit cache. It wrote 19.3% faster at 96K tokens for 3.5 GB.
- If the model has an MTP head, turn it on. It nearly doubled writing on the L4, in both builds.
- If a MoE does not fit, try the expert cache before plain
-cmoe. In our run it was the difference between 82.11 and 7.36 tok/s.
The order these settings go in, and what each one costs, is on our llama.cpp performance flags page. For what a given card writes per second, see LLM tokens per second.
How we tested
Builds: llama.cpp's own prebuilt Ubuntu CUDA 12.8 downloads at fixed commits: b11962 (f5222f25c, 28 October 2026) and v1.0.0 (b12359, b1b770d90, the 1.0 tag).
Cards: One NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, ECC off, with its power limit set to 450 W (its default is 600 W), on 16 November 2026. Every resident run sat at that limit with the GPU 97% to 99% busy, so these are 450 W figures; at 600 W the card would likely be faster. The L4 rows are from 14 November: one L4 24GB at its 72 W cap.
Models: Qwen 3.8 27B UD-IQ4_XS, the 10 November file (14,252,845,984 bytes), with a 131,072-token window. Qwen 3.8 Flash Next UD-Q3_K_XL at revision 766911a6, three files totalling 89,986,353,824 bytes, with a 262,144-token window. Both from Unsloth's GGUF repositories, checked by size before the runs.
Runs: Flash attention on, greedy replies capped at 256 tokens (512 in the expert cache test), three runs per cell and the median reported. Each run started with a unique tag so the prompt cache could not serve a repeat. VRAM is what nvidia-smi reported after the model loaded and at the peak of each cell, converted to GB. GPU utilization and clocks were sampled every half second.
Limits: One card of each kind, one dense model and one MoE, each at one quantization. Other models, other quantizations and other cards can move every number here. Writing speed depends on the reply length, and greedy replies differ between cache types, so small gaps in writing speed are within that noise.
FAQ
Is llama.cpp 1.0 faster than the previous build?
On Qwen 3.8 Flash Next it read prompts about 22% faster than the 28 October build in our run, with writing speed nearly unchanged. On the dense Qwen 3.8 27B the two builds were within 1.3% on an L4 and within a few percent on an RTX PRO 6000 Blackwell.
Should I use the FP8 KV cache in llama.cpp 1.0?
In our runs, no. On Qwen 3.8 27B it saved 0.3 GB against q8_0 and wrote 9.5% slower at about 96K tokens on a Blackwell card, and 19.9% slower on an L4. On Qwen 3.8 Flash Next llama.cpp 1.0 refuses to start with it.
Why is the FP8 cache slower than q8_0?
In our runs the extra time per token grew with the number of cached tokens while prompt reading stayed flat, which fits a conversion step paid each time attention reads the cache. The GPU stayed 98% to 99% busy with both caches.
What does the llama.cpp expert cache do?
With all MoE experts in system RAM (-cmoe), --moe-cache-mib keeps the experts in use on the GPU. With a 24 GiB cache, Qwen 3.8 Flash Next used 36.3 GB of VRAM and wrote 82.11 tok/s, against 114.67 fully on the card and 7.36 with no cache.
Does a q8_0 KV cache slow down Qwen 3.8 Flash Next?
Yes, and more as the prompt grows. On llama.cpp 1.0 it wrote 9.0% slower than F16 at about 24K tokens and 47.7% slower at about 236K, while saving 3.2 GB of VRAM.