llama.cpp Performance Flags That Matter on Low VRAM

Which llama.cpp Flags Actually Improve Performance?

Three, when VRAM is your binding constraint (flag syntax as of llama.cpp master, July 2026): -ngl (GPU layer offload) — the difference between GPU-speed and CPU-speed inference; -ctk/-ctv (KV cache quantization) — the highest-impact memory flag most people never set; and --spec-type ngram-* (n-gram speculative decoding) — the only speed technique that costs zero extra VRAM. Everything else is workload-dependent, and some celebrated flags measurably do nothing on consumer GPUs. This guide judges every flag on one axis: what it costs or recovers in memory, and what that buys in speed — with measured results attached, including the negative ones.

This is not a full flag tutorial. It is the memory-constrained reader’s version — the one that assumes you checked what your model needs and the answer was “barely fits.”

-ngl — The Flag That Matters More Than All Others Combined

-ngl / --n-gpu-layers sets the “max. number of layers to store in VRAM, either an exact number, ‘auto’, or ‘all'” (llama.cpp server docs; the default is auto). Every layer offloaded runs at GPU memory bandwidth. Every layer that doesn’t fit runs on the CPU at system-RAM bandwidth — a link that is many times slower.

The trap is thinking partial offload degrades gracefully. It doesn’t, and the reason is structural: every token must pass through every layer in sequence, so your token rate is gated by the slow segment, not averaged with the fast one. Offloading 30 of 60 layers doesn’t give you half of GPU speed — it gives you something much closer to CPU speed, because the CPU half of the pipeline sets the pace on every single token. (This is mechanism, not a benchmark — your exact ratio depends on hardware — but the shape of the cliff is universal.)

The VRAM-constrained play: the goal is always -ngl all — full residency — and if the model doesn’t fully fit, the highest-leverage move is usually not tuning -ngl downward but making the model smaller until it fits: a lower quantization tier, or freeing memory with the next flag. Full offload of a smaller quant routinely beats partial offload of a bigger one.

-ctk / -ctv — KV Cache Quantization, the Headroom Flag Nobody Sets

Your VRAM bill has two big line items: the weights everyone thinks about, and the KV cache almost nobody does. The cache grows linearly with context length, and by default llama.cpp stores it in f16.

The flags: -ctk / --cache-type-k and -ctv / --cache-type-v, default f16, accepting f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (server docs). Setting both to q8_0 roughly halves the cache’s memory; q4_0 cuts it to roughly a quarter of the f16 size (straight arithmetic from bits per element).

Concrete scale, from our calculator: Qwen3 32B at Q4_K_M holds a 2.1 GB KV cache at 8K context in f16 — and cache size scales linearly, so at 32K context the tool reads 8.6 GB for that line item alone, which exceeds many 8 GB cards before a single weight is loaded. q8_0 cache pulls that back to ~4.6 GB. On an 8–12 GB card, that’s the difference between “8K context, barely” and “32K context, comfortably” — recovered by one flag. We broke down the full KV tax per model in the VRAM requirements guide.

The trade: quantizing the cache is lossy in principle. Community practice treats q8_0 as safe for general use, while q4_0 — especially on the V cache — is where quality complaints appear on demanding tasks (community-reported; test on your own workload). If you take one action from this article, it is: try -ctk q8_0 -ctv q8_0 before you shrink your context window.

N-gram Speculative Decoding — Speed That Costs Zero VRAM

Speculative decoding generates cheap draft tokens and lets the big model verify them in one batched pass. There are two families, and for a VRAM-constrained reader they are opposites:

• Draft-model speculative decoding (--spec-type draft-simple, with --spec-draft-model) loads a second, smaller model to do the guessing. It’s the stronger technique — and it costs VRAM you don’t have. A 0.5B–1B draft model plus its own KV cache lands on the same card that was already full.
• N-gram speculative decoding (--spec-type ngram-simple, ngram-mod, ngram-map-k, ngram-map-k4v, ngram-cache) drafts by pattern-matching against text already in the context — no second model, no meaningful memory cost. You’ll also see this called lookup decoding or prompt lookup decoding; they name the same idea.

For a full card, n-gram is the only speculative decoding you can afford. That’s why it gets the full section.

How it works — and exactly when it doesn’t

The mechanism: if the recent output contains a token sequence that also appears earlier in the prompt or generation, the matcher proposes the continuation that followed last time. The big model verifies the guess in a single batched forward pass; correct guesses land several tokens for the price of one.

That mechanism dictates where it wins: output that repeats its own input. Editing existing code, reformatting a document, structured JSON with recurring keys, boilerplate — the n-gram cache hits constantly. And it dictates where it collapses: novel prose. Text that has never appeared in context can’t be looked up. As the llama.cpp docs put it, n-gram methods “require no additional model but rely on patterns that have already appeared in the generated text.”

Key knobs (llama.cpp docs): for ngram-simple, --spec-ngram-simple-size-n (lookup length, default 12) and --spec-ngram-simple-size-m (draft length, default 48); ngram-mod uses --spec-ngram-mod-n-match (default 24) with min/max draft bounds — and the docs note small n values are not recommended.

The honest numbers

Christopher Maher tested exactly this on consumer hardware — two RTX 5060 Ti 16 GB cards, Q4_K_M, flash attention, 8K context — and published the methodology:

The repeated-prompt row is the important lesson: n-gram benchmarks can look spectacular while measuring nothing but cache memorization. On realistic, diverse prompts, Maher measured nothing. His explanation is the physics of consumer cards: generation is memory-bandwidth-bound — every token already saturates the bus reading weights, so there’s no idle compute for speculative verification to exploit. On MoE models it compounds: different draft tokens activate different experts, forcing more weight reads, not fewer.

So the zero-VRAM speed flag is real, free to try, and worth turning on if your workload is repetitive or structured. On novel prose on a bandwidth-bound GPU, expect what Maher measured: nothing. Both halves of that sentence are the point.

If you do have VRAM spare: the draft-model numbers

Where memory allows a second model, community tester steampunque’s measurements on an RTX 4070 (llama.cpp discussion #10466, settings in-thread) show what n-gram can’t reach — all community-measured:

• Qwen2.5-Coder-14B drafted by Qwen2.5-0.5B: 2.5× at 10 draft tokens
• Llama 3.1 8B drafted by Llama 3.2 1B: 1.83× at 5 draft tokens
• DeepSeek-R1-32B drafted by R1-1.5B: 2.21× on the tester’s “goldcoin” benchmark task
• The consistent pattern: roughly 3× on predictable content, no speedup on unpredictable content
• The failure case: Gemma 2 9B drafted by Gemma 2 2B — no speedup at all. Draft pairing matters; a bad pair costs VRAM and returns nothing.

Budget note before trying it: the draft model’s weights and KV cache share your card with the main model. Run both through the calculator — if the pair doesn’t fit resident, you’ve rebuilt the partial-offload cliff from the -ngl section, and (as we covered in the weight-streaming reality check) memory that spills always costs more speed than any decoding trick recovers.

The Flags That Don’t Help — and Why

• N-gram speculative decoding on novel prose / bandwidth-bound GPUs. Measured above: 0% and +1%. The technique is sound; the bottleneck is elsewhere.
• Any speculative decoding on MoE models on consumer cards. Maher’s Gemma 4 26B MoE result (88.3 → 88.2 tok/s): speculative tokens fan out across different experts, adding weight reads on a bus that’s already the constraint.
• A draft model from the wrong family or without distillation lineage. The Gemma 2 9B/2B pair: full VRAM cost, zero speedup (community-measured). If the draft rarely guesses what the big model would say, verification rejects everything.
• Chasing repeated-prompt benchmark numbers. 419.5 tok/s on the tenth identical run is cache memorization, not throughput you’ll see on real work.

Head-to-Head: The VRAM-Constrained Scorecard

“Not measured” beats an invented number — where this table says mechanism, treat it as reasoning to test on your hardware, not a benchmark.

FAQ

What is the most important llama.cpp performance flag?

-ngl (GPU layer offload). Full offload (-ngl all) runs every layer at GPU memory bandwidth; any layer left on the CPU gates every token at system-RAM speed, so partial offload collapses throughput toward CPU speed rather than degrading gracefully. If the model doesn’t fully fit, shrink the model (lower quant, quantized KV cache) before accepting partial offload.

Does KV cache quantization reduce quality?

-ctk q8_0 -ctv q8_0 roughly halves KV cache memory and is treated as safe for general use in community practice. q4_0 recovers about three-quarters of the cache but carries quality risk, especially on the V cache, on demanding tasks (community-reported). It is the highest-impact VRAM flag for long context on 8–12 GB cards.

Does n-gram speculative decoding use extra VRAM?

Effectively none — that is its entire appeal. It drafts tokens by pattern-matching text already in your context instead of loading a second model. The same technique appears under three names: n-gram speculative decoding, lookup decoding, and prompt lookup decoding. Gains are large on repetitive or structured output and near zero on novel prose.

Why doesn’t speculative decoding speed up my GPU?

Most consumer GPUs are memory-bandwidth-bound during generation: every token already saturates the bus reading model weights, leaving no idle compute for speculative verification to use. Christopher Maher’s dual RTX 5060 Ti test measured 0% (Gemma 4 26B MoE) and +1% (Qwen3-32B) with n-gram speculation on diverse prompts. Draft-model speculation reached 1.3–2.5× on a single RTX 4070 in community tests (up to 2.9× with a second GPU) but requires VRAM for a second, well-paired model.