Run 26B on RTX 4060 (8GB)? What the Claim Leaves Out

Can You Really Run a 26B Model on an RTX 4060?

Yes: and there’s video to prove it. On July 24, 2026, a tester running under the handle @analogalok posted a single RTX 4060 — the 8GB card, the one that sells for around $250 — running Gemma 4 26B A4B, a 26-billion-parameter Mixture-of-Experts model, with vision, at roughly 25 tokens per second and up to 180k tokens of usable context. Multimodal, speculative decoding, long context, on a budget GPU with 8GB of VRAM and 16GB of system RAM. The claim is real, the methodology is fully disclosed, and he included a video.

So this article is not here to debunk it. It’s here to do the two things a VRAM site is uniquely placed to do: explain why it works — down to a KV-cache cliff our own calculator predicts to the gigabyte — and then, because “$250 GPU” is doing a lot of quiet work in that headline, add up what the whole thing actually costs to build. Because the 8GB card is the cheap, visible part. It is not the expensive part.

The Benchmark

Posted by @analogalok on X, July 24, 2026 — with exact llama.cpp server flags, per-context-length prefill and decode numbers, and a video of the runs. The model is Unsloth’s QAT quant gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf (a ~Q4-class, quantization-aware-trained build), on a single RTX 4060 8GB with 16GB system RAM. Three things are stacked on that one card: the model, a Multi-Token Prediction (MTP) draft model for self-speculative decoding (~240 MB), and a vision projector (mmproj, ~1.1 GB).

Text-only, with MTP speculative decoding (28k-token prompt):

With vision loaded on top (28k text + image):

The finding that matters most is that last pair. Stack MTP + vision + Q8 KV cache past 120k context and it falls off a cliff — decode collapses from ~24 t/s to 12.5. Drop the KV cache from Q8 to Q4 and the wall disappears: back to 25.5 t/s at 180k. In the poster’s words, past 120k with vision loaded, “Q4 KV isn’t optional… it’s mandatory.”

Why It Works: Four Tricks, Not One

No single feature makes a 26B multimodal model run on 8GB. Four stack together:

1. MoE sparsity. Gemma 4 26B A4B activates only ~4.1B of its 26B parameters per token. That is the precondition for everything else — most of the model is idle on any given token.

2. CPU expert offload. Because most experts are idle, the bulk of the weights can live in the 16GB of system RAM, with only attention layers and the active/hot experts touching the GPU. This is the same mechanism we cover in depth in how a 118B model runs on a 12GB card — --n-cpu-moe and friends. This is why “8GB VRAM + 16GB RAM” is the headline and not just “8GB VRAM.” The RAM is not a footnote; it is doing the heavy lifting.

3. Sliding-window attention — the unsung hero of the context claim. A normal full-attention 26B model at 120k context would need tens of gigabytes of KV cache alone — impossible on this card at any setting. Gemma 4 caches only a 1024-token window on most of its layers, so its KV stays small even at extreme context. Without this, the 180k-context claim wouldn’t be physically possible regardless of how much you offload. We break down this mechanism in the llama.cpp performance flags guide.

4. MTP self-speculative decoding. A small draft model proposes several tokens at once and the main model verifies them in one pass — a “near-free” decode speedup, per the poster, of ~20–30% up to 120k context (he notes it reached 2× for Gemma 4 31B on an RTX 4090 — model- and card-specific, not a universal number). Past 120k, the draft model’s own overhead starts fighting for the same scarce VRAM, which is where the KV cliff comes from.

Why dropping to Q4 KV clears the wall

That Q8-to-Q4 KV switch isn’t magic — it’s a VRAM budget problem, and our calculator puts numbers on it. Using Gemma 4 26B’s real per-layer attention metadata (5 global layers at 2 KV heads × 512 dims, 25 sliding-window layers at 8 × 256 — read straight from the official GGUF, the same header our calculator’s GGUF reader parses), the KV cache at ~128k context is:

• Q8 KV: ~1.5 GB
• Q4 KV: ~0.8 GB

Quantizing Q8 → Q4 roughly halves the cache, freeing ~0.7 GB. On an 8GB card already carrying the vision projector (~1.1 GB), the MTP draft model (~250 MB), and the resident attention layers and hot experts, that ~0.7 GB is the breathing room between the cliff and 25.5 tok/s. He found the wall empirically; it was always there in the arithmetic.

What the “$250 GPU” Headline Leaves Out

This is where a VRAM calculator earns its keep, because the honest version of this story is a system story, not a GPU story.

The offload that makes this possible doesn’t run on the GPU. It runs on system RAM and the CPU — and those are exactly the two components the headline never specs.

The RAM. 16GB is the minimum to hold ~13GB of offloaded experts plus the OS. But capacity isn’t even the interesting part — bandwidth is. Offloaded decode speed is gated by how fast the CPU can read expert weights out of RAM every token. Dual-channel vs single-channel, the kit’s rated speed, whether it actually runs at that speed on your motherboard — these swing the 25 tok/s figure by a lot. And all of this lands during a period of sharply rising DRAM prices — a fast 32GB+ kit is no longer the throwaway line item it was a year ago.

The CPU. Someone has to run those offloaded experts, and to keep a 4B-active MoE fed at 25 tok/s that CPU has to be capable. The thread doesn’t say which processor was used — and that omission matters, because the answer changes the build cost more than the GPU does. A top-tier chip like an AMD 9950X3D2 lands in the same price bracket as a second graphics card — used RTX 3090 territory. You can absolutely do this on a mid-range CPU with a slower decode number; the point is that “single $250 GPU” describes one line on a receipt that has several more lines.

None of this makes the benchmark false. It makes it incomplete — and “incomplete” is the normal state of viral hardware content, which is optimized for the screenshot, not the bill of materials. The reason our calculator deliberately models VRAM only is precisely this: VRAM cost is deterministic and portable — the same everywhere — while offload speed depends on a CPU and a RAM kit that no calculator can see. We give you the number that doesn’t lie to you, and we’re honest about where the number stops.

What Full Residency Actually Costs

So what would it take to run Gemma 4 26B A4B the straightforward way — everything on the GPU, no offload, no system-RAM dependency, maximum speed? The weights are a known quantity: Unsloth’s Q4 build is a 14.2 GB file on disk (measured, not estimated — drop the GGUF into our calculator and it reads the real size straight off the file). Add the KV cache on top:

Full residency wants a 24GB card (an RTX 4090, or a 32GB RTX 5090 with comfortable headroom for long context). That’s the number that tells you whether you need the offload technique at all — and for most people on an 8/12GB card who want this model, the answer is yes, with the system-level trade-offs above. If you’d rather see what runs on your card without any of this complexity, check what fits your GPU natively.

Should You Try This?

Yes, if:

• You have a single 6–12GB NVIDIA card (the poster names the RTX 3050/3060/3070, 4050/4060, 5050/5060 as good candidates) and want a capable multimodal model
• ~20–25 tokens/second is fast enough for your work (chat, drafting, vision Q&A — not real-time serving)
• You already have, or are willing to buy, a decent CPU and a fast 16–32GB RAM kit — and you’ve priced those in, not just the GPU

No, if:

• You need maximum throughput or are serving concurrent users — full residency on a 24GB+ card wins
• You were sold on “$250” as the total cost — the GPU is real at $250; the build is not
• Your target model is dense, not MoE — offloading a dense model has no sparsity to exploit and the slowdown is far worse

FAQ

Can you run a 26B model on an RTX 4060 8GB?

Yes, if the model is Mixture-of-Experts and you use CPU expert offloading. A first-party benchmark (with video) on July 24, 2026 ran Gemma 4 26B A4B — with vision — at ~25 tokens/second on a single RTX 4060 8GB plus 16GB system RAM. The GPU holds attention layers and active experts; system RAM holds the rest. Held entirely in VRAM instead, the weights alone are a 14.2GB file and the full setup wants roughly 17–23GB depending on context — a 24GB card.

Why must you use Q4 KV cache past 120k context on 8GB?

Because VRAM runs out. At ~128k context, Gemma 4 26B’s KV cache is about 1.5GB at Q8 and 0.8GB at Q4 — quantizing roughly halves it. On an 8GB card already holding a vision projector (~1.1GB) and a speculative-decoding draft model (~250MB), that ~0.7GB is the room that clears the wall Q8 hits past 120k. See our performance flags guide for KV-quantization trade-offs.

What is MTP speculative decoding?

Multi-Token Prediction: a small draft model proposes several tokens at once and the main model verifies them in a single pass, boosting decode speed. In the benchmark it added ~20–30% up to 120k context. For Gemma 4 the draft is a separate ~240MB model; past 120k its overhead competes for scarce VRAM and the gain reverses.

Is the RTX 4060 really all you need to run this?

No — the RTX 4060 is the visible cost. CPU expert offloading also needs enough fast system RAM to hold the offloaded experts (16GB minimum here, bandwidth-sensitive) and a CPU capable enough to run them at speed. Those aren’t specced in most viral benchmarks, and in the current DRAM market they aren’t trivial. Budget the whole system, not just the card.

Why does the VRAM calculator show 17–23GB when it runs on 8GB?

Because they answer different questions. The calculator computes full VRAM residency — the entire model on the GPU, for maximum speed. The benchmark uses CPU offloading, where most weights live in system RAM and only a fraction touch the GPU. Both are accurate; they describe different ways of running the same model. Our calculator intentionally models VRAM only, because offload speed depends on a CPU and RAM kit it can’t measure.