Run 405B on 8GB VRAM? No One Has Published the Speed

Can You Really Run a 405B Model on 8 GB VRAM?

Loading it: yes, demonstrably — AirLLM’s layer-streaming technique is real engineering. Running it at a published speed: nobody has shown that number. As of July 2026, no measured throughput figure — tokens per second — for Llama 3.1 405B via AirLLM exists in public: not in the author’s write-up, not in the project README, not in community testing. The closest thing to a public timing is a GitHub issue reporting 20+ minutes for a single sentence, running the 405B in 4-bit on an A100. Meanwhile the other viral streaming project, Colibri (GLM-5.2, a 744B MoE, on ~25 GB RAM), publishes its own invoice down to the decimal. Both projects’ numbers — and the missing one — side by side, is this page.

Resident vs. Streamed: The Distinction the Headlines Skip

Every “impossible” number in this story comes from one substitution: streaming weights from disk instead of holding them resident in memory.

• Resident memory — the model’s weights sit in VRAM (or unified memory) for the whole session. Every token can touch any weight at memory speed. This is what llama.cpp, Ollama, and LM Studio assume, and it is what our VRAM calculator computes: the memory a model needs to live on your hardware.
• Streamed weights — only a slice of the model is in memory at any instant. The rest lives on an NVMe drive and is read in when a layer or expert is needed, then evicted. Peak memory drops from “entire model” to “largest active slice.” Speed drops with it, because your SSD is now inside the inference loop — and an SSD is orders of magnitude slower than VRAM.

For scale: the calculator puts Llama 3.1 405B at Q4_K_M at 246 GB of model weights — a 287 GB practical baseline once KV cache and runtime overhead are counted (8K context, llama.cpp backend). Select it yourself; the math is shown line by line. So “405B needs ~287 GB” and “405B ran on 8 GB” are both true — they describe different architectures of where the weights are. Resident memory determines interactive speed. Streaming trades throughput for capacity. Neither number lies; they answer different questions.

The Invoice: What Is Published — and What Isn’t

All figures below are from the projects’ own repositories or community tests, as of July 2026. We have not tested either tool — we own no hardware — and every speed figure carries its source class.

Two details worth stating precisely. Colibri’s README is unusually candid for a viral project — it opens with its own limitation, publishes unflattering cold-cache numbers, and addresses drive health directly (reads don’t meaningfully wear SSDs, it notes; the real flags are swap-induced writes and sustained thermal load). And AirLLM’s technique — layer-by-layer inference with 4-bit quantization, per the author’s own write-up — is legitimate: community tests confirm the 70B path works, at roughly 0.07 tok/s on an M2 MacBook Pro up to ~0.7 tok/s with NVMe and a dedicated GPU (community-reported; Nerd Level Tech’s test, no formal methodology published). The 405B headline simply has no measured speed behind it. The one public datapoint on record is a GitHub issue reporting 20+ minutes for a single sentence — running the 405B (bnb-4bit) on an A100, with no token counts or settings: a complaint, not a benchmark. One project publishes its invoice; the other’s flagship number is blank.

Why One Publishes ~1 tok/s and the Other Publishes Nothing

The gap is not code quality. It is physics — specifically, how many parameters each token actually touches.

GLM-5.2 is a Mixture-of-Experts model. Of its 744B total parameters, only about 40B are active per token — each MoE layer routes to a small subset of its experts. Colibri exploits exactly this: the ~17B of dense, always-needed weights stay resident in RAM, and only the experts the router selects (~19 MB each) are pulled from disk per token. Per-token disk traffic scales with active parameters, not total. Streaming a MoE is an engineering problem with a real solution, and the published 0.3–1.2 tok/s warm range (community benchmarks, collected in the repo) is that solution working.

Llama 3.1 405B is dense. Every one of its 405B parameters participates in every token. There is no router to skip anything, so per-token traffic scales with total parameters — at Q4_K_M, that is 246 GB of weights through the compute path per token, bounded by NVMe read speed. No implementation cleverness removes that floor, which is consistent with the record: the measured numbers for AirLLM come from the 70B path, at 0.07–0.7 tok/s (community-reported), and the only public report of the 405B path running at all clocks a single sentence at 20+ minutes — on an A100. Streaming a dense model is arithmetic, not engineering. The arithmetic is presumably why no one — author included — has published a 405B tokens-per-second figure.

Same headline pattern. Different physics. One measured, one blank. That is the entire story.

What Weight Streaming Is Actually For

At the speeds that are published — 0.05 to ~2.1 tok/s across both projects, community-reported — a 200-token reply takes anywhere from under two minutes to over an hour. Streaming is not usable for chat. Stating that is not a criticism; it is the spec. The trade has real uses:

• Batch and offline jobs — overnight summarization, dataset labeling, evaluation runs where latency is irrelevant and only completion matters.
• One-off runs on models you could never load — testing whether a frontier-scale open model even helps your task before renting serious hardware.
• Capability access over interactivity — some questions are worth a 30-minute wait if the alternative is no answer at all.

If your goal is interactive local AI, the resident math still rules, and it is unforgiving: the model must fit in VRAM (or unified memory) at your chosen quantization. Mainstream models are far friendlier than the headlines suggest — the per-model requirements breakdown covers what actually fits on real cards, and surprisingly modest GPU tiers handle serious local work. To see what fits resident on your hardware — including 405B, if you want to see the real bill — run it through the calculator.

Coming next: if streaming answers “can it run at all,” the follow-up is “how do you make what fits run faster” — speculative decoding and the llama.cpp flags that genuinely help on low-VRAM cards (and the ones that don’t). That piece is in progress.

FAQ

How fast is AirLLM with Llama 3.1 405B?

No measured tokens-per-second figure exists as of July 2026 — the author’s write-up and the project README publish no throughput data. The sole public 405B datapoint is a GitHub issue reporting 20+ minutes for one sentence (405B in 4-bit, on an A100) — no token counts or settings; a complaint, not a benchmark. Community tests of AirLLM’s 70B path measure roughly 0.07–0.7 tokens per second (community-reported). A dense model streamed from disk is bounded by drive bandwidth on every token.

Why can a 744B model run on 25 GB of RAM?

Because GLM-5.2 is a Mixture-of-Experts model: only ~40B of its 744B parameters are active per token. Colibri keeps ~9.9 GB of always-needed weights resident and streams the selected experts from a ~370 GB file on NVMe. Published speeds: ~0.05–0.1 tok/s cold from the author, ~0.3–1.2 tok/s warm from community benchmarks, up to 2.06 tok/s on Apple Silicon with the Metal backend (community-reported).

Is weight streaming usable for chat or coding assistants?

No. At the published community-reported speeds — roughly 0.05 to 2.1 tokens per second across streaming projects — a normal-length reply takes minutes to tens of minutes. Streaming suits batch jobs, offline analysis, and one-off runs of models that cannot fit any other way. Interactive use still requires the model resident in VRAM or unified memory.

How much memory does Llama 3.1 405B actually need to run resident?

Per the VRAM calculator: at Q4_K_M with an 8K context on llama.cpp, model weights are 246 GB and the practical baseline is 287 GB including KV cache and runtime overhead. That is the number weight-streaming headlines route around — capacity traded for speed. Resident requirements are unchanged by streaming tools.