FreeToken VRAM Requirements, and Why VRAM Is the Wrong Number
Less than you think, and that is the wrong question. FreeToken runs a 35B model on an 8 GB laptop GPU, a 284B model on a 32 GB desktop card and GLM-5.2 at 753B on a single 96 GB workstation GPU. The number that decides whether it works on your machine is not your VRAM. It is your system RAM, and it has to hold the entire model.
FreeToken is an open-source MoE serving engine with eleven authors, including Song Han of MIT, and Matei Zaharia and Ion Stoica of UC Berkeley, with Shuo Yang as first author. The paper itself prints no affiliations at all, so "a joint team from MIT and UC Berkeley", which is how most of this week's coverage describes it, is secondhand compression rather than something the authors wrote. One corresponding address on the paper is UT Austin. The paper went up on arXiv on August 17, 2026, and the code is on GitHub under Apache 2.0, first tagged release August 19. It went around local AI social feeds this week on three headline figures. Every one of those figures comes with a second number that almost nobody has quoted, and that second number is the whole story for anyone deciding what to buy.
The Table Everyone Cropped
Here is every machine the paper evaluated on, taken from its hardware table:
| System | GPU | VRAM | Host RAM | CPU |
|---|---|---|---|---|
| 4060 laptop | RTX 4060 Laptop | 8 GB | 32 GiB LPDDR5 | Core i9-13900H |
| 3090 server | RTX 3090 | 24 GB | 180 GiB DDR4 | 2x Xeon Gold 6330 |
| 4090 server | RTX 4090 | 24 GB | 240 GiB DDR4 | 2x Xeon Platinum 8358P |
| 5090 server | RTX 5090 | 32 GB | 180 GiB DDR5 | 2x Xeon Gold 6459C |
| 5090 desktop | RTX 5090 | 32 GB | 192 GiB DDR5 | Ryzen 9 9950X3D |
| PRO 6000 | RTX PRO 6000 | 96 GB | 512 GiB DDR5 | Xeon Platinum 8559C |
The headline decode figures, 77 to 83 tok/s on Qwen3.6-35B-A3B and 22 to 25 tok/s on DeepSeek-V4-Flash, were both measured on the 5090 server, the one with 180 GiB of DDR5 behind two Xeons. The 5090 desktop is the machine the abstract means by "a gaming desktop", and the paper puts the cost of moving from the server's many memory channels to that desktop's dual channel at about 4% of decode rate. The 8 GB laptop figure of 39.3 tok/s is Qwen3.6-35B-A3B in NVFP4. GLM-5.2 at 753B ran at 14.9 tok/s on the PRO 6000, against llama.cpp's 7.3 on the same box.
Three of those machines are dual-socket Xeon servers. The workstation box is a single Xeon Platinum 8559C with 48 threads and 512 GiB of DDR5. The two consumer machines are a Ryzen 9 9950X3D desktop with 192 GiB and a Core i9-13900H laptop with 32 GiB.
Read the RAM column, not the VRAM column. "Runs a 284B model on a single RTX 5090" is true. It is also a machine with 180 GiB of system memory, which is roughly four times what a gaming desktop ships with.
The Rule: RAM Holds the Model, VRAM Decides the Speed
FreeToken does not require the model to be resident in VRAM. Expert weights live in host memory and move to the GPU as the router calls them, so VRAM works as a cache rather than as the container. That inverts the sizing question this site normally answers.
Run our own numbers against their hardware and the rule falls out cleanly. These are what each model needs fully resident, from the VRAM calculator, llama.cpp at 32K context:
| Model | Resident requirement | FreeToken's host RAM |
|---|---|---|
| Qwen3.6 35B-A3B at Q4_K_M | 21.6 GiB | 32 GiB |
| DeepSeek-V4-Flash 284B at native 4.7 bpw | 159.4 GiB (171 GB) | 180 GiB |
| GLM-5.2 753B at Q4_K_M | 437.4 GiB | 512 GiB |
Every row is the model plus headroom. FreeToken's own documentation never states a minimum, and we checked the README, the install guide, the model docs, the quickstart and the CLI reference before saying so, so this is our reading of their hardware table rather than a number they publish: the practical requirement is system RAM larger than the whole checkpoint, after which your GPU decides how fast it runs rather than whether it runs at all. The one independent data point either way supports it. A tester with 64 GB watched the 284B loader walk the checkpoint and give up.
It also means the sizing number you already have is the one you need. Put a model into the GPU checker, read the total, and compare it against your RAM instead of your card.
The paper's own account of why a rival engine fails makes the same point better than we can. KTransformers has no way to serve GLM-5.2 on that 96 GB workstation, and the reason the authors give is not the GPU. Its methods want 753 GB to 1.5 TB of host-resident experts against the machine's 512 GiB of system memory. The card was never the binding constraint.
There is a second lesson in that table for anyone about to buy RAM. The two 5090 rows use identical GPUs and differ only in the host, and moving from the server's many memory channels to the desktop's two costs FreeToken 4% of its decode rate. It costs llama.cpp 20%, because its CPU-resident experts starve on two DDR5 channels. How many channels your RAM runs across turns out to matter, not just how many gigabytes you bought.
What It Does Not Do
Two limits will hit our readers before any of the speed numbers matter.
FreeToken runs dense models. It just cannot do the trick on them. The memory sleight of hand is expert placement, so a dense model gives it nothing to move: every token uses the whole network, and the whole network has to be somewhere fast. @thefp4brain made that point on X on August 23. What follows is a sizing rule, not a compatibility one. Qwen3.8 27B, the most-run model on this site by a wide margin, is dense, and FreeToken's own desktop app added it to its catalogue on August 23 in NVFP4, FP8 and BF16. You can load it. You still need the room for it, exactly as you would in llama.cpp, and an 8 GB card is not going to hold a 27B because FreeToken is the one asking.
GGUF is mostly not the path. FreeToken loads HuggingFace safetensors directly, and the docs name MXFP4, NVFP4, FP8 and BF16 as the quantization formats it expects. The one GGUF exception the docs state is native GGUF loading for Gemma-4. Everything else that quantizers publish for consumer cards has to arrive as safetensors, and that showed up immediately in first-hand reports: one tester got Ornith-1.5-35B-A3B running in NVFP4 but could not load an uncensored GGUF build, hedging that he might have done it wrong, and another had to modify FreeToken itself, with Codex doing the work, before it would read the MXFP4 GGUF he already had.
What Independent Testers Measured
The paper claims 1.8 to 2.3x decode over the strongest baseline on Qwen3.6, 1.5 to 1.9x on DeepSeek-V4-Flash, and 2.0x over llama.cpp on GLM-5.2, with worst-turn TTFT staying under 44 seconds where llama.cpp reached 232 seconds, Ollama 179, and KTransformers 946. Those are the authors' own numbers on the authors' own hardware.
Six people published their own within 48 hours, and they agree with the shape of it:
@bountyAIhunter, RTX 3090 with 24 GB and 64 GB of DDR4. Qwen3.6-35B decoded at 51.4 tok/s against llama.cpp's 28.7 on the same box, and TTFT on a 21,000 token prompt fell from 37.2 seconds to 9.4. He puts that at 1.79x and points at the right number to compare it against: the paper's headline 1.8 to 2.3x is measured on a 5090, while Section 5.3 gives 1.3 to 2.1x across its five consumer systems. His 3090 lands inside that one. Then DeepSeek-V4-Flash never produced a token on the same box: the loader walked the checkpoint and gave up, because that model wants 180 GiB and he has 64. His own conclusion is the one we would have written: buy RAM before you buy the next GPU.
He also reports it releasing VRAM back to other applications, with decode falling from 51.4 to 31.6 tok/s while a browser was open.
His 1.79x is above the paper's own 3090 result, and the comparison is not clean enough to call it a win. Section 5.3 gives the per-machine figures: 1.3x on the RTX 3090 and 4090, 1.9x on the 5090 server, 2.1x on the 5090 desktop, 1.8x on the 4060 laptop. The 3090 is the paper's weakest system. But the paper is measuring a coding-agent workload against the strongest of three baselines on a 180 GiB dual-Xeon box, and he measured one prompt against llama.cpp on 64 GB of consumer DDR4. Two numbers on the same model of card, from two different tests.
@TeksEdge, RTX 5080 with 16 GB, a Ryzen 9 9950X3D and 64 GB of system RAM. Qwen3.6-35B-A3B in NVFP4 is about 20 GB, so it does not fit the card at all, and he measured roughly 100 tok/s, with one 1,028 token prompt reaching about 110. That is the clearest demonstration of the rule in the whole set: the model is larger than the VRAM, the RAM holds it comfortably, and the card sets the speed.
@PaulGugAI, RTX 3080 with 10 GB. His comparison is the one nobody else ran, because it is not FreeToken against llama.cpp at the same quantization. It is NVFP4 Gemma 4 26B-A4B under FreeToken against the aggressively quantized IQ2_XXS build he had been running in LM Studio, which is what a 10 GB card forces you into. Output went from 19.8 to 27.7 tok/s at 8K context, and from 18.5 to 27 in his light agentic loop, with prompt processing 2 to 2.5 times faster. His summary is that a 4-bit class model is now beating a 2-bit one on the same card. The speed is the smaller half of that result. The quality tier he can run went up.
@twittyoota, 48 GB of VRAM. DeepSeek-V4-Flash at roughly double llama.cpp, still slower than the M2 Ultra he normally uses.
@kyuphd, RTX 5060 Ti 16 GB. Ornith-1.5-35B-A3B in NVFP4 at 32.0 tok/s over a 1,609 token run, with a momentary peak near 38. He got it running, in his words after a fashion, and an uncensored GGUF build of the same model would not load for him at all.
He also ran the paper's own headline model on that card: Qwen3.6-35B-A3B in NVFP4 at 26.0 tok/s over a 2,394 token run, peaking near 40. The paper's 39.3 tok/s for that model is a laptop 4060 figure and the two are not measured the same way, but a 16 GB desktop card sitting under a laptop's published number is worth knowing before you buy one.
@shiina_chang, RTX 5090 with 192 GB in four 48 GB sticks running at 3600 MT/s. After modifying FreeToken itself so it would load an MXFP4 GGUF of huihui's DeepSeek-V4-Flash, 25 tok/s, against 4 tok/s for the same work in LM Studio.
Every figure in this section is somebody else's measurement, attributed by handle. We have not run FreeToken.
Requirements, and the Windows Question
There are two FreeTokens, and most of this week's confusion comes from mixing them up.
The Python package is the one the paper is about:
uv pip install "freetoken[accel]"
Its documented requirements are Linux x86_64, Python 3.10 or newer, an NVIDIA driver at r580 or newer, and a CUDA 13 toolkit with nvcc on your PATH, because the CUDA kernels compile the first time you run them. There is no Windows build of it, and one independent tester put that first on his list of things that will cost you an evening.
The README's line about Windows is about the other thing: a desktop application, distributed from a separate repository, which does ship Windows builds alongside Debian and Arch packages. Both statements are true, and the reason people are contradicting each other this week is that they are quoting different products. If you want the engine described in the paper, you want Linux. If you want the app, Windows is fine.
Card support on the package side is native for RTX 30, 40 and 50 series.
Is It Worth Buying RAM For?
For a specific kind of reader, yes, and the arithmetic is unusually clean.
192 GB of memory is real money, and the criticism doing the rounds is fair: one commenter on the launch thread put the RAM at 3.5 to 4K without naming a currency, and said the single-5090 framing was useless without obscene amounts of it. Treat that as a floor rather than a ceiling. Memory prices rose through 2026 and a 128 GB DDR5 kit has been tracked above $3,000, so price the kit yourself before deciding. Against that, a GPU with 180 GB of VRAM does not exist at any consumer price, and the models in question are the ones people currently rent a datacentre card to touch.
The honest split:
- You already have 64 GB or less. FreeToken gets you a faster 35B, and on a 16 GB card @TeksEdge measured about 100 tok/s on exactly that setup. A reply on the 3090 thread says the 284B can be coaxed onto 64 GB with disk streaming, at a cost that eats most of the gain. Worth installing, not worth a hardware purchase.
- You are choosing between more VRAM and more RAM. For MoE models specifically, RAM now buys capability that VRAM cannot buy at the same price. This is the first release where that is measurably true rather than theoretically true.
- You run dense models. FreeToken will load them, and the memory story does not apply. Size normally.
This is the fifth technique we have covered that trades resident memory for something slower: CPU offload, disk streaming, quantization, runtime flags, and now bandwidth-adaptive expert placement. Each one moved the floor. None of them removed it.
Sources
- The paper: FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution, Yang et al., arXiv 2608.16157, August 17 2026. Hardware table, decode figures, speedups and the Section 5.3 per-machine results all come from here.
- The code: FlashML-org/FreeToken, Apache 2.0. Install requirements, supported formats and the model list come from its README and docs.
- Every measurement that is not the paper's belongs to the person named beside it on X: @bountyAIhunter, @TeksEdge, @PaulGugAI, @kyuphd, @shiina_chang, @twittyoota, @thefp4brain and @3DPBelgian. Each is one tester's own report on their own machine, published within 48 hours of the release, and none of it has been reproduced by us.
FAQ
How much VRAM does FreeToken need?
The paper's smallest configuration is an 8 GB RTX 4060 laptop GPU running Qwen3.6-35B-A3B in NVFP4 at 39.3 tok/s. There is no stated minimum, because VRAM sets the speed rather than the limit. The binding requirement is system RAM large enough to hold the whole model.
How much system RAM does FreeToken need?
More than the model's total size. The paper pairs 32 GiB with a 35B, 180 GiB with a 284B and 512 GiB with GLM-5.2 at 753B. A tester with 64 GB could not load the 284B model at all.
Is FreeToken faster than llama.cpp?
On the models it supports, yes. The paper reports 1.8 to 2.3x decode on Qwen3.6 and 2.0x on GLM-5.2. An independent test on an RTX 3090 posted 51.4 tok/s against llama.cpp's 28.7 on the same machine, which sits inside the 1.3 to 2.1x the paper reports across its five consumer systems.
Does FreeToken work with GGUF files?
Only for Gemma-4. The repository's model docs say FreeToken loads HuggingFace safetensors directly, plus native GGUF for Gemma-4. For everything else you want a safetensors checkpoint in MXFP4, NVFP4, FP8 or BF16. One tester modified FreeToken himself, with Codex doing the work, so that it would load an MXFP4 GGUF checkpoint.
Does FreeToken help with dense models like Qwen3.8 27B?
It runs them. The repository lists dense checkpoints including Qwen3.6-27B and Gemma-4, and the desktop app added Qwen3.8-27B on August 23, 2026. What it cannot do on a dense model is the expert offloading that produces the headline numbers, because there are no experts to place. Size a dense model exactly as you would for llama.cpp.
What models does FreeToken support?
The paper says more than 20 MoE models, and names Qwen3.6-35B-A3B, DeepSeek-V4-Flash and GLM-5.2 in its evaluation.
Is FreeToken free?
Yes. The code is open source under the Apache 2.0 licence. The models you run under it carry their own separate licences, which are not affected by FreeToken's.