Nemotron 3.5 Bolt: Why a 74B Fits Where a 70B Can’t

How much VRAM does Nemotron 3.5 Bolt need?

On a 24GB card: 21.1 GB at 8K, 22.2 GB at 128K, and the full 262,144-token window at 23.3 GB, 97% of the card, headless. One llama.cpp flag, a Q8_0 KV cache, brings the full window back to 22.6 GB and 94%, monitor on. Nemotron 3.5 Bolt is the 74B model NVIDIA released on 15 October 2026 with "24GB" in its name, and the name undersells the GGUF: the file leaves just enough room on a 24GB card for the entire advertised window. Not with your desktop running on top. That is what the flag is for.

Every figure below is a live read of our tools on the evening of launch day, after our catalogue entry followed the file it is named for: the GPU checker for the card rows, the model-first calculator for the breakdowns, all on the llama.cpp backend, one card. Bolt is a Mamba hybrid, and only 8 of its 88 blocks cache the full context, which is why a 74B model stays this flat.

Read that against the launch posts. @0xBakeer described Bolt as a model "that fits in 24GB VRAM on a single RTX 3090 … 256K context." On tonight's reads both halves hold at once, with an asterisk the size of your desktop: at the full window the F16 cache leaves 97% of the card used, which our checker passes only as a headless fit, since it keeps a 5% reserve for the runtime. The last two rows are the whole story of launch night: quantize the cache with -ctk q8_0 -ctv q8_0 and the same window fits at 94% with room to spare. Cache quantization is a speed trade as well as a memory one; our llama.cpp flags page carries the measured costs.

Why a 74B model behaves like a much smaller one

Bolt is 74B parameters with 9B active, a Mixture-of-Experts hybrid that stacks Mamba-2 blocks, attention blocks and LatentMoE blocks. The count that matters for your card is the middle one. Of its 88 blocks, 8 are attention and carry a KV cache, 40 are Mamba-2 and hold a fixed-size state, and 40 are MoE and hold nothing per token.

So the cache grows at 8 KB a token, not at the rate a dense 74B transformer would produce, and the 40 Mamba blocks keep a constant 164.69 MiB of state whatever the window. From 8K to 128K the requirement climbs about 1.1 GB, 21.1 to 22.2, instead of the several-fold jump a full-attention model of this size would need. The model-first calculator shows the split at 8K: 20.4 GB of weights, 0.2 GB of cache, 0.5 GB of runtime overhead. Push the window to 262,144 tokens and the cache segment alone prints 2.4 GB, which is what eats the last of the card, and 1.5 GB with the Q8_0 cache, which is what gives it back.

This is the skeleton the family runs on: Nemotron 3.5 Lightning, the 30B-A3B model we sized in August, caches 6 of its 52 blocks, and Super VL below caches the same 8 of 88.

Why a 74B fits where a 70B does not

Llama 3.1 70B is the comparison everyone reaches for, and in our catalogue it is the opposite build: a dense model with 80 layers, every one of them attention, 8 KV heads and a head dimension of 128. At the F16 default each token costs 80 x 8 x 128 x 2 x 2 = 327,680 bytes, 320 KB. Bolt caches in 8 of its 88 layers, with 2 KV heads of the same dimension, which comes to 8,192 bytes a token, 8 KB. At 131,072 tokens that is 40 GiB of cache for Llama against 1 GiB for Bolt.

The GPU checker shows the result on an RTX 3090, read on 15 October 2026 with llama.cpp and an F16 cache. Llama 3.1 70B at 8K "needs ≥ 25.9 GB at Q2_K", so it does not fit a 24GB card even at 2-bit. Bolt at 128K reads "Q2_K · 22.2 GB (92%)", sixteen times the context on the same card.

The other difference is how much work each token does. Llama is dense, so all 70B parameters run on every token. Bolt is a Mixture-of-Experts with 9B active. Every figure in this section is for a model held fully in VRAM.

Two builds called 24GB, and the tight one is NVIDIA's

The "24GB" in the name is NVIDIA's design target and its official build: a 24.08 GB safetensors checkpoint, designed to fit into 24GB of memory and decode at 80 TPS or better. Our calculator carries it on the NVFP4 rung, which the tool registry labels "NVFP4 Blackwell only," and it is the heavier of the two routes on a 24GB card: 23.0 GB at 8K, 96% of a 24GB Blackwell card, flagged TIGHT and headless (22.3 GB of weights, 0.2 of cache, 0.5 of overhead), and 24.1 GB at 128K, past the pool. That route is the vLLM path, an RTX 5090 Laptop or an RTX PRO 4000 Blackwell 24GB running headless at short context. On a 3090 or 4090 the checker routes you to the GGUF instead; we have not tested the official build on a non-Blackwell card and make no claim about it.

The GGUF is where the full-window story lives. NVIDIA's own single-file build is 22.02 GB plus a 1.73 GB mmproj for vision, and on launch evening Unsloth's repo shipped the same file, byte count matching to the last digit: 22,018,326,528. The swap happened at 17:10 UTC. Unsloth removed its own ladder, in the words of the change that our catalogue entry cites, to "Ship NVIDIA's official IQ2_XXS with the LiLiCorr drafter; remove the other quants and DSpark." The guide printed the old four-rung ladder on launch day; it has since been replaced by the single IQ2_XXS row, one more sign the repo swap was real:

For most of a day the repo carried that single official file. On 16 October the ladder came back, rebuilt around it: the UD rungs now start at 18.5 GB with UD-IQ1_S, and UD-IQ2_XXS is 22.02 GB, the same rounded size as the official IQ2_XXS. Our checker's Q2_K row reads the repo: 22.02 GB, the official IQ2_XXS, which is why the fit numbers above land about a GB under what the launch-day ladder suggests. The guide budgets about 32GB of RAM or unified memory for short-context use and 48 to 64GB for headroom, which stays sane advice once the mmproj and a drafter join the model. The drafter question that was open at launch closed the same evening: the repo's 1.00 GB dflash-NVFP4 file is the LiLiCorr route, and the DSpark default the guide described on launch day (833MB, about 1 to 2GB extra) went out with the old ladder; the guide now names the LiLiCorr file and its 998MB.

What Bolt does on the rest of the hardware

A 32GB card turns the 97% row into comfort: on an RTX 5090 the full 262,144-token window needs 23.3 GB, 73% of the pool, F16 cache, no flags, monitor, browser, the lot.

16GB does not run Bolt at any published rung. The Q2_K row needs 21.1 GB against a 16 GB pool, and the checker asks for at least 22.3 GB with its reserve. If that is your card, the GPU checker will show you what it does run.

Not every card sold as 24GB holds 24. The L4 gives 22.5 GB with ECC on, and that changes the verdicts: Bolt runs there at 8K (21.1 GB, 94%) and up to 128K headless (22.2 GB, 99%), but the full window misses even with the Q8_0 cache, where the checker asks for at least 23.9 GB. The 256K-on-24GB story is a 3090, 4090 and consumer-Blackwell story, not an every-24GB-card story. We ran exactly this on an L4 the day after launch, and every verdict held; the measurements are two sections down.

The DGX Spark takes Bolt at every context without working, because its usable pool is 112 GB against the 128GB on the label. The Q2_K row sits at 21.3 GB at 8K, 19% of the pool, and 23.5 GB at the full window, 21%. On a Spark the constraint NVIDIA designed for is not capacity at all: it is a machine that runs the model alongside your other applications, which is the whole point of the 24GB target.

We ran Bolt ourselves: an L4 and an RTX PRO 6000

On 16 October we tested Bolt on two cards. The L4 reports 23,034 MiB with ECC on, which is the 22.5 GB pool the checker uses for it. The second was an RTX PRO 6000 Blackwell, 96 GB, ECC on. Both ran driver 580.82.07 and llama.cpp b11696, the prebuilt CUDA 12.8 release, with every layer on the card (-ngl 999), flash attention on and one slot. The file was Unsloth's IQ2_XXS as the repo served it that morning, 22,018,326,464 bytes; NVIDIA's own GGUF is 64 bytes larger. Each load answered one request whose prompt filled 90% of the window, and nvidia-smi was read every quarter second for the peak. We wrote the calculator's L4 verdicts down before the run.

Six verdicts, six as written. On every load that fit, the calculator sat about 0.5 GB above the measured peak, the safe side to miss on. The Llama row failed the way the 70B section predicts: llama.cpp asked for 24,817 MiB in one buffer and the L4 refused it. The Q8_0 row is the other half of the flag story, measured: the quantized cache saved 0.4 GB at 128K and cost decode speed, 25.4 tokens a second against 32.6.

On the RTX PRO 6000 the window stops mattering. Bolt peaked at 22.1 GB at 128K and 23.2 GB at the full 262K window, against 22.2 and 23.3 on the calculator, and it ran fast: 124.4 tokens a second of decode at 128K and 109.5 at 262K, with prompts read at 4,852 and 4,334 tokens a second. Those are llama.cpp figures with no drafter, over a 128-token answer, so they are not comparable with NVIDIA's 83.99 above, which used the LiLiCorr drafter on an RTX Spark laptop.

Nemotron 3.5 Super VL: the 120B parent, and a 1M window on one Spark

Released the same day, Super VL is the 120B-A12B model Bolt was compressed from, with 12B active and a context window up to 1M tokens. It shares the 88-block Mamba-hybrid skeleton, so its cache stays small too, and it adds native Multi-Token Prediction for speculative decoding. NVIDIA's blog claims up to 5x faster throughput than others in its class on the Artificial Analysis leaderboard; that is the vendor's figure.

The interesting number for our readers is what fits on one DGX Spark. At Q4_K_M, a 70.10 GB build from lmstudio-community, Super VL fits a single Spark at every context we ran: 66.5 GB at 8K (59% of the 112 GB pool), 68.7 GB at 262K (61%), and 75.5 GB at the full 1M (67%). Step up to Q5_K_M (110.70 GB) and it still fits one Spark to 262K, at 106.9 GB (95%), but the 1M window pushes it to 113.7 GB, over the pool. Q6_K and Q8_0 miss a single Spark outright, at 114.3 GB and 121.4 GB.

Two linked Sparks take Q8_0 with room: 121.4 GB at 8K, 54% of the 224 GB pool, and 130.4 GB at 1M (58%). A single 80GB datacenter card such as an H100 PCIe runs the Q4_K_M build at 66.2 GB (83%) and 68.4 GB at 262K (86%). The BF16 reference weights are 241.52 GB, which is a 4x H100 or 2x GB200 job per NVIDIA's own recipes, not a local one.

The download sizes span publishers and they do not agree, so name the file. BF16 is 241.52 GB and Q8_0 128.47 GB from ggml-org; Q4_K_M is 70.10 GB from lmstudio-community; Unsloth's UD rungs run 120.99 GB (UD-Q6_K_XL), 110.70 (UD-Q5_K_XL), 65.76 (UD-Q3_K_XL) and 55.88 (UD-IQ2_M). NVFP4 lands near 75 GB depending on who built it: NVIDIA's own NVFP4 repo totals 75.01 GB, lmstudio-community's GGUF 75.24, ggml-org's 78.27 and Unsloth's 78.98. That spread is why @UnslothAI's "runs on 82GB RAM" and @wikiwayne's "75 GB of NVFP4 … to fit a single DGX Spark" are both defensible: they are different files. Our calculator carries the ladder so you size the one you actually downloaded, which is the same trap we walked through for DeepSeek V4 Flash on the Spark.

Super VL on one RTX PRO 6000: the full 1M window, and llama.cpp against TensorFold

We tested Super VL on the same RTX PRO 6000 Blackwell, from ggml-org's NVFP4 GGUF (76,918,795,552 bytes), with the method above. It fits one 96 GB card at every window, the full 1M included.

The Mamba skeleton shows up in hardware here. Going from 128K to the full 1M window cost 7.9 GB, on a 120B model. We read the calculator's figures for this card after the run, with no change to the calculator in between, and they sat 0.86 to 0.94 GB above each peak.

Then we put a second engine on the same card. TensorFold 1.0.12, the inference server that promises its draft tokens never change a token of the reply, served Super VL from NVIDIA's own NVFP4 checkpoint (75.06 GB). Its README lists that model for one or two GB10s, the DGX Spark chip, not for this card. It loaded anyway, with 72.01 GiB of model in VRAM. Both engines then got the identical request: the same document, 113,108 tokens at the 128K window and 228,708 at 262K, the chat template with thinking off, and a question asking for about 400 words, temperature 0. Every answer stopped on its own, between 333 and 434 tokens.

Three results, and they point different ways. TensorFold reads the prompt faster: 1.7 times llama.cpp at 128K and 1.5 times at 262K, which on a 228,708-token document is the difference between about 66 seconds and 44 before the first word. Token for token, llama.cpp decodes faster: with drafting off on both sides it leads 72.7 to 50.1 at 128K and 68.9 to 43.6 at 262K. TensorFold's drafts turn that around, to 96.3 against 72.7 at 128K and 85.7 against 68.9 at 262K, and they changed nothing in the answer: with drafts on and off its replies were byte-identical at both windows.

Two cautions belong with that table. The document is one sentence repeated thousands of times, which is easy to draft; TensorFold accepted 226 of 490 drafts at 128K and 282 of 518 at 262K, and real prose may accept fewer. And TensorFold's rates are timed from its token stream, so they include the cost of streaming, while llama.cpp's come from its own server timings. Memory is a wash: within 0.2 GB with drafts on, and the drafter costs TensorFold about 1.9 GB.

NVIDIA's own benchmark tables, and the rows they lost

Every speed and accuracy figure in this section is NVIDIA's own. The company measured Bolt with llama.cpp on Windows 11 ARM x64 on one RTX Spark, a Surface Laptop Ultra with 128GB, against three models its research page says were chosen because they fit within 24 GB of memory at 4-bit quantization.

On AIPerf SPEED-Bench-Coding, Bolt with LiLiCorr scored 83.99 tokens per second, behind Qwen3.6 35B-A3B at 96.63 and Gemma 4 26B-A4B at 100.56, and ahead of the dense Qwen3.8 27B at 28.26. On the AA AgentPerf Local replay, where lower is better, Bolt took 600.80 seconds against 453.97 for Qwen3.6 and 480.68 for Gemma 4, with the dense Qwen3.8 27B at 1043.28. NVIDIA designed Bolt for at least 80 TPS, reports 84 on the AIPerf bench, and frames the model as "a well-balanced option on the Pareto frontier of intelligence & speed for small models." The LiLiCorr drafter behind that number is 1.13B parameters and reaches a mean acceptance length of 4.68 greedy and 4.25 probabilistic at seven drafted tokens.

On accuracy, the Bolt card's 2-bit column scores above both peers its own selection picked for the 4-bit class on Terminal-Bench 2.1 (50.98 against 44.38 and 37.22), BrowseComp (66.75 against 48.74), GPQA Diamond (86.43 against 83.40) and GDPVal-AA-V2 (28.47 against 25.75). The row it loses is PinchBench: 87.82 to Qwen3.6's 88.07, a gap of 0.25.

The sharper finding sits on the Super VL card, and @superalesha put it in front of 11,355 people on launch day: NVIDIA compared its new 120B model against Chinese models in its own table, and Qwen3.8 Flash-Next came out higher on the rows he counted. The three he named match the card exactly.

The same card also gives Qwen3.8 Flash-Next 78.70 to Super's 71.10 on SWE-bench Verified, 89.0 to 78 on BrowseComp and 1561 to 1211 on GDPval-AA v2. Super VL holds rows too: GraphWalks, 89.71 to 76.1, and SciCode's subtask score, 47.13 to 46.6. His closing line was that if you want the best model for one RTX 3090 today, it is still the Chinese one, "By Nvidia's own numbers," and that he is still happy about the release, because open weights, open training data and a commercial licence beat no release.

The community read the tables the same way and argued about the frame. On the r/LocalLLaMA release thread the first fight was over who NVIDIA chose to compare against. Several users noted that a dense Qwen3.8 27B at Q4 also fits 24GB: u/Thiom wrote that the Qwens "blow Nemotron" away, and u/Ledeste that "they're comparing MoE, how convenient." The defence, from u/Artic_Temperature203, u/Expensive-Paint-9490 and u/FullstackSensei, was that a dense 27B is far slower and more verbose and that Bolt targets low-bandwidth RTX Spark laptops, where a dense 27B cannot keep pace. u/ga239577's framing was that Bolt was built to compete with Qwen3.6 35B-A3B, not the dense 27B. That is exactly the comparison a sizing page has to keep straight, and it is why every row in our tables names the file and the card.

The 2-bit bet, and what shipped around it

Bolt's premise, in NVIDIA's words, is that for a given size limit it is better to compress a large model into the 24GB budget than to train a smaller one from scratch. The company used Star Elastic to cut the parent's expert top-K from 22 to 9, then quantized the experts to IQ2_XXS at 2.0625 bits per element with quantization-aware distillation, kept the Mamba linear layers at NVFP4, and held the embeddings, attention and output head at Q8_0. The accuracy rows above are NVIDIA's case that the bet pays. The thread's instinct was to check it: u/jacek2023 posted a BF16-against-2-bit comparison with the caption "before you start screaming." At 2 bits the quality question outranks the fit question. NVIDIA's compression budget targeted a footprint below 24 GB at batch size 1 and 128K context, and Unsloth's guide also stops its context advice at 128K.

The plumbing landed on launch day. llama.cpp pull request #30520, "Add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support," merged on 15 October, so the GGUF route works now, and NVIDIA lists Ollama, LM Studio, Unsloth and vLLM alongside it. Bolt is built for the RTX Spark laptops that go on sale 16 October and also runs on GeForce RTX PCs, DGX Spark, DGX Station and RTX PRO workstations. Both models carry the OpenMDW-1.1 licence, and both cards say commercial or non-commercial use is fine. The smaller Bolt E4B 4GB, aimed at conversational and voice agents on GeForce RTX, Jetson and Spark, follows on 23 October.

If you are sizing a card around this launch, run your own numbers rather than trusting a name: the VRAM calculator takes the model, the context and the backend, and the GPU checker starts from the card you already own. For the Qwen side of these tables, see our Qwen3.8 VRAM page and our rundown of the Chinese open-weight models that keep topping them.

FAQ

Does Nemotron 3.5 Bolt fit on a 24GB card?

Yes, at every context NVIDIA publishes. On an RTX 3090 or 4090 the 2-bit GGUF needs 21.1 GB at 8K (88% of the card), 22.2 GB at 128K (92%) and 23.3 GB at the full 262,144-token window. That last row is 97% of the card, so with the default F16 cache it is a headless fit; add -ctk q8_0 -ctv q8_0 and it drops to 22.6 GB (94%) with headroom. Cards sold as 24GB with ECC, like the L4 at 22.5 GB usable, top out at 128K.

What context window does Nemotron 3.5 Bolt actually support?

NVIDIA's card says up to 256K tokens, its vLLM recipe sets --max-model-len 262144, and Unsloth's guide advises up to 128K. The two NVIDIA numbers are one window, since 256 x 1,024 is 262,144. On a 24GB card the full window fits headless at F16 cache or with headroom at Q8_0; a 32GB card holds it at 73% however you run it.

How fast is Nemotron 3.5 Bolt?

NVIDIA measured 83.99 tokens per second on AIPerf SPEED-Bench-Coding on one RTX Spark, with its LiLiCorr drafter, and designed the model for at least 80 TPS. On the same bench Qwen3.6 35B-A3B scored 96.63 and Gemma 4 26B-A4B 100.56, with the dense Qwen3.8 27B at 28.26. These are NVIDIA's own figures on NVIDIA hardware. Our own runs, llama.cpp with no drafter, measured 124.4 tokens a second of decode at 128K on an RTX PRO 6000 and 32.6 on an L4.

How does Bolt compare with Nemotron 3.5 Lightning?

Lightning is the 30B-A3B sibling we sized in August; Bolt is 74B-A9B, larger and built for the same 24GB class. NVIDIA's published speed table for Bolt lists Qwen3.6 35B-A3B, Gemma 4 26B-A4B and Qwen3.8 27B; a like-for-like number between the two Nemotrons needs both on one card at one quant and context. On the accuracy card, Bolt's 2-bit column scores Terminal-Bench 2.1 at 50.98 against the Lightning BF16 column's 23.46.

Can a DGX Spark run Nemotron 3.5 Super VL?

Yes, and it is the natural home for it. A single Spark, with the 112 GB usable pool our checker uses, runs the Q4_K_M build at every context: 66.5 GB at 8K and 75.5 GB at the full 1M. Q5_K_M fits one Spark to 262K at 106.9 GB. Q8_0 needs two linked Sparks, at 121.4 GB of the 224 GB pool. @wikiwayne's launch-day framing, 75 GB of NVFP4 built to fit a single Spark, lines up with the Q4 rows.

What is the difference between Bolt and Super VL?

Super VL is the 120B-A12B parent with a 1M-token window; Bolt is the 74B-A9B model compressed from it, quantized to 2-bit for a 24GB budget and a 256K window. Same 88-block Mamba hybrid, same small KV cache, different size and target hardware. Bolt is for a 24GB card and an RTX Spark laptop; Super VL is for a DGX Spark or a datacenter card.

Do these models take images and video?

Both are vision-language models. Bolt takes text, images and video in and produces text, with video as mp4 up to 2 minutes. In the GGUF route, vision needs the separate 1.73 GB mmproj file, so budget it on top of the weights. Super VL carries a C-RADIO vision encoder too (Bolt's card names the v4-H variant), and its published window is 1M tokens.

Why does a 74B model fit a 24GB card at all?

Because it is a Mamba hybrid and only 8 of its 88 blocks cache the full context. The other 80 are 40 Mamba-2 blocks holding a fixed 164.69 MiB of state and 40 MoE blocks holding nothing per token, so the cache grows at 8 KB a token whatever the window. A dense transformer caches on every layer. That is why this ladder is nearly flat and a dense 74B would not be.