How Much VRAM Does Nemotron 3.5 Lightning Need?
About 20.1 GB at the smallest published builds, and about 26.6 GB at Q4_K_M, including an 8K cache and llama.cpp overhead. Nemotron 3.5 Lightning is NVIDIA’s 30B-A3B Mamba hybrid, published on August 11, 2026 with roughly 3.6B parameters active per token. The number most people will want is the second one, because at Q4_K_M this model does not fit a 24 GB card, on any of the four repositories that publish it.
The more useful discovery is what happens when you try to shrink it. Almost nothing does.
| Build | File on disk | Total at 8K | 24 GB card | 32 GB card |
|---|---|---|---|---|
| UD-IQ1_M | 19.43 GB | 20.7 GB | Yes | Yes |
| UD-IQ2_XXS | 19.43 GB | 20.7 GB | Yes | Yes |
| UD-IQ2_M | 19.43 GB | 20.7 GB | Yes | Yes |
| UD-IQ3_XXS | 19.76 GB | 21.0 GB | Yes | Yes |
| UD-Q3_K_XL | 21.24 GB | 22.5 GB | Yes | Yes |
| MXFP4_MOE | 23.21 GB | 24.5 GB | No | Yes |
| UD-Q4_K_M | 25.27 GB | 26.6 GB | No | Yes |
| UD-Q5_K_M | 30.22 GB | 31.7 GB | No | No |
| Q8_0 | 35.00 GB | 36.6 GB | No | No |
File sizes are Unsloth’s published GGUF builds, read from the repository on August 12, 2026. bartowski, ggml-org and lmstudio-community publish their own ladders, quoted below where they differ. Totals add our KV cache figure and llama.cpp’s runtime overhead, which we model as a fixed 0.75 GB plus 2% rather than a flat percentage, because the bulk of it is CUDA context and compute buffers that do not scale with the model.
Ask for Less Precision, Get the Same File
Read the top of that table again. UD-IQ1_M and UD-IQ2_XXS are the same file size to the byte, 19,429,059,648 of them. UD-IQ2_M adds three megabytes and lands on the same 19.43 GB. A nominal 1-bit build, a 2-bit build and a larger 2-bit build, separated by a rounding error.
It is not a one-off. Further up the same ladder UD-Q3_K_XL and UD-IQ4_NL are also byte-identical at 21,235,202,112, and so are Q8_0 and UD-Q6_K_XL at 35,004,643,392. Three pairs in one repository, where the quantizer was asked for different precision and the file came back the same size.
There is a floor, and it sits at 18.84 GB. That is bartowski’s IQ2_XXS, the smallest file anyone has published for this model as of August 12. Unsloth stops at 19.43 GB. Between those two numbers is every 1-bit and 2-bit build in existence.
bartowski’s repository makes the point harder than Unsloth’s does. Ten separate quantization types, from IQ2_XXS up through Q2_K_L, land inside a quarter of a gigabyte of each other: 18.84, 18.84, 18.85, 18.85, 18.91, 18.92, 18.92, 18.94, 19.06, 19.09. Two of those are 4-bit builds. His IQ4_XS is 18.92 GB, which is smaller than his own IQ3_XXS at 19.80 GB. Ask for less precision and you do not get a smaller file. Sometimes you get a larger one.
This matters because of what people do when a model does not fit. The standard move is to drop a tier: Q4 too big, try Q3, still too big, try Q2. On most models each step buys two or three gigabytes. Here the entire span from 4-bit i-quants down to 2-bit is a quarter of a gigabyte. Someone on a 16 GB card can work down the whole ladder, destroy the model’s quality, and still be about four gigabytes short.
The Quant Name Has Stopped Predicting the Size
Which leads to the practical version of all this. Four repositories publish a build called 4-bit. They do not agree, and the spread is not small.
| 4-bit build | File | Total at 8K | 24 GB card |
|---|---|---|---|
| bartowski IQ4_XS | 18.92 GB | 20.2 GB | Fits |
| bartowski Q4_0 | 19.06 GB | 20.3 GB | Fits |
| unsloth UD-IQ4_NL | 21.24 GB | 22.5 GB | Yes |
| lmstudio Q4_K_M | 24.52 GB | 25.9 GB | No |
| unsloth UD-Q4_K_M | 25.27 GB | 26.6 GB | No |
| bartowski Q4_K_M | 25.48 GB | 26.8 GB | No |
A 6.6 GB spread between two files that both say 4-bit on the label. The i-quant builds fit a 3090 with room to spare. Q4_K_M does not fit on any of the four, running 25.9 to 26.8 GB in use. On this architecture the quantization name is no longer a size, and you have to read the byte count before you download.
Why the Floor Exists
Nemotron 3.5 Lightning is a Mamba hybrid, and that is the whole explanation.
Its config.json declares NemotronHForCausalLM across 52 blocks, but those blocks are not all the same thing. Only 6 of them run attention. Twenty-three are Mamba state-space blocks holding a fixed recurrent state, and the rest are feed-forward. Mixture of experts sits on top: 128 routed experts with 6 active per token, plus one shared expert.
Quantization compresses weight matrices well. It does not compress state-space parameters, convolution kernels or the shared-expert path anything like as well, and those components stay near full precision in every published build. So the compressible half shrinks, the incompressible half does not, and the total converges on a floor.
You can see the same thing from the other direction. Unsloth’s UD-Q4_K_M is 25.27 GB carrying 32.9 billion tensors, which is 6.14 effective bits for every parameter in the file, against the 4.85 that “Q4_K_M” implies on a conventional transformer. Measure it against NVIDIA’s headline 31.6B instead, which leaves out the 1.3B multi-token-prediction head that Unsloth ships inside the GGUF, and it comes to 6.40. Either way it is a quarter to a third above the label.
llama.cpp reads this architecture as nemotron_h_moe, and the project’s own ggml-org account published GGUFs on launch day, so none of this needs a patched branch.
This Model Broke Our Own Calculator, and We Found Out Writing This
Until August 12 our calculator returned about 22.1 GB for this model at Q4_K_M. The real file makes it 26.6 GB. The tool was telling people with a 24 GB card that it fits.
The mechanism is the one described above, arriving as an error in our own arithmetic. The calculator multiplied a parameter count by a bits-per-weight figure, which is right for a dense transformer and right for most mixtures of experts, and has no way to know that a large part of this particular model refuses to compress.
It is fixed. The engine now takes a per-model table of observed published file sizes and prefers them to its own arithmetic wherever one exists, falling back to the calculation everywhere else. Five Nemotron entries were seeded from Unsloth’s ladders; the audit that found them checked every hybrid in the catalogue and cleared the rest, with the Qwen hybrids and LFM2.5 landing inside two to three percent.
Publishing the miss costs us something. Doing it anyway is the only reason to believe the other numbers on this page, and the correction was cheap while that is not.
Long Context Is Nearly Free, Which Is the Other Half of the Story
The same architecture that blocks quantization does something generous with context.
Only 6 blocks hold a KV cache that grows with your conversation. The 23 Mamba blocks hold a state whose size is fixed no matter how long the input runs, which is the defining property of a state-space model. NVIDIA’s model card puts the ceiling at up to 1M tokens, noting that a single H100 deployment is served at 256K.
| Context | Cache, F16 | A 52-layer transformer with the same heads |
|---|---|---|
| 8,192 | 0.10 GB | 0.44 GB |
| 131,072 | 0.86 GB | 6.98 GB |
So the trade is legible once you see it. This model costs you a great deal more at the bottom of the quant ladder and a great deal less at the far end of the context window. Pick it for long documents, not for a small card.
What Fits Your Card
| Card | Best build | Verdict |
|---|---|---|
| RTX 5090 / RTX PRO, 32 GB | UD-Q4_K_M at 26.6 GB | The card this model wants |
| RTX 3090 / 4090, 24 GB | bartowski IQ4_XS at 20.2 GB | Fits, and it is nominally 4-bit. Unsloth’s UD-Q4_K_M at 26.6 GB is a different build |
| RTX 5080, 16 GB | none that fit | Builds exist down to 18.84 GB. None of them fit. The floor is above the card |
| Mac, 32 GB unified | UD-IQ3_XXS or Q3_K_XL | Comfortable, watch the wired-memory limit |
| 64 GB unified or 2x 24 GB | Q8_0 at 36.6 GB | Full quality, long context |
The 16 GB row is the one worth pausing on, because it is unusual. Normally a card that small means “use a smaller quantization”. Here the smaller quantizations are all the same size. Offload and KV quantization flags can move some of it to system RAM, at a speed cost that makes the exercise questionable.
A Strange Week for Models That Fit One Card
Nemotron 3.5 Lightning landed on August 11. Meta’s Muse Glimmer 30B landed the day before. Qwen3.8’s 2.4T flagship shipped on the 12th with a 27B sibling scheduled for Friday. Three labs, one week, all of them talking about hardware normal people own.
They did not arrive at the same answer. Muse Glimmer fits a 24 GB card at 4-bit with vision, the speculative decoding drafter and its full 128K context, at about 21.9 GB on Unsloth’s UD-Q4_K_XL. Nemotron 3.5 Lightning fits the same card only if you take one of the i-quant 4-bit builds; its Q4_K_M wants 32 GB. Same headline size, same week, two very different bills.
Which is the argument for reading the file sizes rather than the parameter count. “30B” told you almost nothing useful about either of them.
On capability, Artificial Analysis scored Nemotron 3.5 Lightning at 24 on its Intelligence Index, level with gpt-oss-120b, at roughly a quarter of the parameters and a nine-point jump over Nemotron 3 Nano. That is their own evaluation published on launch day, run on a pre-release endpoint that they disclose, rather than a number NVIDIA handed them. One independent evaluator on day one is still one, so it is worth a second run before anyone treats it as settled.
If it holds up, the practical reading is that a used RTX 3090, which our rent-versus-buy guide puts at around $700 on the second-hand market, now runs a model scoring level with something four times its size. Used-card prices move, so treat that as a ballpark from August 2026 rather than a quote. At 4-bit i-quants, with the caveats above, and that is still a remarkable sentence to be able to write. Our GPU buying guide covers the used 3090 case in detail.
The Other Thing That Happened This Week
On August 11, Anthropic published how Claude marks AI-generated content. New Claude models weave an imperceptible watermark directly into the text. It travels with the text when copied and pasted, may persist through some editing, applies across every Claude surface, and is worldwide rather than EU-only.
The regulatory backdrop is public and worth stating precisely, because it is usually stated loosely. The EU AI Act’s marking obligations took effect on August 2, 2026. Ahead of that date, roughly 190 organisations signed the European Commission’s Code of Practice on Transparency of AI-generated Content, a voluntary instrument whose signatories include Anthropic, Google, Meta, Microsoft, Mistral, OpenAI, Cohere, Aleph Alpha and Synthesia. About half are small companies.
Two distinctions the coverage tends to blur. The law requires machine-readable marking; the invisible in-text watermark is Anthropic’s chosen implementation of it, not a thing the EU specified. And signing a voluntary code is not the same as shipping a watermark, so do not read Anthropic’s method onto the other 189 signatories.
What that has to do with a VRAM article is narrow and, I think, real. When the weights and the runtime sit on your own hardware, nothing upstream is writing a marker into your token stream, because there is no upstream. That is a statement about control rather than capability, and the capability gap is genuine: nothing in this article claims a 30B on a 3090 matches a frontier model. But the two axes move independently, and this week only one of them moved.
There is a further question underneath it, about models modified to remove their refusal behaviour and what that means for people relying on them. That deserves its own article rather than a paragraph, and it is the next one we are writing.
Check what your own GPU fits, or size Nemotron 3.5 Lightning against your hardware in the calculator, which now carries the published file sizes for all five Nemotron entries.
FAQ
How much VRAM does Nemotron 3.5 Lightning need?
About 26.6 GB at Unsloth’s UD-Q4_K_M with an 8K context and llama.cpp overhead, or about 20.1 GB at the smallest published builds. The Q4_K_M file alone is 25.27 GB. At Q4_K_M it needs a 32 GB card, though some 4-bit i-quant builds are far smaller.
Does Nemotron 3.5 Lightning fit on a 24GB card?
Yes, but not at Q4_K_M. bartowski’s IQ4_XS is 18.92 GB on disk and lands at about 20.2 GB in use, which fits an RTX 3090 or 4090 with room for a longer context. Unsloth’s UD-IQ3_XXS at 21.0 GB also fits. What does not fit is Q4_K_M, which runs 25.9 to 26.8 GB depending on whose build you take. That one needs a 32 GB card.
Does Nemotron 3.5 Lightning fit on a 16GB card?
No, and unusually there is no smaller build to fall back on. The smallest GGUF anyone has published is bartowski’s IQ2_XXS at 18.84 GB, before the cache and runtime overhead. Dropping to 1-bit does not help, because his whole 2-bit family and both of his 4-bit i-quants sit inside a quarter of a gigabyte of each other.
Why are the 1-bit and 2-bit builds the same size?
Because this is a Mamba hybrid. Only 6 of its 52 blocks run attention; 23 are state-space blocks, and the model also carries 128 routed experts with a shared expert path. State-space parameters, convolution kernels and the shared expert do not compress the way weight matrices do, so they stay near full precision in every build. Once the compressible part is squeezed out, the total stops moving. The floor works out to 4.58 effective bits per weight against the 32.9 billion tensors the file actually contains, or 4.77 against NVIDIA’s headline 31.6B.
Why does your calculator show a lower number than this article?
It no longer does. Until August 12 it multiplied a parameter count by a bits-per-weight figure, which is right for dense transformers and most mixtures of experts and wrong for this architecture, and returned about 22.1 GB at Q4_K_M against a real 26.6 GB. The engine now prefers published file sizes wherever we have recorded them. That is the general rule here anyway: observed beats theoretical.
How long a context can it actually hold?
NVIDIA’s model card states up to 1M tokens, and notes that a single H100 deployment is served at 256K. Long context is cheap on this architecture: only 6 blocks hold a cache that grows, so 128K costs about 0.86 GB in F16 against roughly 6.98 GB for a conventional 52-layer transformer with the same head configuration.
How does it compare to Muse Glimmer 30B?
On memory they are not close, despite both being called 30B and launching a day apart. Muse Glimmer fits a 24 GB card at 4-bit with its vision encoder, its speculative decoding drafter and the full 128K context, at about 21.9 GB on Unsloth’s UD-Q4_K_XL, with nothing to spare. Nemotron 3.5 Lightning fits the same card only on an i-quant 4-bit build; its Q4_K_M needs 32 GB. Nemotron holds far more context for far less cache. Different tools.
Is it really as capable as gpt-oss-120b?
Artificial Analysis puts it at 24 on their Intelligence Index, the same score as gpt-oss-120b, at roughly a quarter of the parameters. They ran that themselves on a pre-release endpoint and say so. It is a single day-one evaluation on one index, which is a real result and not yet a consensus, so weigh it accordingly.
What quantization should I actually use?
On 32 GB, Q4_K_M from any publisher. On 24 GB, bartowski’s IQ4_XS at 18.92 GB, which is the best quality that fits and is nominally 4-bit. Do not reach for 1-bit or 2-bit builds on this model: they are within a quarter of a gigabyte of the 4-bit i-quants, so you would be giving up quality for nothing.
Does running it locally avoid the AI content watermarks?
A locally run open-weight model does not inherit a vendor’s marking pipeline, because there is no vendor in the path. That is a point about control rather than quality. The EU AI Act’s marking obligations took effect on August 2, 2026, and about 190 organisations signed the European Commission’s voluntary transparency code ahead of it, including every major lab. Anthropic has published the most detail on its own implementation, which embeds an imperceptible watermark in the text itself.