Inkling 2 Mini VRAM: The Hidden 12 GiB in llama.cpp

Thinking Machines heard the complaint. Inkling 2 Small was a 594B model, and on 20 November 2026 the lab shipped the answer to everyone who said it was too big: Inkling 2 Mini, a 26.4B mixture-of-experts model with 3.4B active parameters, open weights under Apache 2.0, small enough that the lab's own demo ran it offline on a MacBook Pro M5 Pro with 48 GB.

The scores are what made it spread. By the lab's figures it leads every open-weights model under 100B total parameters on DeepSWE v1.1 at 48.2% and on Vals Index v2.2 at 46.5%. Unsloth had GGUF files up on day one and llama.cpp added the architecture in b12600.

The model it has to beat on a home GPU is Qwen3.8-27B, the dense 27B most people with a 24 GB card already run. Here is Thinking Machines' own comparison from the model card:

Benchmark Inkling 2 Mini Qwen3.8-27B
Parameters, active / total 5B / 26B 27B / 27B
AA Index v5.0 20.1 20.1
DeepSWE v1.1 48.2% 42.2%
SWEBench Pro v2 (Hard) 77.9% 73.8%
SWEBench Verified 79.2% 77.8%
Terminal Bench v2.1 71.6% 73.0%
Vibe Code Bench v1.1 77.9% 64.9%
BrowseComp 79.0% 61.6%
MCP Atlas 82.7% 77.3%
LongBench v2 66.9% 52.3%
MMMU Pro 74.9% 76.3%
GDPVal-AA v2.1 1319 1427

These are the lab's figures, and the lab notes that evaluation setups vary by model. By its own summary, Inkling 2 Mini is ahead on agentic coding and behind only Qwen3.8-27B on the rest. The memory angle is the part that matters here: Qwen3.8-27B reads all 27B weights for every token, Inkling 2 Mini reads 5.4B. That is why it decodes fast. Whether it fits is the question below.

So we measured Inkling 2 Mini VRAM on an RTX 3090 the next day. The file sizes say a 24 GB card holds the 4-bit model with room to spare. On llama.cpp's default settings, every 128K setup we tried failed to load, down to the 2-bit file. The reason is a table no file size shows, and one flag decides how big it gets.

A 24-layer model that runs like 48

Inkling 2 Mini stores 24 layers in four blocks of six. The first and last blocks run once; the two middle blocks run three times for every token. The lab calls it "48 layer applications per token from 24 stored layers". The weights are stored once, so the file stays small. That is why a model with these scores comes in at about 20 GB.

Two numbers follow from the loop:

  • Active parameters. The lab gives 3.4B active and 5.4B "weight activations per token to account for looping". Both are right: 3.4B distinct weights a token, read 5.4B times' worth once the middle runs three times. Decode speed follows the 5.4B.
  • The cache. In llama.cpp (b12600), each pass of the loop keeps its own KV cache, so the 24 stored layers become 48 cache slots: 8 that see the whole context and 40 that keep only a 1,024-token window. With a 16-bit cache that is 1.19 GiB at 32K, 2.22 GiB at 64K, 4.28 GiB at 128K and 8.41 GiB at 256K.

That much a spec sheet can tell you. What it cannot is the next part.

What we ran

One RTX 3090 24GB (24,576 MiB, 21 MiB in use at idle), Linux, llama.cpp b12606 (commit e412c6f) from the project's own CUDA 12.8 build, flash attention on, one slot, automatic fitting switched off so llama.cpp could not quietly shrink anything, 256-token answers, greedy. The files were Bartowski's Q2_K, Q3_K_M, Q4_K_M and Q5_K_M and Unsloth's UD-Q4_K_XL, each checked byte for byte against the published size before use. Each setup read one prompt filled to about 85% of its window while we sampled VRAM every half second. Speeds are llama.cpp's own timings.

Before the run we wrote down what our calculator predicted for each setup, so the Inkling 2 Mini VRAM figures here are tested against a prediction made in advance. Then we let the card answer.

The table nobody priced in

Q2_K at 128K, the smallest file we had, refused to start. llama.cpp asked for a 13,484,916,992-byte compute buffer and ran out of memory. At 64K, Q4_K_M and Q5_K_M asked for 6,841,139,456 bytes and failed the same way. Of 12 setups at llama.cpp's default batch, 3 loaded and 9 did not. Every 128K and 256K setup failed, and every 4-bit or larger file at 64K.

The cause is in llama.cpp's own source for this model. Inkling 2 adds a learned bias to attention based on how far apart two tokens are. llama.cpp builds that bias as one full 32-bit table: attention heads, times every token in the context, times every token in the batch being processed, and the graph holds two copies. With Inkling 2 Mini's 24 heads and llama.cpp's default batch of 512 tokens, that is 2 x 24 x 131,072 x 512 x 4 bytes, 12.9 GB at 128K, on top of the weights and the cache. Flash attention does not avoid it; the table is built before attention runs.

The same code serves the bigger Inkling 2 Small, with twice the heads.

One flag takes it back

The table scales with the batch, and -ub sets the batch. Cut it, and the table shrinks in proportion. On the same card:

Setup Default batch (512) Smaller batch Peak VRAM
Q3_K_M, 32K, 16-bit cache 18.01 GiB -ub 128 15.61 GiB
Q4_K_M, 64K, 16-bit cache did not load -ub 128 21.21 GiB
Q3_K_M, 128K, q8_0 cache did not load -ub 128 18.98 GiB
Q2_K, 128K, q8_0 cache did not load -ub 128 16.48 GiB
Q4_K_M, 128K, q8_0 cache did not load -ub 64 21.23 GiB
Q4_K_M, 128K, 16-bit cache did not load -ub 64 23.18 GiB

With -ub 64, the 4-bit file runs at 128K on a 24 GB card. The price is reading speed. The batch is how many prompt tokens llama.cpp processes at once, and a small one reads a long prompt slowly.

The speed you pay

Setup Prompt Prefill Decode
Q3_K_M, 32K, default batch 32K 667.4 tok/s 51.32 tok/s
Q4_K_M, 32K, default batch 32K 655.1 tok/s 48.4 tok/s
Q3_K_M, 64K, q8_0, default batch 64K 460.3 tok/s 44.8 tok/s
Q4_K_M, 64K, -ub 128 64K 413.9 tok/s 43.77 tok/s
Q4_K_M, 128K, -ub 64 128K 255.6 tok/s 35.63 tok/s
Q2_K, 128K, q8_0, -ub 128 128K 268.3 tok/s 36.3 tok/s

The 32K and 64K rows are the median of three runs; the 128K rows are one run each. Decode holds up well: 48.4 to 51.32 tok/s at 32K and still 35.63 at 128K. Prefill is the cost. A full 128K prompt at 255.6 tok/s is about 8 minutes before the first token. For a coding agent that rereads a long repository on every turn, that matters more than decode.

Inkling 2 Mini VRAM by card: what our calculator says now

The calculator had the cache right and missed the table. It now charges it on llama.cpp: 3.00 GiB at 32K, 12.00 GiB at 128K, 24.00 GiB at 262K, at the default batch. After that change it lands within -1.7% to +0.1% of every peak we measured, and it gives the card's own verdict on all 12 default-batch setups.

What fits at llama.cpp's defaults:

Card What fits
16 GB (RTX 5060 Ti 16GB) Q2_K at 32K only, tight: 14.98 GiB with a q8_0 cache
24 GB (RTX 3090 / 4090) Q4_K_M at 32K (21.80 GiB), Q3_K_M at 64K (22.07 GiB); nothing at 128K
32 GB (RTX 5090) Q8_0 at 32K (30.28 GiB), Q4_K_M at 64K, Q3_K_M at 128K with a q8_0 cache (28.24 GiB)

Lower -ub and every row above moves up a step, at the prefill cost in the table before. The calculator answers the default because that is what a fresh install runs.

Does it run on 16 GB?

Unsloth's launch post says it "runs on 16GB RAM". Read their own table: it is labelled RAM plus VRAM, or unified memory, and it puts 4-bit at 17-18 GB. A 16 GB graphics card holds 14.9 GiB, and every 4-bit file is bigger than that before the cache and the table.

So on a 16 GB card, Inkling 2 Mini VRAM use rules out everything but 2-bit at short context on the card itself, or with experts spilled to system RAM. @niklaslenz_ai did the second on an RTX 5060 Ti 16 GB with ggml-org's IQ4_NL file at a 32K window: the model "doesn't fully fit, so llama.cpp puts some experts on the CPU", and decode came in at 43.1 tok/s on code and 43.2 on prose, with 267 tok/s of prefill at 8K. That is usable, and it is the price of a file that does not fit.

Where the big numbers come from

The fastest results so far are not on llama.cpp. @niklaslenz_ai ran an RTX 5090 32 GB in vLLM with the official Inkling 2 Mini image and DSpark drafts at a 93K window: 416.6 tok/s decoding code, 319.3 on prose, 17,536 tok/s of prefill at 8K and 954.7 tok/s across 8 requests, the median of 2 runs per prompt with 512 tokens.

@Tech2Wild pushed one DGX Spark to the full 1M context in vLLM with the official NVFP4 weights, DSpark and an FP8 cache: a 3.3M-token cache pool, a 950K-token needle test passed, about 47 tok/s for one user, 47.3 in his eval run, about 1,690 tok/s of prefill over 200K and 18.4 tok/s decode at 950K. The vLLM recipe Thinking Machines points to caps a Spark at 262,144 tokens; his build is an image from 20 November, version 0.1.dev25855, a pre-release of 0.35.

If you have the memory and want long context today, vLLM is where this model runs fastest. If you are on llama.cpp, set -ub.

What to run, by machine

  • 16 GB card: Q2_K at 32K on the card, or a 4-bit file with experts in system RAM at about 43.1 tok/s.
  • 24 GB card: Q4_K_M at 32K on defaults; Q4_K_M at 128K with -ub 64 if you can wait for prefill.
  • 32 GB card: Q8_0 at 32K, or Q3_K_M at 128K with a q8_0 cache.
  • Mac with unified memory: the lab's demo is a 48 GB MacBook Pro. Check yours on the Mac checker.
  • DGX Spark: the NVFP4 checkpoint in vLLM, up to the full 1M.

Every one of these is a live answer in the calculator: pick Inkling 2 Mini, your card and your context. What LLM can I run lists everything that fits a card, and the speed tool estimates decode from the 5.4B the loop actually reads.

The files

File Who Size
NVFP4 (the lab's own, Blackwell) Thinking Machines 19.53 GB, plus a 0.64 GB MTP head
BF16 Bartowski 52.23 GB
Q8_0 Bartowski 27.82 GB
Q6_K Bartowski 24.70 GB
Q5_K_M Bartowski 21.22 GB
Q4_K_M Bartowski 18.66 GB
UD-Q4_K_XL Unsloth 17.63 GB
IQ4_NL ggml-org 17.63 GB
Q3_K_M Bartowski 14.55 GB
UD-Q3_K_XL Unsloth 13.63 GB
Q2_K Bartowski 11.88 GB
UD-Q2_K_XL Unsloth 9.14 GB

Vision needs a separate 0.64 GB projector file. If you only chat with text, leave it out.

Sources

The RTX 3090 figures are our own run, described above. The RTX 5090, RTX 5060 Ti and DGX Spark figures are their testers' own, credited by name.

FAQ

How much VRAM does Inkling 2 Mini need?

Inkling 2 Mini VRAM depends on the file, the context and llama.cpp's batch. On llama.cpp defaults, Q4_K_M at 32K with a 16-bit cache needs 21.80 GiB, which fits a 24 GB card. Longer context needs a smaller -ub: with -ub 64 we measured Q4_K_M at 128K peaking at 23.18 GiB on an RTX 3090.

Why does llama.cpp run out of memory at long context?

It builds Inkling 2's position bias as a full table of heads x context x batch, held twice. At 128K and the default batch of 512 that is about 12.9 GB on top of the weights and cache. Lowering -ub shrinks it in proportion.

Does Inkling 2 Mini run on a 16 GB GPU?

On llama.cpp, Inkling 2 Mini VRAM use fits a 16 GB card only at 2-bit and short context on llama.cpp defaults: Q2_K at 32K is a tight fit. Every 4-bit file is larger than the 14.9 GiB a 16 GB card holds, so 4-bit means experts in system RAM.

How fast is it on an RTX 3090?

In our run, 48.4 tok/s decode and 655.1 tok/s prefill for Q4_K_M at 32K, and 35.63 tok/s decode with 255.6 tok/s prefill at 128K with -ub 64.

Is it 3.4B or 5.4B active?

Both, by the lab's own count: 3.4B distinct weights per token, 5.4B weight activations once the middle blocks run three times. Speed follows the 5.4B.

Did we run this?

The RTX 3090 numbers, yes: one card, llama.cpp b12606, the files and settings in "What we ran". Everything else on this page is credited to the person who ran it.