Thinking Machines heard the complaint. Inkling 2 Small was a 594B model, and on 20 November 2026 the lab shipped the answer to everyone who said it was too big: Inkling 2 Mini, a 26.4B mixture-of-experts model with 3.4B active parameters, open weights under Apache 2.0, small enough that the lab's own demo ran it offline on a MacBook Pro M5 Pro with 48 GB.
The scores are what made it spread. By the lab's figures it leads every open-weights model under 100B total parameters on DeepSWE v1.1 at 48.2% and on Vals Index v2.2 at 46.5%. Unsloth had GGUF files up on day one and llama.cpp added the architecture in b12600.
The model it has to beat on a home GPU is Qwen3.8-27B, the dense 27B most people with a 24 GB card already run. Here is Thinking Machines' own comparison from the model card:
| Benchmark | Inkling 2 Mini | Qwen3.8-27B |
|---|---|---|
| Parameters, active / total | 5B / 26B | 27B / 27B |
| AA Index v5.0 | 20.1 | 20.1 |
| DeepSWE v1.1 | 48.2% | 42.2% |
| SWEBench Pro v2 (Hard) | 77.9% | 73.8% |
| SWEBench Verified | 79.2% | 77.8% |
| Terminal Bench v2.1 | 71.6% | 73.0% |
| Vibe Code Bench v1.1 | 77.9% | 64.9% |
| BrowseComp | 79.0% | 61.6% |
| MCP Atlas | 82.7% | 77.3% |
| LongBench v2 | 66.9% | 52.3% |
| MMMU Pro | 74.9% | 76.3% |
| GDPVal-AA v2.1 | 1319 | 1427 |
These are the lab's figures, and the lab notes that evaluation setups vary by model. By its own summary, Inkling 2 Mini is ahead on agentic coding and behind only Qwen3.8-27B on the rest. The memory angle is the part that matters here: Qwen3.8-27B reads all 27B weights for every token, Inkling 2 Mini reads 5.4B. That is why it decodes fast. Whether it fits is the question below.
So we measured Inkling 2 Mini VRAM on an RTX 3090 the next day. The file sizes say a 24 GB card holds the 4-bit model with room to spare. On llama.cpp's default settings, every 128K setup we tried failed to load, down to the 2-bit file. The reason is a table no file size shows, and one flag decides how big it gets.
A 24-layer model that runs like 48
Inkling 2 Mini stores 24 layers in four blocks of six. The first and last blocks run once; the two middle blocks run three times for every token. The lab calls it "48 layer applications per token from 24 stored layers". The weights are stored once, so the file stays small. That is why a model with these scores comes in at about 20 GB.
Two numbers follow from the loop:
- Active parameters. The lab gives 3.4B active and 5.4B "weight activations per token to account for looping". Both are right: 3.4B distinct weights a token, read 5.4B times' worth once the middle runs three times. Decode speed follows the 5.4B.
- The cache. In llama.cpp (b12600), each pass of the loop keeps its own KV cache, so the 24 stored layers become 48 cache slots: 8 that see the whole context and 40 that keep only a 1,024-token window. With a 16-bit cache that is 1.19 GiB at 32K, 2.22 GiB at 64K, 4.28 GiB at 128K and 8.41 GiB at 256K.
That much a spec sheet can tell you. What it cannot is the next part.
What we ran
One RTX 3090 24GB (24,576 MiB, 21 MiB in use at idle), Linux, llama.cpp b12606 (commit e412c6f) from the project's own CUDA 12.8 build, flash attention on, one slot, automatic fitting switched off so llama.cpp could not quietly shrink anything, 256-token answers, greedy. The files were Bartowski's Q2_K, Q3_K_M, Q4_K_M and Q5_K_M and Unsloth's UD-Q4_K_XL, each checked byte for byte against the published size before use. Each setup read one prompt filled to about 85% of its window while we sampled VRAM every half second. Speeds are llama.cpp's own timings.
Before the run we wrote down what our calculator predicted for each setup, so the Inkling 2 Mini VRAM figures here are tested against a prediction made in advance. Then we let the card answer.
The table nobody priced in
Q2_K at 128K, the smallest file we had, refused to start. llama.cpp asked for a 13,484,916,992-byte compute buffer and ran out of memory. At 64K, Q4_K_M and Q5_K_M asked for 6,841,139,456 bytes and failed the same way. Of 12 setups at llama.cpp's default batch, 3 loaded and 9 did not. Every 128K and 256K setup failed, and every 4-bit or larger file at 64K.
The cause is in llama.cpp's own source for this model. Inkling 2 adds a learned bias to attention based on how far apart two tokens are. llama.cpp builds that bias as one full 32-bit table: attention heads, times every token in the context, times every token in the batch being processed, and the graph holds two copies. With Inkling 2 Mini's 24 heads and llama.cpp's default batch of 512 tokens, that is 2 x 24 x 131,072 x 512 x 4 bytes, 12.9 GB at 128K, on top of the weights and the cache. Flash attention does not avoid it; the table is built before attention runs.
The same code serves the bigger Inkling 2 Small, with twice the heads.
One flag takes it back
The table scales with the batch, and -ub sets the batch. Cut it, and the table shrinks in proportion. On the same card:
| Setup | Default batch (512) | Smaller batch | Peak VRAM |
|---|---|---|---|
| Q3_K_M, 32K, 16-bit cache | 18.01 GiB | -ub 128 |
15.61 GiB |
| Q4_K_M, 64K, 16-bit cache | did not load | -ub 128 |
21.21 GiB |
| Q3_K_M, 128K, q8_0 cache | did not load | -ub 128 |
18.98 GiB |
| Q2_K, 128K, q8_0 cache | did not load | -ub 128 |
16.48 GiB |
| Q4_K_M, 128K, q8_0 cache | did not load | -ub 64 |
21.23 GiB |
| Q4_K_M, 128K, 16-bit cache | did not load | -ub 64 |
23.18 GiB |
With -ub 64, the 4-bit file runs at 128K on a 24 GB card. The price is reading speed. The batch is how many prompt tokens llama.cpp processes at once, and a small one reads a long prompt slowly.
The speed you pay
| Setup | Prompt | Prefill | Decode |
|---|---|---|---|
| Q3_K_M, 32K, default batch | 32K | 667.4 tok/s | 51.32 tok/s |
| Q4_K_M, 32K, default batch | 32K | 655.1 tok/s | 48.4 tok/s |
| Q3_K_M, 64K, q8_0, default batch | 64K | 460.3 tok/s | 44.8 tok/s |
Q4_K_M, 64K, -ub 128 |
64K | 413.9 tok/s | 43.77 tok/s |
Q4_K_M, 128K, -ub 64 |
128K | 255.6 tok/s | 35.63 tok/s |
Q2_K, 128K, q8_0, -ub 128 |
128K | 268.3 tok/s | 36.3 tok/s |
The 32K and 64K rows are the median of three runs; the 128K rows are one run each. Decode holds up well: 48.4 to 51.32 tok/s at 32K and still 35.63 at 128K. Prefill is the cost. A full 128K prompt at 255.6 tok/s is about 8 minutes before the first token. For a coding agent that rereads a long repository on every turn, that matters more than decode.
Inkling 2 Mini VRAM by card: what our calculator says now
The calculator had the cache right and missed the table. It now charges it on llama.cpp: 3.00 GiB at 32K, 12.00 GiB at 128K, 24.00 GiB at 262K, at the default batch. After that change it lands within -1.7% to +0.1% of every peak we measured, and it gives the card's own verdict on all 12 default-batch setups.
What fits at llama.cpp's defaults:
| Card | What fits |
|---|---|
| 16 GB (RTX 5060 Ti 16GB) | Q2_K at 32K only, tight: 14.98 GiB with a q8_0 cache |
| 24 GB (RTX 3090 / 4090) | Q4_K_M at 32K (21.80 GiB), Q3_K_M at 64K (22.07 GiB); nothing at 128K |
| 32 GB (RTX 5090) | Q8_0 at 32K (30.28 GiB), Q4_K_M at 64K, Q3_K_M at 128K with a q8_0 cache (28.24 GiB) |
Lower -ub and every row above moves up a step, at the prefill cost in the table before. The calculator answers the default because that is what a fresh install runs.
Does it run on 16 GB?
Unsloth's launch post says it "runs on 16GB RAM". Read their own table: it is labelled RAM plus VRAM, or unified memory, and it puts 4-bit at 17-18 GB. A 16 GB graphics card holds 14.9 GiB, and every 4-bit file is bigger than that before the cache and the table.
So on a 16 GB card, Inkling 2 Mini VRAM use rules out everything but 2-bit at short context on the card itself, or with experts spilled to system RAM. @niklaslenz_ai did the second on an RTX 5060 Ti 16 GB with ggml-org's IQ4_NL file at a 32K window: the model "doesn't fully fit, so llama.cpp puts some experts on the CPU", and decode came in at 43.1 tok/s on code and 43.2 on prose, with 267 tok/s of prefill at 8K. That is usable, and it is the price of a file that does not fit.
Where the big numbers come from
The fastest results so far are not on llama.cpp. @niklaslenz_ai ran an RTX 5090 32 GB in vLLM with the official Inkling 2 Mini image and DSpark drafts at a 93K window: 416.6 tok/s decoding code, 319.3 on prose, 17,536 tok/s of prefill at 8K and 954.7 tok/s across 8 requests, the median of 2 runs per prompt with 512 tokens.
@Tech2Wild pushed one DGX Spark to the full 1M context in vLLM with the official NVFP4 weights, DSpark and an FP8 cache: a 3.3M-token cache pool, a 950K-token needle test passed, about 47 tok/s for one user, 47.3 in his eval run, about 1,690 tok/s of prefill over 200K and 18.4 tok/s decode at 950K. The vLLM recipe Thinking Machines points to caps a Spark at 262,144 tokens; his build is an image from 20 November, version 0.1.dev25855, a pre-release of 0.35.
If you have the memory and want long context today, vLLM is where this model runs fastest. If you are on llama.cpp, set -ub.
What to run, by machine
- 16 GB card: Q2_K at 32K on the card, or a 4-bit file with experts in system RAM at about 43.1 tok/s.
- 24 GB card: Q4_K_M at 32K on defaults; Q4_K_M at 128K with
-ub 64if you can wait for prefill. - 32 GB card: Q8_0 at 32K, or Q3_K_M at 128K with a q8_0 cache.
- Mac with unified memory: the lab's demo is a 48 GB MacBook Pro. Check yours on the Mac checker.
- DGX Spark: the NVFP4 checkpoint in vLLM, up to the full 1M.
Every one of these is a live answer in the calculator: pick Inkling 2 Mini, your card and your context. What LLM can I run lists everything that fits a card, and the speed tool estimates decode from the 5.4B the loop actually reads.
The files
| File | Who | Size |
|---|---|---|
| NVFP4 (the lab's own, Blackwell) | Thinking Machines | 19.53 GB, plus a 0.64 GB MTP head |
| BF16 | Bartowski | 52.23 GB |
| Q8_0 | Bartowski | 27.82 GB |
| Q6_K | Bartowski | 24.70 GB |
| Q5_K_M | Bartowski | 21.22 GB |
| Q4_K_M | Bartowski | 18.66 GB |
| UD-Q4_K_XL | Unsloth | 17.63 GB |
| IQ4_NL | ggml-org | 17.63 GB |
| Q3_K_M | Bartowski | 14.55 GB |
| UD-Q3_K_XL | Unsloth | 13.63 GB |
| Q2_K | Bartowski | 11.88 GB |
| UD-Q2_K_XL | Unsloth | 9.14 GB |
Vision needs a separate 0.64 GB projector file. If you only chat with text, leave it out.
Sources
- Thinking Machines, Introducing Inkling 2 Mini, 20 November 2026
- Model card, config and evaluation table
- llama.cpp b12600, "model : Inkling 2 Mini (#32694)", and the bias code in src/models/inkling2.cpp at b12606
- GGUF files: Bartowski, Unsloth and ggml-org, sizes read 21 November
- @UnslothAI launch post and hardware table
- @niklaslenz_ai, RTX 5090 and RTX 5060 Ti runs, 21 November
- @Tech2Wild, one DGX Spark at 1M, and the recipe
The RTX 3090 figures are our own run, described above. The RTX 5090, RTX 5060 Ti and DGX Spark figures are their testers' own, credited by name.
FAQ
How much VRAM does Inkling 2 Mini need?
Inkling 2 Mini VRAM depends on the file, the context and llama.cpp's batch. On llama.cpp defaults, Q4_K_M at 32K with a 16-bit cache needs 21.80 GiB, which fits a 24 GB card. Longer context needs a smaller -ub: with -ub 64 we measured Q4_K_M at 128K peaking at 23.18 GiB on an RTX 3090.
Why does llama.cpp run out of memory at long context?
It builds Inkling 2's position bias as a full table of heads x context x batch, held twice. At 128K and the default batch of 512 that is about 12.9 GB on top of the weights and cache. Lowering -ub shrinks it in proportion.
Does Inkling 2 Mini run on a 16 GB GPU?
On llama.cpp, Inkling 2 Mini VRAM use fits a 16 GB card only at 2-bit and short context on llama.cpp defaults: Q2_K at 32K is a tight fit. Every 4-bit file is larger than the 14.9 GiB a 16 GB card holds, so 4-bit means experts in system RAM.
How fast is it on an RTX 3090?
In our run, 48.4 tok/s decode and 655.1 tok/s prefill for Q4_K_M at 32K, and 35.63 tok/s decode with 255.6 tok/s prefill at 128K with -ub 64.
Is it 3.4B or 5.4B active?
Both, by the lab's own count: 3.4B distinct weights per token, 5.4B weight activations once the middle blocks run three times. Speed follows the 5.4B.
Did we run this?
The RTX 3090 numbers, yes: one card, llama.cpp b12606, the files and settings in "What we ran". Everything else on this page is credited to the person who ran it.