Strata vs Basalt: A 54.5 GB Model on a 24 GB Laptop

The model does not fit, and that is the test

Qwen3.8-Flash-Next at IQ3_S is a 54.5 GB file. An RTX 5090 Laptop has 24 GB of VRAM. No engine can hold the weights on the card, so the question is not which engine is fastest in the ordinary sense. It is which engine makes a model that lives in system RAM and streams its experts over PCIe feel like a local model rather than a batch job.

@uctive ran that test on one laptop: a 24 GB RTX 5090 Laptop, 64 GB of host RAM, an AMD Zen server CPU with 15 pool workers, PCIe 5.0 x8, Linux and CUDA 13.0. He put the same 54.5 GB IQ3_S file in front of Strata 0.1.62 and Basalt 0.4.1: the same prompts, the same temperature, the same streaming client. His full report is in ViktorIXAI/strata-vs-llama-load.

The first llama.cpp arm (b12531, --n-cpu-moe 48) was excluded by @uctive himself as misconfigured. He re-ran it a day later with llama.cpp b12533 and its MoE expert cache tuned, against the same Strata build; that re-test has its own section below.

The short version: both engines run it, and they win in different places. Strata wins cold long prompts, concurrency, energy per token and cache transparency. Basalt wins single-stream decode and keeps a conversation cache on disk across restarts. Neither wins everything. Tuned llama.cpp gains 37% to 49% from its expert cache and still trails both on cold prompts by an order of magnitude.

What was run

Strata 0.1.62 Basalt 0.4.1
Context 196,608 196,608
KV cache int8 int8
Expert handling 16,888 MiB cache, 8,697 of 24,576 slots, pcie_frac=0.10 resident experts, 46 GiB budget
Drafting MTP 4 plus lookup 3 spec 8
Conversation cache 2,560 MiB, four slots 6,144 MiB RAM, 20,480 MiB disk

The file is the same. The engines use their own pack conversions and their own tokenizers, so the token counts on identical text differ slightly. That is one of the reasons this is a comparison of complete configurations, not a controlled test of one kernel.

Cold prefill: Strata holds a flat line

Strata's cold prefill is the strongest single number in the run. It reads prompts at about 3.2 to 3.3 k tok/s from 39K to 189K tokens, with no falloff:

Prompt Strata time to first token Strata prefill
5,250 tokens 4.80 s 1,094 tok/s
39,280 13.71 s 2,866
75,023 23.74 s 3,160
152,642 47.65 s 3,204
188,724 58.31 s 3,236

Basalt's cold prefill in the same run measured 50,177 tokens in 16.53 s, about 3.0 k tok/s, and a live 11,496-token prompt in 4.66 s, about 2.5 k tok/s. That is close to Strata at the short end, and a little behind at the long end.

Read Basalt's ladder with care. Its p32k, p64k and p128k arms show a time to first token of about 5 s, which looks like a 7.9k to 30.7k tok/s prefill. It is not. The engine's own accounting says 99.99% of those prompt tokens were reused from its persistent 20 GiB disk conversation cache, which had seen the same seeded prompts in earlier sessions. Those rows are warm-cache reloads, not cold reads. The report labels them that way, and so should anyone quoting them.

Single-stream decode: Basalt is faster

Basalt's engine-reported decode is 139 to 172 tok/s. Strata's is 94 to 156. That is a real win for Basalt on one request at a time, and it is the number to use if your workload is one person typing into one chat.

The client-side number is higher, and not directly comparable. The client measures generated tokens over wall time, and on both purpose-built engines it runs 1.6 to 2.3 times the engine's own rate, because the engine counts a drafted token differently and because thinking tokens are counted differently. The report says to treat the client figure as an upper bound and the engine figure as the conservative one. That is the right reading.

Warm turns: Strata restores faster

A 50K-token conversation, two unrelated chats in between, then a follow-up:

Engine Cold first token Follow-up first token Speed-up
Basalt 16.53 s 4.39 s 3.8x
Strata 15.72 s 1.43 s 11.0x

Strata parks and restores conversations across four slots, which is why it keeps the 11x while other agents are running. Basalt's 3.8x comes from a large snapshot reloaded from RAM or disk; in exchange, that snapshot survives a server restart.

Concurrency: Strata wins at eight, but read the control

At eight concurrent 32K prompts, Strata aggregates 126.5 tok/s against Basalt's 101.6. At four they are level: 51.2 against 51.0.

The honest lower bound is the control arm. Four unique 32K prompts at c=4 aggregate 19.0 tok/s, against 51.2 for the shared-prompt arm. The shared-prompt numbers include prompt reuse, and there is no prefill parallelism to find: four 39K cold prompts in 53.8 s is about 2.9 k tok/s, exactly one stream's prefill rate. If your agents send different prompts, the shared-prompt concurrency figure is not your number.

Strata's own counters show what it is doing: 2,560 MiB of conversation cache, four slots, and 51.3% of prompt tokens reused at c=8. Its per-agent /metrics rows expose reuse, decode rate, draft acceptance and PCIe share per request. Basalt exposes per-slot cache lines.

Energy: Strata uses less per token

Over 30 minutes at c=2, with 192-token answers:

Strata Basalt
Requests completed 1,285 829
GPU energy 82.35 Wh 76.64 Wh
Wh per 1,000 answer tokens 0.334 0.481
J per answer token 1.20 1.73

Strata draws more power on average, 164.4 W against 153.0 W, and still spends less energy per answer because it finishes more requests. Nothing thermally throttled on the laptop: the software and hardware thermal slowdown bits never fired, temperatures peaked at 74 to 77 C, and only the normal power cap appeared. That makes this a fair sustained comparison for this box, not a short burst.

The cache claim: 275x is real, and narrower than it sounds

The run tested the "500x cached prefill" claim directly, with unique salted prompts and a fresh serve.

  • An identical 32K prompt repeated immediately is 275x faster to first token.
  • The same repeat on a fresh serve is 24.9x.
  • Append one line to the prompt and reuse is partial: 3x.
  • Flip one character in the middle and the prefix breaks: back to a full re-prefill.
  • Re-query a conversation after four 39K intruder chats filled all four cache slots: 0.835 s instead of 11.9 s.

So the fast path is genuine, prefix-granular and survives eviction pressure. It is not a general 500x, and it disappears in a wave of unique prompts: the c=4 unique arm is 19.0 tok/s aggregate against 51.2 for shared prompts.

The part that matters most: at temperature 0, the cached path gave a different answer from the cold path. The cold answer was 974 characters and 229 generated tokens; the cached answer was 953 characters and 222 tokens, with the first divergence at character 412. The two did not just paraphrase. The cold run invented a CONTROL record from the prompt's salt; the cached run correctly said there were no CONTROL records in the dump. One trial, so it is an existence proof of divergence, not a rate, and here the cached answer was the right one. But it means a cached-prefill speed-up cannot be sold as answer-identical at temperature 0; @uctive marks it for repeated investigation.

The second laptop: a 4090 says the same thing

@blue0orbit ran a smaller version of the same question on an RTX 4090 Laptop, Windows, Qwen3.8-Flash-Next at IQ3_XXS. His Strata side is his daily setup with MTP window 4; his Basalt side is an experimental Ada port, not an official build, and the conversion is not a bit-identical copy of the GGUF. He says so.

At a 7,876-token prompt: Strata 1,926 tok/s prefill, 72.0 tok/s decode, 4.15 s first token; the Basalt port 1,979, 33.9, 4.05 s. At 32,103 tokens: Basalt reads the prompt faster, 3,022 against 2,181 tok/s, and reaches the first token in 10.70 s against 14.80 s; decode is still 35.4 against 70.3 for Strata.

The shape is the same as the 5090 run. Strata is the better daily engine on a non-Blackwell laptop. Basalt's official target is Blackwell, and this port is a data point, not a verdict on Basalt.

And llama.cpp, tuned

The re-test gave llama.cpp its best shot. @uctive moved to llama.cpp b12533, switched on its new MoE expert cache and tried nine configurations before picking one: an 11,968 MiB expert cache on the card, --moe-cache-cpu-share 0.98, a Q8_0 KV cache, 15 CPU threads, one 198,144-token slot, no speculative decoding. Strata stayed at engine 0.1.62.

Tuning paid. Prefill at 32K went from about 170 to 237 tok/s (+39%), generation at 32K from about 30 to 41 tok/s (+37%) and at 4K from about 29 to 43 (+49%). A same-version control credits the expert cache, not the newer binary.

54.5 GB IQ3_S, RTX 5090 Laptop Strata 0.1.62 llama.cpp b12533, tuned
Cold prefill, about 39K tokens 2,866 tok/s 237 tok/s
Cold prefill, about 70K to 75K tokens 3,160 tok/s 225 tok/s
First token, about 39K prompt 13.7 s 165.9 s
First token, 150K to 170K prompt 47.6 s 790.3 s
Eight agents, aggregate 126.5 tok/s 7.2 to 8.3 tok/s
30 minutes, successful requests 1,285 270
GPU energy per generated token 1.20 J 4.94 J

Read the table with its caveats. The long-prompt row compares prompts whose token counts differ. The eight-agent row is not cache-normalized: Strata's result benefits from prompt reuse. Energy is GPU-only. Both engines found 5/5 planted facts, and neither failed a request in 30 minutes.

Two places llama.cpp holds up. Single-stream decode is not a clean contest: Strata's 32K figure, about 249 tok/s at the client, includes speculative decoding, while llama.cpp's 41 tok/s is plain decode whose client and engine counts agree. And its prompt cache is fast: an identical 39K prompt came back in 0.168 s, and a warm follow-up took 1.53 s against Strata's 1.43 s. That cache does not survive a restart, and at 128K and above the tuned configuration generated more slowly than its old settings.

Basalt 0.5.0 shipped the same evening, after this test; its headline addition is a new model, Tencent's Hy4-flash. The Basalt figures here are 0.4.1's.

The verdict

There is no universal winner.

Choose Strata for long cold prompts, several agents at once, energy per token, and cache behaviour you can see per request. On this laptop it reads 189K tokens in 58.31 s, holds 126.5 tok/s at eight streams and spends 1.20 J an answer token.

Choose Basalt for one stream at a time and a conversation cache that survives a restart. Its 139 to 172 tok/s decode is the best single-request number here, and its 20 GiB disk cache is a real feature, not a benchmark artefact.

Keep llama.cpp for what it is good at here: a warm, repeated prompt. Tuned, it answers an identical 39K prompt in 0.168 s, but it reads a cold one at 237 tok/s, and eight agents get 7.2 to 8.3 tok/s between them.

Do not quote the 500x cache claim without its conditions. It is 275x on an identical prompt, 3x after one appended line, gone after one changed character, and it changed a temperature-0 answer in this run.

What this test does not settle

One box, one run, one model file. The engines use different tokenizers and thinking-token accounting, so the decode columns are not the same quantity. Energy is GPU-only; the CPU side of expert streaming is not instrumented. Basalt's ladder is warm, its cold prefill is one 50K prompt and one live 11,496-token probe, and @blue0orbit's Basalt side is an experimental port on a card Basalt does not target. The report lists every failed and missing arm verbatim, including a tokenizer spread at 184K it could not explain.

Those caveats do not overturn the shape. They are why it is worth trusting.

Sources

FAQ

Can a 24 GB card run a 54.5 GB model?

Yes, if the engine streams experts from system RAM over PCIe. Strata, Basalt and a tuned llama.cpp all ran it on this laptop. llama.cpp's expert cache made it usable, at 237 tok/s cold prefill against Strata's 2,866 at about 39K tokens.

Which is faster, Strata or Basalt?

It depends on the request. Basalt wins single-stream decode, 139 to 172 tok/s engine-reported against Strata's 94 to 156. Strata wins cold long-prompt prefill, eight-stream aggregate throughput and energy per answer token. The full report has both sets.

Is the 500x cached prefill claim real?

For an identical prompt it measured 275x on the same serve and 24.9x on a fresh one. It falls to 3x when one line is appended, to 1x when one character in the middle changes, and it disappears in a wave of unique prompts. At temperature 0 the cached path also gave a different answer in this run.

Does a cached prompt change the answer?

In this run, once, at temperature 0: 974 characters cold against 953 cached, 229 tokens against 222, first divergence at character 412. The cold answer invented a CONTROL record; the cached answer correctly said there was none. One trial, so treat it as a warning, not a rate.

What about an RTX 4090 Laptop?

@blue0orbit measured Strata at 72.0 tok/s decode against an experimental Basalt Ada port at 33.9 on a 7,876-token prompt, and 70.3 against 35.4 at 32,103 tokens. Basalt read the longer prompt faster, 3,022 against 2,181 tok/s. The port is not an official Basalt build.

Did we run this?

No. The 5090 numbers are @uctive's run, published with the raw rows and the failed arms in his repository. The 4090 numbers are @blue0orbit's. We read both at source and banked the 5090 rows, Strata, Basalt and tuned llama.cpp, in our private corpus.