Qwen 3.8 Flash Next RTX PRO 6000: Does the Fast One Hold Up?

Which engine should you run Qwen 3.8 Flash Next with on an RTX PRO 6000?

Qwen 3.8 Flash Next RTX PRO 6000 is a six-engine question: Strata writes one chat fastest and Mia's TensorFold recipe serves several chats fastest. We ran it six ways on one NVIDIA RTX PRO 6000 Blackwell Server Edition (97,887 MiB, ECC enabled, power limit 600 W, Linux, driver 580.82.07), every figure the median of three runs.

• One chat at a time: Strata v0.2.12. With its IQ3_S file it wrote 215.3 tok/s on prose and 257.4 on code, the fastest single stream we measured.
• Several chats or an agent with parallel requests: Mia's AI Lab TensorFold recipe. Three prose streams at once reached 376.4 tok/s in total, 1.47x Strata's 255.9 on its NVFP4 pack.
• Long documents: Strata again. At a 236,023-token prompt its decode held at 214.67 tok/s, where the TensorFold recipe dropped 14.0%.
• A file bigger than the card but inside your RAM: tandem.cpp over llama.cpp's own expert cache, though on a 96 GB card you would not choose that file.

The rest of this page shows the numbers behind each line and what they do not prove.

How did we run the test?

Six setups, one card, one protocol. The table names what each engine ran:

Decode used Mia's bench_streams prompts: one, two and three streams at once, 256 tokens, temperature 0.7, median of 3. "Total" is the sum of each stream's own rate, which is her script's definition. Prefill used random prose at seven sizes, three fresh texts per size, and the rate is prompt tokens minus cached tokens over time to first token, with no overhead correction; four of the six engines reported no cached tokens at all, so on most setups the subtraction is a no-op. Our own long test filled 90% of a 128K and a 262K window (118,048 and 236,023 tokens) and asked for 512-token answers; all 36 runs wrote 512 tokens.

These are not three copies of one file. FP4-Local, NVFP4 and the GGUF quants are different 4-bit and 3-bit files with different quality, and the quality figures we can put side by side are further down.

How fast is one chat?

Strata on its NVFP4 pack wrote one prose stream 1.22x as fast as the TensorFold recipe, and code 1.57x. Strata's own server log agrees with our client clock: 206.1, 212.0 and 222.2 tok/s at 128K against our median of 212.27. Our tokens-per-second checker sorts rates like these into its slow, ok and fast bands.

Some Strata prose streams returned fewer than the 256 tokens we asked for, even with min_tokens and ignore_eos sent, and those streams sit inside the decode medians above. Code streams and the other engines had none.

What happens with three requests at once?

The order flips. Strata, by default, does not use its MTP drafts when it batches several streams (its own log says so, and --batch-mtp turns them on), so its total rises slowly; our Strata engine page carries the batching and release context behind that default. The TensorFold recipe keeps scaling.

Three prose streams on the TensorFold recipe came out 1.47x Strata's total on the NVFP4 pack; on code, 357.1 against 258.0, 1.38x. Strata's faster IQ3_S file scales to 283.6 by default, and TensorFold still leads at three. Strata's own batch table for this card (--batch 4, IQ3_S) reaches 276.6 tok/s at three in its own test (different stories, 1,024 tokens each, mean of two rounds) and only with both of its opt-in switches on. We did not run those switches.

How long does a 262K document take?

Doubling the prompt from 118K to 236K moved decode by +1.1% on Strata's NVFP4 pack and -1.7% on IQ3_S. The TensorFold recipe lost 14.0%, llama.cpp 31.3% and tandem.cpp 38.0%. Strata on its NVFP4 pack streams most of its KV cache from RAM and still served 95.54 to 95.91% of KV block reads from VRAM at 262K.

Prompt reading on random prose ran from 5,885.7 tok/s at about 4K to 7,033.5 at about 228K on the TensorFold recipe, and from 7,356.0 to 7,875.3 on Strata's IQ3_S, its best curve.

Do these match Mia's published numbers?

Her prompt reading does. Against Mia's AI Lab figures at 600 W (her README, measured with sparkDash), our TensorFold setup reached 91.6% of her rate at about 32K, 93.6% at about 64K and 94.6% at about 128K. At about 4K it reached 63.8%: her harness subtracts a calibrated fixed overhead, about 70 ms implied by her own 4K pair, and ours does not. That closes under half of the 4K gap, and about a third of the rate gap; prompt content and setup are the larger part of what is left. Her 600 W table is one run per limit, and her prefill runs overlapped a 96 GiB download, so her own numbers may read a little low.

Her decode headline is a different measurement. Our one-stream prose figure is 61.5% of her 400 W README number, which she measured with sparkDash on a different prompt and reply length. Against her earlier 300 W table our one, two and three stream totals land at 97.3%, 95.5% and 92.6%. Her own rule applies: a decode figure from another harness compares only with the same prompts, sampling and thinking setting. Her card is not our Server Edition either.

What about vLLM?

Upstream vLLM nightly with NVIDIA's NVFP4 checkpoint and the n-gram table on disk was the slowest of the engines that keep their experts on the card: 66.6 tok/s for one prose stream and 206.3 tok/s of prompt reading at about 4K. Its long-document test read 13.8 times faster than its own random-prose test at a similar size (13,686.9 against 994.3 tok/s), so that figure reflects a repeated sentence, not a real document.

This says nothing about vLLM on this card in general. NVIDIA's FP4-Local card says upstream vLLM cannot read that checkpoint's scale format yet and points to its own containers, where it validates 850K tokens of FP8 KV cache on one RTX PRO 6000 with the n-gram table offloaded. We ran a different checkpoint, upstream, with the table on disk.

Is tandem.cpp worth it?

On the same UD-Q4_K_XL file, tandem.cpp beat llama.cpp's own expert cache on every test: 1.88 times on one prose stream, 3.28 times on three, 2.14 times on one code stream, and 9.13x prompt reading at about 4K. The first token of a 236,023-token prompt came after 246.5 s against 367.6 s. Treat those ratios as this machine's, not a promise about yours: the gain comes from the CPU and PCIe split, which differs between machines.

That file fits a 96 GB card only partly, which is the case tandem.cpp was built for, and its README warns that files much larger than RAM lose the gain. Its author, @wei_wang, reports 29.8 to 97.3 tok/s on an RTX 4090 against stock llama.cpp with an IQ3_S file, GSM8K, greedy, single user. His own tables give Strata's exact mode 103.6 on that box and 185.9 against tandem's 182.5 on the 6000 Ada, and his README says the target is one user at a time: concurrent requests work but gain less. On this card, the engines that keep their experts on the GPU were faster still.

IQ3_S or NVFP4: what does the smaller file cost?

On Strata, IQ3_S against the NVFP4 pack: one prose stream +6.1%, code +13.4%, prompt reading at 128K +22.3%, 17.9 GiB less VRAM and a 42.8 GB smaller download. What each build needs from a machine is the subject of our Flash Next requirements page, and the NVFP4 build article sizes NVIDIA's own 4-bit checkpoint against real boxes. The price is quality. Strata's own yardstick against Q8_0 puts IQ3_S at 87.69% same top token (KL 0.144), measured on an AMD box with experts on the CPU, and its NVFP4 pack at 92.09% (KL 0.070), measured on an RTX 3060 box, also with experts on the CPU. Neither is a measurement on this card.

What else does each engine need from the machine?

RAM in use when measuring started: Strata 91.3 GB on NVFP4 and 72.3 GB on IQ3_S, llama.cpp 85.7 GB, tandem.cpp 86.2 GB, the TensorFold recipe 13.2 GB and vLLM 11.1 GB. "In use" is total RAM minus what the kernel reported available when measuring started; cached pages count as available, so the engines that read from disk look lighter here than they run once warm. Strata ran the n-gram table from RAM, not locked (47.7 GiB for NVFP4, 26.8 GiB for IQ3_S), so a Strata machine with this card wants that RAM free.

With less VRAM the picture changes. In our earlier run Strata v0.2.11 on IQ3_S still read prompts at 6,851.3 tok/s with 64 GB of RAM and 8,342.9 with 128 GB, but wrote 70.79 and 84.41 tok/s at a near-128K prompt. That card is our 96 GB Server Edition ballasted to 16 GB free, not a 16 GB card: it keeps the big card's compute, so treat the speed as Strata-with-a-small-budget, and a real small card will be lower. The Strata calculator shows how your card and RAM split the file between GPU, RAM and SSD.

FAQ

Which engine is fastest for one user on an RTX PRO 6000? Strata v0.2.12: 215.3 tok/s for one prose stream on IQ3_S, 203.0 on its NVFP4 pack, median of three on the Server Edition.

Which is best for agents or several chats? Mia's AI Lab TensorFold recipe: 376.4 tok/s across three prose streams, 1.47x Strata's NVFP4 pack at its default settings.

Does Strata slow down at 262K? Not in our runs: 214.67 tok/s at a 236,023-token prompt against 212.27 at 118,048 on the NVFP4 pack.

Can vLLM run Qwen 3.8 Flash Next on one card? Upstream nightly ran NVIDIA's NVFP4 checkpoint with the n-gram table on disk, at 66.6 tok/s for one stream. NVIDIA's FP4-Local checkpoint needs NVIDIA's own containers.

Is tandem.cpp worth it on a 96 GB card? It is 1.88 times llama.cpp's expert cache on the same file, but on this card the engines that keep their experts on the GPU were faster.