TensorFold 1.2.7 Resumes a 34K Chat in a Third of a Second

What TensorFold is

TensorFold is an open-source inference server for Apple Silicon and NVIDIA GPUs that speeds up decoding with draft tokens, and promises that the drafts never change a single token of the reply. It comes from @ashxhart, it is licensed under Apache 2.0 since 0.6.0 (releases up to 0.5.0 stay MIT), and it serves OpenAI and Anthropic-compatible APIs on your own machine. Since 1.0.0 on 7 October it runs as a native Zig binary, with no Python or MLX installation needed to serve.

There is also a separate Mac desktop app, TensorFold for Mac 1.0.0, published on 2 November 2026. It bundles engine 1.2.0 and updates its engine inside the app; it is a product of its own, not an engine release, and has its own section below.

The idea is speculative decoding: a small, fast predictor proposes several tokens, and the model checks them together instead of writing one at a time. Most engines do this. What TensorFold adds is a rule its release notes check almost every time. A drafted token is accepted only when it equals the token the same engine would have produced serially, and each release checks that drafted replies are byte-identical to the same request sent with drafting off. The README puts it plainly: "A draft is accepted only when it equals the token the same engine would produce serially." The promise has a scope: it holds for the same engine, weights and settings, not between the Mac and NVIDIA engines or between two quantizations.

Up to 0.6.6 its README listed seven model families: Qwen 3.8 27B, Qwen 3.8 Flash Next, Nemotron 3.5 Lightning, GLM 5.3 Flash, Gemma 4 26B-A4B, DeepSeek V4 Flash and Ternary Bonsai 2 27B, plus Qwen3.6 35B-A3B on NVIDIA GPUs. 1.0 admits a shorter list, below, and keeps the Python 0.6.6 engine on its python-0.6 line for the rest. The checkpoints it names come mostly from Vontra, the Hugging Face organisation its author links from their GitHub profile, and from TensorFold's own organisation.

TensorFold 1.0: a native engine and a shorter list

1.0.0 rewrote the engine and the server in Zig. The binary reads checkpoints, tokenizes requests and runs Metal or CUDA kernels itself; release archives target macOS on Apple Silicon and Linux on the GB10, the chip in the DGX Spark, and a Linux x86_64 archive adds NVIDIA's Ampere cards since 1.0.3 and Blackwell cards (the RTX 50 series and RTX PRO 6000) since 1.0.6; since 1.0.12 its kernels also carry the A100's compute capability. With the rewrite, TensorFold only admits the model and hardware pairs it has qualified. The README's table of 25 October, written for 1.1.4, lists these (it also carries Step-5-Preview, Kolibri-1, Liquid AI's d1-3B and the Drama 3 speech model):

Qwen 3.8 27B joined the list in 1.0.3, on 9 October. With its DFlash2 drafter on an M5 Max, the release notes put 512-token replies at 64 to 77 tok/s on prose and 151 on code, against 32 to 33 for stock mlx_lm 0.31.3; without the drafter it is slower than mlx_lm, and its prompt reading trails mlx_lm for now (on an M3 Ultra, 252 tok/s at 8K against 441). Since 1.0.6 sampled requests draft too: on an M3 Ultra a sampled code reply decodes at 125 tok/s and a prose one at 51, where 1.0.5 decoded both at about 20. Several requests now share each decode round, and eight greedy streams reach 124 tok/s in total against 73 for one reply at a time on an M3 Ultra. At 1.0 Bonsai, Gemma 4, Qwen3.6 and DeepSeek V4 stayed on the python-0.6 line, "still under qualification for 1.0.x". Qwen3.6-35B-A3B and Gemma 4 on Metal moved to the native line in 1.1.0, and the 1.1.4 notes add Gemma 4 on CUDA. The 1.1.4 announcement on X says release binaries do not serve DeepSeek checkpoints yet. Everything this page measured before 7 October, the RTX 40 cards included, ran on that Python line.

1.0.8 to 1.0.11, 12 to 14 October. 1.0.8 serves GLM 5.3, the whole 744B model, on two Macs: on two M5 Ultras one stream decodes at 21 tok/s without drafts and 31 to 32 with them, reading prompts at about 200 tok/s, and it needs two 256 GB Macs. The 27B on a Mac keeps its weights in memory once, so at a 70,000-token context it was ready at 29.6 GiB against 47.2 on an M5 Max and 24.6 against 42.2 on an M3 Ultra. Nemotron conversations on CUDA resume 1.7 to 2.7 times sooner, and on two M5 Ultras GLM 5.3 Flash resumes 3,165 of 3,181 tokens after an unrelated chat, where 1.0.7 resumed none. Living Weights learns user facts better (ten facts told in turn bring back 18 of their 20 questions on an M5 Ultra against 12), but the notes admit that 100 facts taught in turn kept 69 and brought back 82 of their 200 questions. 1.0.9 adds --cortheon, web research inside the server, off by default; with it on, the search queries the model writes leave the machine. It also speeds Nemotron on CUDA: on a GB10 eight streams decode 15.1 to 16.0% faster (258.5 against 224.5 tok/s greedy), and cold prompts read 7.3%, 7.0% and 6.4% faster at 8k, 16k and 32k. On an M5 Ultra the 27B's greedy code decode went from 406.1 to 417.0 tok/s, and a Mac mini M6 with 32 GB ran it. 1.0.10 serves MLX's mixed-width oQ checkpoints on Macs: on the author's M3 Ultra the 27B's oQ2 decodes 2.0x (code) and 1.6x (chat) faster and reads prompts 2.4 to 3.3 times faster, and its follow-ups start in 0.14 to 0.98 s instead of 15 to 63. 1.0.11 is a security release for --cortheon alone: without the flag nothing changes. One gap runs through four of these notes. 1.0.9 and 1.0.10 say no Blackwell sm_120 card ran them, 1.0.11 says the same for sm_86 and sm_120, and the 1.0.12 notes say no A100, RTX 30 or Blackwell card ran its release build, so the RTX PRO 6000 claims below rest on runs outside that release gate.

RTX 30 cards, since 1.0.3. The new Linux x86_64 archive runs Nemotron 3.5 Lightning on Ampere cards with compute capability 8.6: the RTX 30 series, RTX A6000, A10 and A40. The A100 was not supported then; 1.0.12 adds its kernels, though FP8 and NVFP4 checkpoints need compute capability 9.0 or newer. On an RTX 3090, the notes put 256-token replies at 252 tok/s drafted on prose and 313 on code (194 without drafts), 1.44 to 1.69 times llama.cpp's faster mode on the same card, and prompts at about 4,800 tok/s at 2K and 4,200 at 16K. Several streams at once are slower in total than one for now. Since 1.0.5 Nemotron keeps each conversation's prompt state between requests on CUDA as on a Mac, so a later turn reads only its new tokens: on a GB10, a resumed turn of a 7,200-token chat reaches its first token in 0.19 to 0.21 seconds instead of 1.06 to 1.09, and on an M3 Ultra the second turn of a 19.8K-token chat reads its prompt in 0.16 seconds instead of 8.1.

RTX 50 and RTX PRO 6000, since 1.0.6. The Linux archive now carries kernels for compute capability 12.0, so Blackwell cards no longer stop at kernel load. On an RTX PRO 6000 the notes put Nemotron at 874 tok/s greedy on code and 510 to 607 on prose with drafts. NVIDIA's own NVFP4 checkpoint of Nemotron also serves on a GB10: prompts at 6,131 to 7,281 tok/s from 2K to 32K tokens, and 96 to 182 tok/s drafted.

Living Weights, experimental, since 1.0.3, rebuilt in 1.0.7. Started with --slide, a Nemotron server learns facts while it runs: give it a fact, a document or a URL, and any TensorFold server started from that folder answers with them after a restart. A fact is kept only if related questions it was never trained on keep their answers, and since 1.0.6 a lesson that misses its held-out questions or disturbs another answer is taken out whole. 1.0.3 wrote each fact into the model's output-projection weights; on its five-fact test document 5 of 11 recall questions were answered and 1 of 16 neighbouring questions leaked a fact. 1.0.7 instead gives every layer's shared expert a pocket of 64 extra neurons and teaches each fact into about ten of them, one in each of about ten layers, leaving the model's original weights as they were. It learns about four times faster (the rounds for "I like blue." take 52 seconds on an M3 Ultra, from 217), and /slide draws what it has learned in 3D. Learning runs on Macs and, since 1.0.6, on a GB10. Facts taught from one document can still answer each other's questions: on a five-fact handbook part, 1 of 5 on a Mac and 2 of 5 on a GB10, where facts taught one at a time showed no such leaks. The model's files change, so the notes say to point it at a copy, and since 1.0.4 it refuses a folder whose files are links shared with other copies. 1.0.5 lets it learn from Japanese and other text written without spaces. 1.0.12 gives it a way back: tensorfold slide rollback FOLDER puts the files back after checking their hashes, and a --slide server now refuses to start without an API key.

GLM 5.3 Flash on one Mac, since 1.0.5. 1.0.4 refused it on a single Mac: the 4-bit checkpoint with its MTP head, 180.6 GB, is over TensorFold's one-Mac limit of 70% of RAM. 1.0.5 leaves the head on disk when only the head is over, and its notes put the full model on one 256 GB M5 Ultra at 1,859 to 1,990 tok/s reading 2K to 64K-token prompts, 78 tok/s writing prose and 165 on code edits. The README's qualified table still lists two M5 Ultras. On two, prompts now read at 2,090 to 2,513 tok/s against 1,417 to 1,803 in 1.0.4, and code edits decode at 262 tok/s, from 239; drafted prose came out about 4% slower than in 1.0.4, 106 tok/s against 111, and 1.0.6 fixed it by timing its draft windows at load: 112.5 tok/s on a 256-token story.

1.0.2, the same day: GLM 5.3 Flash on two M5 Ultras now splits the attention heads, the dense layers and the vocabulary head between the Macs as well as the experts. Its release notes put one stream at 91 tok/s without drafts (83 before), 114 to 131 with drafts on prose and 138 to 148 on code, and eight requests at once at 199 tok/s in total against 100 in 1.0.1, each reply matching its solo output. A resent 7,300-token conversation resumes in 0.18 seconds, and --learn keeps shared prompt prefixes on disk so a restarted server resumes them too.

NVIDIA Inception. On 8 October @ashxhart announced that TensorFold has joined NVIDIA's startup programme, with early access to the next Nemotron, and that the team is building day-zero support for it on both MLX and CUDA.

Flash Next on DGX Sparks, from @MiaAI_lab's recipes. Not yet on 1.0.0's qualified list for CUDA, but published as TensorFold recipes: on one Spark, Int4 AutoRound on TensorFold's Zig engine, 64.4 tok/s on prose for one stream, 140 at four and 200 at eight, reading prompts at about 2,500 tok/s; and on two Sparks, in a recipe published the day before that does not name the engine (two-Spark Flash Next serving came in 0.6.4, on the Python line), 67.4 tok/s for one stream in NVFP4 and 89 in Int4, both with a 1M context.

1.1.0, 18 October. The native CUDA line widens into the models this site covers. Qwen3.5-2B and Qwen3.8-27B serve on CUDA through the native server, the 27B drafting with DFlash2; Qwen3.8 Flash Next joins on its MLX affine 4-bit checkpoint with the MTP head; and GLM-5.3-Flash runs on two CUDA ranks reading Mia-AiLab's EXL3 checkpoint, taking images and video through --vision on rank 0. Nemotron 3.5 Bolt (74B-A9B) serves on CUDA from NVIDIA's 24 GB checkpoint, and the notes' own run on an RTX PRO 6000 decodes 153 to 155 tok/s against 144 to 146 for the contributor's own build in #897, with the same tokens: the authors' receipts, not our test. Around the models, the fixes readers feel: Nemotron 3.5 Super VL's structured output and forced tool calls work now (its omni layout keeps the text vocabulary size in llm_config and the server was not reading it there, #951); streamed replies set TCP_NODELAY, so small writes leave as they are written instead of waiting behind Nagle's algorithm (#929); the Mac app reads checkpoint metadata up to 64 MiB where 4 MiB stopped it and marked GLM-5.3-Flash's 11.5 MiB shard index incomplete (#967), and it skips the ._ companion files macOS leaves on exFAT and FAT drives, which could stop a scan of an external drive (#970); Nemotron on Metal loads on M1 to M4 chips where macOS 26.3 ships no tensor-unit kernels those builds need (#955); the CUDA kernel contract runs on sm_80 through sm_89 (#954); and Windows on ARM64 builds the native CUDA engine with NVIDIA's RTX Spark N1X (sm_121) among its targets (#1010). One known issue travels with it, stated plainly in the notes: GLM-5.3-Flash's own NVFP4 checkpoint on two CUDA ranks was not run for this release (two GB10s leave no room for its streams beside the memory reserve), so the EXL3 checkpoint is the measured two-rank path. The repository stood at 1,372 stars and 262 forks on 18 October.

TensorFold 1.2.3: Less Memory a Rank, and PDFs as Input

TensorFold 1.2.3, published on 5 November 2026, is mostly memory and serving work, with a few new paths. It does not replace the 1.2.0 Mac decode record or the 1.2.1 and 1.2.2 speedups; it adds to them. Where a speedup was checked for equal output, the note says so; the opt-in switches change nothing unless set.

The headline path is Qwen3.8-Flash-Next NVFP4 on two GB10s at --tp 2. Plain decode is 21% to 23% faster, drafted decode 9% to 11% faster at one stream and 76% to 77% at two, and prompts about 6% faster. Each rank holds 2.62 GiB less of the model, and a stream's KV at 65,536 tokens takes 40% less. The release says replies are token-equal to one GPU.

• Flash Next on an M3 Ultra, --parallel above one: a 1,024-token reply beside a 64K prompt finishes in 8 s instead of 48 s. The prompt's first token comes at 54 s instead of 90 s, with the same tokens.
• Flash Next FP4 on a GB10: one-stream decode up to 11.5% faster (chat +8.7%, code +6.3% at temperature 0), with every reply token-equal to 1.2.2.
• Flash Next on a Strix Halo (gfx1151): the first request after start decodes at 35.8 tok/s instead of 24.4, and a cold 7K prompt starts in 28.9 s instead of 48.4 s.
• HIP on a gfx1151: Qwen3.8-27B at 16 streams goes from about 51 to 75 tok/s on code. These are the contributor's records; we have no AMD GPU.
• GLM-5.3-Flash on CUDA: TP=2 with --drafter gets a cold prompt's first token 6.1% sooner at 32K and 7.0% sooner at 64K, for 80 MiB more a rank. At TP=3 the first token comes about 5% sooner.
• Flash Next MLX 4-bit on a GB10, against 1.2.2: one stream is 3.5% to 6.6% faster and eight streams 1.1% to 3.8% faster, with prompts level and the same tokens. Decode after a long prompt is faster in 18 of 20 cells, by 2.3% to 14.8%; two cells are slower, 32K by 5.2% and 64K by 3.1%.
• Qwen3.8-27B on an M5 Ultra and on a GB10, and Nemotron on a GB10: level against 1.2.2, within 0.4% on the M5 Ultra and 0.7% on the GB10 across the cells read.

New paths. dots.mocr runs natively on HIP with its own template and tokenizer. Qwen3.5-MoE runs on Metal on the 27B's engine, including Ornith-1.5-122B on one M5 Ultra. Qwen3-Embedding-8B and Qwen3-Reranker-8B serve on HIP: 16 documents of about 260 tokens take 2.69 s instead of 3.22 s on a Radeon AI PRO R9700. LFM3-1.7B runs on Intel's Iris Xe. PDF documents are input with --vision --vision-pdf. Nanbeige4.3-33B-INT4 on Metal needs Metal 4 tensor units, so M1 to M4 Macs are refused at load.

Kimi K3 and K3.1 are groundwork, not a served model. The native server renders their chat, reasoning and tool calls, but they have run only on a synthetic four-layer graph, no real weights have run, the full 1.45 TB checkpoint does not fit one Mac through this path, and K3.1 is capped at 2,047 tokens until its sparse index is qualified.

Memory and what changes. --prompt-cache-gib now keeps its size on CUDA KV pages. GLM-5.3-Flash csf0_v1 checkpoints save about 13 GiB across a TP=2 pair. Qwen 3.5 frees 0.46 GiB on the 27B. Saved disk prompt states from earlier versions are not reused, so the first long prompt after the upgrade prefills once. Dory's RoPE values moved by up to 5 ulp to match transformers, so a reply can change where a token was a near-tie. The release also refuses more malformed checkpoints and a K3.1 request past 2,047 tokens. We did not run 1.2.3; every figure here is the release's own test.

TensorFold 1.2.7: A Long Chat Picks Up Where It Stopped

TensorFold 1.2.7, published on 11 November 2026, is mostly about resuming long conversations and serving them. The measured headline is a follow-up in a 34K chat on an M3 Ultra: about 0.3 s to its first token instead of 85 s, when the client sends earlier replies back without their reasoning (#2811).

• The resume itself: with --prompt-cache-gib 5, such a turn resumes about 34,100 tokens instead of 7,424, and replies are unchanged. A client that keeps the reasoning resumes as before.
• An independent check, on an M5 Ultra 256 GB: @niklaslenz_ai measured Qwen3.8-27B MLX 4-bit with DFlash2 in a 34K chat. A follow-up's first token went from 0.87 s to 0.22 s without reasoning history and from 0.51 s to 0.10 s with it; the first prompt was 20.5 to 21.8 s on both versions. Median of three follow-ups in one run per version and history type, fresh server, thinking on, temperature 0. Replies matched token for token. That is his run, not ours.
• Flash Next on Metal, M1 to M4: a decode window of five rows or more now groups its 4-bit experts, so rows that pick the same expert share a launch. A copy stretch decodes 7.7% faster on an M3 Ultra (256.6 to 276.3 tok/s), with tokens unchanged (#2784).
• The opt-in scale index: TF_FLASHNEXT_SCALE_INDEX=1 makes drafted decode 1.3% to 3.9% faster on an M3 Ultra (prose 137.3 to 139.1 tok/s, a copy-heavy edit 262.8 to 273.2) and "draft": false 0.8% to 2.2% faster, for 7.3 GiB more in use and about 4 s more load (#2760).
• Qwen3.8-27B on an M5 Ultra: sampled verify rounds take 1.6 ms less and sampled decode is 6% faster, with the same candidates and replies; greedy is unchanged (#2507). Five cold 24K to 64K prompts in a row with a 14 GiB prompt cache reach their first tokens 3.5% to 5.9% sooner in total (#2725).
• GLM-5.3-Flash on two GB10s at TP=2 with DFlash2: four-stream decode is 3.7% faster greedy and 3.5% faster at temperature 0.7 (#2793). With TF_GLM_REQUEST_STARTS=1, a chat of about 58.7K tokens resumes its later turns at 58,735 instead of 57,344 (#2404).
• Qwen3.8-27B on CUDA beside a DSpine drafter, on a GB10: the dense model copies from its context after 12 matching tokens instead of 20, so agent turns decode about 6% to 7% faster at one stream (#2703). On HIP, 8K prompts reach their first token about 5% sooner on Flash Next and 6% on Nemotron 3.5 Lightning (#2576). Flash Next at TP=2 on two Radeon AI PRO R9700s holds 0.74 GiB less a card (#2670).
• New paths: a Responses API WebSocket mode that client.responses.connect() uses unchanged; LightOnOCR-3 0.8B and 4B image chat on CUDA; prompt-cache and SSD-expert panels in the dashboard and /stats; a Qwen3.8-27B conformance profile for XPU on an Arc Pro B70.
• Fixes: zig build -Dgpu=<gpu> compiles again; a damaged JPEG is refused; a malformed GLM tool call stays as text; a failed round fails only its own streams; two failed disk restores in a row resume from RAM instead of token 0.

What changes for a deployment. Grouped Flash Next experts are the default on M1 to M4, and TF_FLASHNEXT_GROUPED_ROWS=0 turns them off. A long Qwen 3.5 CUDA prompt's own first token comes later with --decode-share. A checkpoint that sets no tie_word_embeddings loads untied. A prompt-cache reply's cached_tokens now reports the state it actually resumed from. We did not run 1.2.7.

TensorFold 1.2.6: Sixteen Agents, One Prompt Cache

TensorFold 1.2.6, published on 10 November 2026, makes the prompt cache hold up when many agents send follow-ups together. The measured headline is on an M3 Ultra: at 16 agents a follow-up's median first token drops from 546 s to 15.9 s, and follow-up decode rises from 5.9 to 88.5 tok/s. Every agent's replies equal the previous version's at every agent count (#2699).

• At eight agents on the M3 Ultra: 50.1 s to 1.42 s to first token, and 15.4 to 79.5 tok/s on the follow-up. On a GB10 the same cells go from 14.6 s to 7.7 s and from 47.7 to 93.9 tok/s; at sixteen agents, 105 s to 25.8 s and 33.6 to 100.8.
• At one and four agents, and on Qwen3.8-Flash-Next: level with the previous version. First-turn speed is unchanged.
• DSpine-b16 on CUDA, opt-in: Qwen3.8-27B drafts with Tencent's drafter when --drafter names it, and every drafted reply equals the same request with drafts off. Against DFlash2, agent-shaped requests decode 2.4% to 19.8% faster; chat and code are 10% to 19% slower, which is why it is off by default (#2633). This site covers the drafter in Qwen 3.8 27B Runs Three Ways.
• Flash Next on HIP, on a Radeon 8060S (gfx1151): cold prompts of 2,048 to 63,728 tokens finish 29% to 41% sooner, with replies unchanged (#1877). Qwen DeltaNet prompts take 4.6% to 6.0% less time on Qwen3.8-27B and 5.7% to 9.9% less on Qwen3.6-35B-A3B (#2582), and Qwen3.6-35B-A3B NVFP4 prompts on a GB10 take 9.9% to 14.1% less time from 2K to 64K (#2537).
• Loaders and decode: Spark X2.6 4B 8-bit cold loads on a GB10 go from 5.03 to 2.53 s, and Dory NVFP4 from 4.79 to 3.79 s (#2449). GLM-5.3-Flash at TP=2 on two GB10s fills 2K and 8K prompts 23% and 13% faster and decodes one-stream code 32% faster (#2257). Flash Next on an M3 Ultra decodes 2.4% to 6.0% faster (#2593).
• Qwen3.8-27B on Macs: sampled decode with DFlash2 is 4% to 8.8% faster at one stream on an M5 Ultra (#2644); on an M3 Ultra chat rises from 75.3 to 77.0 tok/s and code from 106.8 to 109.8 (#2646). A Metal prompt pass skips its last layer's attention and MLP where no row keeps logits, which is 1.6% faster at 4K and 2.3% at 16K on an M3 Pro (#2436).
• New paths: --unix-socket PATH, POST /v1/images/generations, reproducible tool-call ids, --instruct-sampling card, and tools/conformance.py.
• What changes for a deployment: persisted prompt states are keyed by the loaded run, so upgrading starts every kept state once cold. Spark 2.5 and Dory on CUDA use direct reads by default. Nemotron's saved Metal prompt states are not reused. DeepSeek-V4.1 prepared folders rebuild once, and at TP=2 on two GB10s the first boot took about 10.5 minutes to weights loaded. GLM's seven 0/1 switches accept exactly 0 or 1, and a blank setting reads as unset.

We did not run 1.2.6.

TensorFold 1.2.5: A Local Web App, and an 18% Faster Prompt

TensorFold 1.2.5, published on 8 November 2026, is a model and serving release with a local web app. The measured headline is Qwen3.8-27B MLX 8-bit on an M5 Ultra: cold prompts are 18% faster and decode 7.6% faster, with the same tokens and prompt state.

• Qwen3.8-27B MLX 8-bit (groups of 64) on an M5 Ultra: cold prompts 18% faster (649 to 768 tok/s at 2K), decode 7.6% faster, same tokens and prompt state (#2518).
• Flash Next on CUDA, on a GB10 (NVFP4): BF16 prompt projections on a 128 x 128 tile take 3.7% to 4.4% less time at 4K to 34K tokens; K32 fragments from the same K64 pack take another 0.6% to 0.8% off cold prompts, with the same bytes.
• HIP on gfx12 with --policy matrix=on, on a Radeon AI PRO R9700: reranking and embedding take 2.9% to 4.8% less time, same bytes.
• Qwen 3.5 and 3.6 on HIP, on gfx1151: sampled streams with an MTP head make 4% to 12% more tokens a second at temperature 1.0.
• Qwen3.8-27B on Metal, on an M3 Pro: prompt chunks of 64 rows or more are 2% to 3% faster, bit-equal.
• New paths: TensorFold Web (tensorfold web); LFM2.5-24B-A2B on one NVIDIA GPU from the MLX 4-bit export; Nemotron 3.5 Parse and LightOnOCR-3 on HIP; Clef's joint schema head on CUDA; a CPU projection and RMS norm in the kernel contract.
• Serving: an explicit thinking off the template cannot honour is a 400; a refused disk read tries one shorter kept state; Qwen 3.5 and 3.6 CUDA streams take less memory; a request's stats list its prompt pass widths; the PDF render worker no longer crashes on macOS.
• Opt-ins: Qwen CUDA history starts with drafts (TF_QWEN_REQUEST_STARTS=1), Gemma 4's 2,048-row solo chunk, CUDA prompt-cache keep-others, and Qwen3.8-27B Metal KV pages.

Level against 1.2.4: Flash Next MLX 4-bit on a GB10 is level, every prompt and decode cell from 2K to 64K within 0.8%, and every sweep within 1.1% apart from greedy chat at four streams 3.6% faster; Qwen3.8-27B on an M5 Ultra is within 0.4% in every sweep and 1.2% in every cell, on a GB10 all 56 cells within -1% to +2%, and on an M3 Ultra within 1.3% in every cell; Nemotron 3.5 Lightning on a GB10 is level, with decode after a 32K prompt 2.5% lower in both runs. We did not run 1.2.5.

TensorFold 1.2.4: A 319K Chat Back in Seconds, and Faster Flash Next

TensorFold 1.2.4, published on 7 November 2026, is a serving and memory release with a few new model paths. The headline is a saved conversation: on two GB10s at TP=2 with TF_DSV41_PREFILL=replay, a 319K-token DeepSeek-V4.1 conversation restores from NVMe in 1.2 to 1.4 s instead of 155 to 178 s, with replies byte-identical to a cold prefill.

• Flash Next on CUDA at --tp 2 with drafts: a chat with about 28,500 tokens of history replies in 1.5 s when resumed, against 21.2 s fresh.
• Flash Next on one CUDA GB10: prompts at --parallel 1 are 15% to 23% faster, eight streams decode 3% to 6% faster, and a new short prompt's first token comes 19% sooner.
• Flash Next on an M5 Ultra, --parallel 2 or more: two overlapping clients finish in 47 s instead of 127 s, and a 64K prompt beside a 1,024-token reply in 21 s instead of 36 to 38 s.
• Gemma 4 and Qwen 3.5 on CUDA: cold loads are 28% to 31% faster, for about 0.6 GB more host memory.
• A lone "draft": false Flash Next reply on an M3 Ultra at --parallel above one is 17% to 18% faster; HIP rerank is about 25% faster and one-stream HIP decode up to 2.7% faster.
• New paths: Gemma 4 E4B (NVIDIA Nemotron 3.5 Bolt E4B) on CUDA; Cloudflare's Clef-Flash joint schema decisions and BF16 Qwen3.5 backbone on HIP; LFM3-1.7B text on Metal; the HIP runtime on Windows; DeepSeek-V4 and V4.1 prompts rendered with their own encoder.
• Serving: an engine refusal reaches the client as a 400 before any header; Flash Next on Metal enforces structured output at --parallel 1; unsupported penalties are refused; a refused prompt-cache restore tries a shorter kept state; DeepSeek-V4.1 tool calls stream sooner; /v1/models lists what each model supports; a kept prompt state is reused only by the same build and checkpoint files.

Level against 1.2.3: Flash Next MLX 4-bit on a GB10 is level on prompts and one stream, with two to eight streams up to 5.1% faster; Qwen3.8-27B is level within 1.2% on an M5 Ultra and 1.0% on a GB10, and level on an M3 Ultra; Nemotron 3.5 Lightning on a GB10 is level, with decode after a 32K prompt 2.0% to 2.6% faster. We did not run 1.2.4.

TensorFold 1.2.2: Flash Next on Two GPUs, and a Smaller Server

TensorFold 1.2.2, published on 3 November 2026, adds new models on CUDA and HIP and cuts the loader's memory and time. It does not replace the 1.2.0 Mac decode record or the 1.2.1 speedups; it adds to them. Where a speedup was checked for equal output, the note says so; the opt-in switches change nothing unless set.

The headline path is NVIDIA's own Qwen3.8-Flash-Next NVFP4 checkpoint on CUDA at --tp 2, on two GPUs, one rank a host. The BF16 attention, DeltaNet and shared-expert projections, the n-gram table and the MTP head's E4M3 experts load as stored. Each rank owns half the routed experts whole, so replies are one GPU's bytes. On two GB10s, drafted replies equal "draft": false in 6 of 6, and concurrent replies equal their solo runs in 8 of 8 cells.

• Qwen3.8-27B EXL3 on a GB10: prompt prefill 38% to 42% faster from 2K to 64K (494 to 705 tok/s at 2K), decode 5% to 12% faster, same tokens.
• HIP decode on a gfx1151: 7.5% to 18.9% faster on Qwen3.6-35B-A3B and Qwen3.8-27B at 32K and 64K, identical tokens. On Ryzen AI Max, about 18% on Qwen3.6 chat and 5% on the 27B.
• Flash Next on an M5 Max with --ple-on-ssd: decode 36% to 39% faster (75.8 to 105.1 tok/s on code), and a cold 2K prompt in a third of the time, same replies.
• Flash Next on a GB10: a cold start drops from about 74 s to 48 s with the direct expert load; cold prompts run 4% to 5% faster.
• Flash Next on a gfx1151: first cold prompt 28.55 s instead of 48.44 s; page cache 31.5 to 2.8 GiB.
• GLM on three GB10s at TP=3: cold prompts of 32K to 167K finish 5% to 14% sooner, identical replies.

New models and backends. GLM-OCR runs natively on HIP, its CogViT image encoder and GLM decoder in Zig and HIP, checked against the model card's Transformers examples. Qwen3.5-9B runs on Metal on the 27B's engine. DeepSeek V4.1 on Metal has its kernels and decoder checked against float64 references, with serving to follow. Intel Gen12LP GPUs without DPAS serve Nemotron-H with a -Dxpu-dpas=false build. The CUDA server now serves --tp with --rank, --master and --master-port, and the ranks compare a cached digest of every checkpoint byte, so TENSORFOLD_CHECKPOINT_SHA256 is no longer required.

Memory. Loaders drop their checkpoint from the page cache once the weights are loaded, so unified memory does not hold them twice; files read at runtime, such as PLE rows and DeepSeek's Engram tables, are kept. On CUDA, a served Qwen3.8-27B on a GB10 frees about 15 GiB. On an M5 Ultra, a loaded 27B frees about 14 GiB and Nemotron 3.5 Lightning about 17 GiB.

First native Windows on RTX Spark. On 4 November @u1tra_instinct published a native Windows 11 Arm64 benchmark on an RTX Spark laptop, the first we have seen, with Cyber Frost 3.8 and the same clients and settings for all three arms. TensorFold native Windows reads code at 50.4 tok/s, against 43.8 for TensorFold under WSL and 38.6 for EXL3 under WSL; prose is 37.3, 35.7 and 27.8. The first token is 0.22 to 0.28 s on a short prompt and 10.9 s at 17.85K, against 10.2 s under WSL and 19.1 s for EXL3. Four requests at once total 70.0 tok/s, against 65.6 and 36.2. Native Windows is 0% to 15% faster on most lines and 5% to 7% slower on DevOps and the long prompt, which is inside the noise of three samples per class. It also reports 113 GB of GPU memory and leaves 19.0 GiB for requests, against 15.4 GiB under WSL. The figures are client-side end-to-end tok/s and include prefill, so they are not decode rows; this is a user run, not our test.

What changes for you. A malformed checkpoint header is refused instead of read; a GLM with a non-bf16 tensorfold_activation_dtype is refused at load; --tp is refused by name for families that open without ranks, and --idle-unload with it; a Qwen DFlash2 drafter that declares a full-attention layer is refused on CUDA, and --prompt-cache-disk cannot be combined with --learn-conversations or --learn-dir. The release also adds a zig build -Dgpu=<gpu> server for one GPU, and reads GGUF files with every length and offset checked as the first step toward native GGUF on Apple Silicon. TensorFold Starlight, the enterprise platform, now lives in its own repository.

TensorFold 1.2.1: New CUDA Models and a One-Second Resume

TensorFold 1.2.1, published on 1 November 2026, is a CUDA and Metal feature release. It does not replace the 1.2.0 Mac decode record; the 1.2.0 table below still stands. The new paths:

• MiniMax-M3.1-Flash on CUDA across two GB10s, reading Mia-AiLab's EXL3 pack as stored. It is tensor parallel over exactly two ranks; anything else is refused.
• Gemma 4 31B on a GB10 with Google's MTP assistant: 28.3 tok/s against 12.2 undrafted at one stream.
• Qwen3.6-35B-A3B NVFP4 on a GB10 with a DFlash2 drafter: 130 tok/s at one stream on a 4.2K-token prompt, against 99 with MTP.
• Qwen3.6-35B-A3B on one GB10 at 1M tokens: a 1,048,512-token prompt prefills in 26 minutes and finds a passphrase placed 5% in.
• Gemma 4 E4B on Intel's Arc Pro B70: prompts read 2.0 to 2.6 times faster, with the same logits.
• Nemotron 3.5 Lightning on a GB10: +13.3% at eight streams, from grouped 4-bit projections.
• GLM-5.3-Flash on GB10s at TP=3 with the 1M window: a 64K message sent again resumes in about 1.7 seconds instead of 36 seconds. It is opt-in.

The release also adds server load reporting at GET /v1/load, bulk fact deletion in Living Weights, and a fix for Flash Next's long-prompt slowdown on a GB10. Where a speedup was checked for equal output, the note says so; the opt-in switches change nothing unless set.

TensorFold 1.2.0: Qwen 3.8 27B up to 44% faster on Macs

TensorFold 1.2.0, published on 31 October 2026, makes Qwen3.8-27B faster on Macs, with every measured reply token-equal to 1.1.9. The release check uses Qwen3.8-27B at MLX 4-bit with the DFlash2 drafter. On an M3 Ultra, one stream rose from 80.0 to 91.6 tok/s (+14.4%), four streams from 127.0 to 150.3 (+18.3%), and sixteen streams from 145.2 to 209.4 (+44.2%); prefill rose 17%. On an M5 Ultra, one stream rose from 168.5 to 178.3 (+5.8%), four streams from 298.0 to 334.6 (+12.3%), and sixteen streams from 378.5 to 474.0 (+25.2%). On a GB10, none of the ten models the release read got slower.

The release also adds three paths this page did not carry. MiniMax-M3.1-Flash is a new family on one 256 GB M5 Ultra, with 44 to 58 tok/s on one stream with DFlash2 and the mix256k checkpoint. NVIDIA's NVFP4 Qwen3.6-35B-A3B serves on CUDA as downloaded: one GB10 decodes 66 tok/s without drafts and 112 to 125 with its MTP layer. Dense EXL3 Qwen3.8-27B now loads on CUDA and drafts with DFlash2: at 4.00 bpw on a GB10, 45.8 tok/s for chat and 71.9 for code against 15.3 and 15.5 without. Gemma 4 E4B runs on Intel's Arc Pro B70 through the XPU backend at 135.6 tok/s on one stream and 426 across four.

One change is deliberately not token-equal. Qwen-VL pictures on Qwen3.8-27B, Qwen3.6-35B-A3B and Qwen3.8 Flash Next are now prepared as transformers' Qwen2VLImageProcessorFast prepares them, so replies to pictures can differ slightly from 1.1.9. Text-only replies are unchanged. Two known issues travel with the release: a repeated 32,768-token prompt on DeepSeek-V4.1 at TP=2 can change its reply (#1395), and DeepSeek-V4.1's prompt cache resumes only at 2,048-token boundaries and ignores --prompt-cache-gib (#1625).

1.1.5 to 1.1.9: the releases between

TensorFold 1.1.4: Qwen 3.8 27B prompts up to five times faster on CUDA

TensorFold 1.1.4, published on 24 October, makes cold prompts for Qwen3.8-27B on CUDA up to five times faster, according to its release notes. The notes measure it on a GB10, the chip in the DGX Spark, against 1.1.3, as cold prefill in tok/s:

Decode gains about 2% at 1 to 8 streams. The new recipe changes prompt arithmetic, so replies can differ in bits from 1.1.3. The notes say its quality check matches the previous recipe at 27 of 40 exact cases, and that every stream still equals its run alone and a resumed prompt equals a fresh one. --prompt-precision int8 is a further opt-in, with bf16 the default. @ashxhart's post of 24 October carries the same figures. We did not run this release, and every figure here is the author's.

Gemma 4 on CUDA. The 12B and 31B now run natively on CUDA from MLX affine checkpoints, with their drafters. On a GB10 the 12B decodes 37 tok/s on one stream, its MTP assistant takes greedy chat from 30 to 67 tok/s (2.2x), and z-lab's DFlash drafter takes the 31B from 12.6 to 29.9 tok/s (2.4x). Image input is in, but its GB10 image cells had not run when the notes were written.

Flash Next with several users. On CUDA, 1.1.4 verifies several streams' drafted windows in one pass. The served sweep on a GB10 gains 35%, 75% and 116% at 2, 4 and 8 streams, with tokens unchanged.

Serving. --idle-unload SECONDS frees a model's memory after that many idle seconds and reloads it on the next request. A sticky CUDA fault now turns /health to 503 instead of leaving a dead engine that looks ready. Drama 3, the speech model, can prefill prompts in row chunks as an opt-in: first audio comes after 0.76 s instead of 18.2 s on a GB10.

The cost the notes state. On a GB10, Nemotron 3.5 Lightning's greedy chat at 4 streams is 3.8% slower (338.7 to 325.8 tok/s). On a Mac Studio M3 Ultra the notes put Qwen3.8-27B and Lightning prefill within 0.1% of 1.1.3 and every sweep cell within 1.6%.

1.1.1 to 1.1.3, 22 and 23 October. 1.1.1 adds native CUDA families, larger verification trees for Qwen on CUDA, and Flash Next's images, video and a 1,048,576-token window on one Mac, and lands DeepSeek-V4.1's CUDA pieces without a release binary that serves its checkpoints. 1.1.2 keeps request text from spelling control tokens, caps open HTTP connections at 128, refuses an API-key file that other users can read, and takes Flash Next's CUDA prompts through faster tiles. 1.1.3 lets Flash Next image chats resume from their cache, keeps rejected tool calls visible, and builds on Windows again.

Nemotron 3.5 Super VL on CUDA, from 1.0.12

1.0.12, published on 16 October, brings NVIDIA's Nemotron 3.5 Super VL to CUDA, on one GB10 or two, with image input and NVIDIA's own FP4 and FP8 math. On two GB10s the second rank holds only its share of the experts (31.5 GiB, against 40.1 GiB on the first), and all 17 replies to the eight release prompts matched one GB10's. The notes' mean speeds, greedy and served, are 38.2 tok/s on one GB10 with MTP drafts and 18.8 without, and 46.8 and 20.2 on two. On one GB10 the weights take 70.8 GiB and 73.2 GiB with the image tower. --precision checkpoint, the default on Blackwell, read a cold 8,192-token prompt in 4.41 s against 7.96 s for full (1.81x) and decodes a little slower. Known issues include that --slide is refused for Super VL and that its MTP head drafts an image's prompt from the placeholder ids, which costs acceptance but not exactness. The same notes report a build of this work giving the GB10's tokens in all 17 cells on an RTX PRO 6000, and the release archive decoding at 124.1 to 223.1 tok/s drafted and 62.2 plain on that card. The same notes elsewhere say no sm_120 card ran the release build: read that as the release gate, because this bullet itself reports the archive running on an RTX PRO 6000, and so does our run in the next section. Those figures are the eight short release prompts served greedy; our 96.3 is one 113,108-token prompt timed from the stream, so the two measure different things and sit side by side as receipts, not as a comparison. The README, though, lists Super VL on one GB10 or two, and the RTX PRO 6000 only for Lightning. That is the gap the next section tests.

1.0.12 also brought images to the 27B on a Mac with --vision, LFM2.5-24B-A2B on Macs at 4, 6 and 8 bits (194, 168 and 160 tok/s on an M3 Ultra), and Nemotron on Macs from oQ quants, whose greedy prose on an M3 Ultra decodes at 162 tok/s with oQ4 against 188 for the 4-bit checkpoint. --recall, off unless you ask for it, lets drafts copy from text the server has read before: repeated requests after a restart ran 2.4x faster with the 27B on an M3 Ultra and 3.0x with Nemotron on a GB10, with every copied token verified. Intel Arc and AMD GPUs get previews you build from source; the notes' Arc Pro B70 runs decode the 27B at 122.9 tok/s on code and 99.3 on chat. From 1.1.0, structured output and forced tool calls work with Super VL: the omni layout keeps its text vocabulary in llm_config, the server read the size from text_config and the top level, and refused those requests (reported in #951; the notes thank @swbiggart and @BobClawblaw).

Our test: Super VL on one RTX PRO 6000, TensorFold against llama.cpp

This test is history, not the current engine: it ran TensorFold 1.0.12, before the 1.1.4 and 1.2.0 releases above. We ran Nemotron 3.5 Super VL on one RTX PRO 6000 Blackwell with 96 GB, a card the README does not list for that model, with TensorFold 1.0.12 and with llama.cpp. The receipts: ECC on, driver 580.82.07; llama.cpp b11696, the prebuilt CUDA 12.8 release, every layer on the card (-ngl 999), flash attention on and one slot, loading ggml-org's NVFP4 GGUF (76,918,795,552 bytes); TensorFold 1.0.12 serving NVIDIA's own NVFP4 checkpoint (75.06 GB), 72.01 GiB of model in VRAM at the 128K window. Both engines got the identical request: the same document (113,108 tokens at the 128K window, 228,708 at 262K), the chat template with thinking off, a request for about 400 words, at most 512 tokens, temperature 0. Every answer stopped on its own at 333 to 434 tokens, and TensorFold ran with drafts on and with drafts off.

What it does better. TensorFold reads the prompt 1.7 times as fast at 128K and 1.5 times at 262K. Its drafts change no token: the answers were byte-identical with drafts on and off, and 226 of 490 drafts were accepted at 128K and 282 of 518 at 262K. With drafts on, TensorFold decodes 1.32 times as fast as llama.cpp at 128K and 1.24 times at 262K. What it does worse. Without drafts, llama.cpp decodes 1.45 times as fast at 128K and 1.58 times at 262K, so the speed-up over llama.cpp is the drafts', not the engine's. Memory is a tie: with drafts on the peaks sit within 0.2 GiB of each other, and the drafter costs TensorFold about 1.9 GiB.

Two cautions. TensorFold's rates are timed from the stream, prompt over time to first token and then tokens over the rest, so they include streaming overhead, while llama.cpp's are its own server timings. And the prompt is one sentence repeated, which is easy to draft; real text may accept fewer drafts. It is one request per cell on one card, so read it as a first receipt, not a benchmark suite.

From 0.2 to 1.0 in twelve days, and on

The repository dates from June, but the line people run now began with a rewrite on 25 September and reached 1.0 on 7 October. What followed, from the project's releases:

From 0.3.5 on, the notes thank contributors by handle; 0.6.0 alone lists 16 contributors. On 28 September the repository had 538 stars and 54 forks, on 2 October 895 and 133, on 8 October 1,113 and 191, on 10 October 1,215 and 213, and on 16 October 1,346 and 253, and on 18 October 1,372 and 262. On 25 October the repository page showed 1,474 stars.

The Python line, 0.4 to 0.6.6: new cards, long context, many users at once

These results come from the Python engine, which 1.0 keeps on the python-0.6 line for the models it has not yet qualified.

RTX 40 cards. 0.6.0 runs the CUDA engine on compute capability 8.9. On one RTX 4090, its notes put Qwen 3.8 27B with the DFlash 2 drafter at a 40,182-token window, exact, reading prompts at 2,400 to 2,600 tok/s from 2K to 32K tokens and writing replies at 64 to 86 tok/s. NVFP4 and FP8 checkpoints need compute capability 9.0, so they stay on Hopper, Blackwell and the DGX Spark. RTX 30 cards, the 3090 included, are refused at startup for now; its author says 3090 support is planned.

Long context on a Spark. 0.5.0 rewrote how the 27B's attention reads its cache on CUDA. On one DGX Spark, with MLX 4-bit weights and DFlash 2, decode at 128K went from 18.4 to 38.9 tok/s and at 96K from 23.6 to 45.9; a cold 128K prompt reads at 1,173 tok/s instead of 860, and a 255K one at 809 instead of 545. On the same NVFP4 weights, its prompt reading runs 1.16 to 1.27 times vLLM's at every depth from 32K to 255K.

Waiting less. In 0.6.0 a queued prompt fills inside the rounds that decode the live replies instead of stopping them. On one Spark, Flash Next's first tokens came 2.8 to 3.1 times sooner than on 0.5.0; on an M3 Ultra, short requests queued behind a long prompt got their first token in a median 9.9 seconds instead of 113. An identical resend now resumes from the kept prompt on Flash Next, Qwen3.6, Nemotron and GLM-5.3 on CUDA too: an 18.7K-token Flash Next resend went from 8.3 seconds to 0.08.

Many users at once on CUDA, fixed in 0.6.1. @WescheNex1q measured Qwen 3.8 27B on one DGX Spark, the same NVFP4 weights and DFlash 2 drafter on every engine. With one request on 0.6.0, TensorFold led at 83 tok/s against 60 for SGLang and 57 for vLLM. With 16 requests at once it served them one at a time: 62 tok/s in total and a first token after 117 seconds, where vLLM gave 253 tok/s and 0.6 s. They reran it on 0.6.1 with --parallel 16: 372 tok/s, a first token in 0.3 s, and a 128K prompt read at 1,515 tok/s instead of 882, with answers byte-identical alone and in the crowd. The 0.6.1 notes give the mechanism: prompts that arrive together now fill in one pass, which on a Spark took the slowest first token from 1.6 s to 0.15 s.

NVFP4 in its own math (0.6.1). NVIDIA's NVFP4 build of the 27B now runs the 4-bit arithmetic its checkpoint names on Blackwell cards, with --precision full for 16-bit arithmetic on the same weights. On an RTX PRO 6000 at its 250 W limit, the notes give one stream 1.4 to 2.0 times vLLM with MTP=3, cold prompts at 0.95 to 0.97 times vLLM, and top-1 agreement with a full-precision reference of 92.9% (vLLM 93.0%) in 4-bit math and 98.9% in full. 0.6.2 adds 1 to 4% at 4 and 8 streams; its author puts 8 streams of code at about 1,232 tok/s.

The numbers on a Mac

The benchmark behind TensorFold's speed tables uses two public prompts, a short code task and a chat question, 64-token replies, five seeds and the median decode rate. That is a narrow test, and the notes say which cells are code and which are chat because the two behave differently.

Qwen 3.8 27B on an M3 Ultra, 0.3.4, MLX 0.32.0, 64-token replies, thinking off:

On code that is 3.7 to 4 times serial on the same Mac; on chat, about 1.9 times. 0.3.5 then measured another 17 to 20% on the same M3 Ultra against 0.3.4.1. For the same model on other hardware and engines, see Qwen 3.8 27B speed.

Qwen 3.8 Flash Next on an M3 Ultra, 0.3.4.1, from the notes: 135.5, 111.1, 142.3 and 122.2 tok/s across the same four cells. GLM 5.3 Flash on an M3 Ultra, 0.3.6: 1.86 to 2.08 times mlx-vlm, and the notes say plainly that this model misses the project's own floor of three times.

Qwen 3.8 27B on an M5 Max, 0.3.4, with the DFlash 2 draft model: 153.2 and 154.2 tok/s on the code cells, 69.8 and 72.8 on chat. The first release, 0.2.0, had given 120 to 124 on short answers.

On an M5 Ultra, 0.6.0's notes take the 27B with DFlash 2 on an oQ4e checkpoint from 33 to 131 to 160 tok/s, and four streams from 315 to 320 up to 332 to 334. On 1 October @u1tra_instinct posted a fine-tune of the 27B, AEON Ultimate, in 4-bit MLX on 0.6.0 with DFlash 2 on an M5 Ultra: 102 tok/s on prose, 228 to 378 on structured output, and 1,300 to 1,500 tok/s reading the prompt. It is a fine-tune, not the base model, so it sits beside these figures rather than in them.

Three Mac engines, one drafter. On 1 October @WescheNex1q ran Qwen 3.8 27B in 4-bit on an M4 Max through all three Mac engines with the same DFlash 2 drafter, the same prompts and thinking off:

With drafting off, all three run 29 to 31 tok/s, so the race on a Mac is the drafter, not the engine. One setting matters: mlx-serve 26.10.1 picks its MTP drafter over DFlash 2 by default, which made this build about twice as slow on their M4 Max, so their figures use --no-mtp.

GLM 5.3 Flash on a Mac. @volatilemarkts' M3 Ultra went from 27 to 58 tok/s with TensorFold, the first token in 0.08 s. On an M5 Ultra at 32K, @cryptosibbers measured oMLX at 56 tok/s and TensorFold at 79, the same model and question.

Read Mac comparisons with the file in mind. @plotarmordev ran TensorFold against oMLX on an M5 Ultra with Qwen 3.8 Flash Next and points out that the two were not reading the same weights: TensorFold's 4-bit pack reads 4.26 GB per token, oMLX's 5-bit pack 5.34 GB, at a prose perplexity of 7.78 against 7.12. In their run TensorFold led on drafted generation by 15%, on memory (109 against 124 GB) and at eight streams (353 against 226 tok/s); oMLX read prompts 1.8 times faster and was 7% faster without drafting.

The numbers on a DGX Spark

The 1.1.4 prompt figures in the section above are a GB10 too; the runs below predate them and were made with earlier releases.

On NVIDIA hardware the comparison is against vLLM with multi-token prediction at three drafts, the baseline TensorFold's recipe uses. From TensorFold's Qwen 3.8 27B recipe, one DGX Spark, medians of 15 runs a cell:

The EXL3 pack is turboderp's, read through TensorFold's experimental EXL3 path. It wins on code and loses on chat, and the recipe gives the reason: the DFlash 2 draft model was trained on the unquantized model, so fewer of its drafts survive on the 3-bit pack in chat, 3.0 to 3.3 tokens a round against 4.2. To check what a card of your own should deliver, the tokens per second checker estimates it.

One independent run. On 26 September @WescheNex1q ran TensorFold on one DGX Spark against vLLM with MTP=3 on NVFP4, on three prompts of their own: 102.9 tok/s against 34.9 averaged over the three (75.7 to 126.9 against 32.7 to 37.0 per prompt), TensorFold on its 4-bit MLX weights, with byte-identical output to serial and a first token in 0.12 s. Their own caveat, in the post: the two runs were not weight-matched.

Flash Next on one Spark. @ViC305, who wrote the shared EXL3 module that 0.3.6 uses on NVIDIA GPUs, measured Qwen 3.8 Flash Next as a 3.05 bpw EXL3 pack on one DGX Spark against vLLM: 80.8 against 42.4 tok/s on sampled code, 77.7 against 40.9 on greedy code, and 69.2 against 37.6 on greedy chat. The 0.3.6.2 release that followed, with their cache work in it, adds int8 and int4 caches for Flash Next on CUDA; the release notes put them at about 1.7 and 2.6 times the default window in the same memory.

GLM 5.3 Flash on two Sparks. This is where most of the week's runs landed. Each tester used their own prompts, so read the rows across, not down:

@jayleaton's build kept identical output hashes in every round and used 2.1 GB less memory than their build of the day before. @WescheNex1q's resent 64K prompt returned its first token in 0.27 seconds. @MiaAI_lab's recipe defaults to a 1M context with a 2.7M-token cache pool and reads prompts at about 1,950 tok/s, and @ChrizBogota verified context up to 1M. 0.6.0 makes GLM-5.3's DFlash 2 drafter read only its sliding window: 8% faster at 52K and 3.8 GiB lighter on each Spark.

The full GLM-5.3 on four Sparks. @u1tra_instinct reports the 753B model, not the Flash, running on four DGX Sparks through TensorFold at 2.75 bits in EXL3 with DFlash 2: prose above 30.9 tok/s, code 40 at peak, a 1M context. The four-Spark path went to TensorFold as a pull request.

What it needs in memory

TensorFold's README has a table of Mac memory classes and what each can run. On 1 October one row was filled, 64 GB, from a contributor's run; every other cell still read "TBD", pending measured peak memory for each result. What the README does give is the rule: the process stays inside 70% of RAM, or the GPU's recommended working set if that is lower, and 3 GiB of that is reserved for everything outside the model. GLM 5.3 Flash takes 85% on a Mac with 256 GB or less, and TENSORFOLD_MEMORY_LIMIT_GB raises or lowers the budget for any model.

So the question can be sized from the files. The Qwen 3.8 27B checkpoint TensorFold names is 16.05 GB, and its DFlash 2 draft model 3.85 GB. Our engine puts the 27B's cache at 64 KiB per token of context at 16 bits, on its 16 full-attention layers, plus a fixed 0.15 GiB for the other 48. Against TensorFold's budget:

These are ceilings, and TensorFold's own admission sits below them: it also reserves the reply, a prompt workspace and room to keep the last prompt. On 32 GB its README is explicit: the 27B needs more than the default budget, with or without its draft model, so the 0.9 GiB left on paper does not start it. The one measured row shows where a 64 GB Mac lands: on an M5 Pro, @benwilson's run with 0.3.5.1 fitted 140,288 tokens for Qwen 3.8 27B with its DFlash 2 draft at a peak of 43.7 GiB, 152,576 tokens without the draft at 39.9 GiB, and Nemotron 3.5 Lightning's full 262,144 at 42.2 GiB. On Macs where macOS gives the GPU less than 70%, the ceiling is lower still. TensorFold reserves the reply length and a prompt workspace before it admits a request, and it prints the figure it settles on at startup. Our Mac checker sizes the same model under macOS's own GPU limit.

The larger families follow the same arithmetic. Flash Next's MLX checkpoint with its MTP head is 113.21 GB, so it sits fully in memory only from a 192 GB Mac up. TensorFold's 0.3.6 adds --ple-on-ssd, which reads Flash Next's n-gram tables from disk: on a 128 GB budget it peaked at 85.6 GiB and decoded at 0.91 to 1.03 times the 256 GB run. --ssd-experts goes further and streams routed experts from the SSD, fitting Flash Next in a 64 GB Mac and GLM 5.3 Flash in a 128 GB one, at 0.31 to 0.39 and 0.13 to 0.17 times their resident speed. Those are TensorFold's measurements on an M3 Ultra under emulated budgets.

How it compares with Inco Splash

Both engines arrived in the same fortnight and both bet on drafting for Macs. They make different bets. Inco Splash compiles kernels for each model's exact shapes and, a week after launch, added Unsloth GGUF files. TensorFold verifies drafts in batched lanes, guarantees output identical to serial decoding, and since 1.0 runs as a native binary. Its Python line also runs on NVIDIA GPUs from the RTX 40 series up, where the 27B, Flash Next and GLM 5.3 Flash have published DGX Spark numbers, and reads MLX and EXL3 checkpoints. Their published numbers come from different Macs and different prompts, so this page does not rank them.

TensorFold for Mac 1.0.0: a desktop app, not an engine release

TensorFold for Mac 1.0.0 is a separate product from the engine releases above. Published on 2 November 2026 under the app-v1.0.0 tag, it is the first stable desktop release for Apple silicon Macs on macOS 13.3 or later, signed with a Developer ID and notarized by Apple. The DMG is TensorFold-1.0.0-arm64.dmg.

The app bundles engine 1.2.0. Engine 1.2.1 is available separately through Models > Details > Engine updates; checking, downloading and activating are explicit actions, the app waits for local work to finish before switching, and you can return to the bundled engine without replacing the app or the model files. The desktop tag is separate from the engine tags, so a newer engine release does not by itself update the app.

What the app adds: an Ask TensorFold bar with explicit window selection, a capture preview and local text extraction where available; the main window closes into the menu bar; connection health and a unified inbox show account state and provider errors; Portuguese is in the interface languages. Google Workspace and Microsoft connections need your own credentials and consent.

The release notes report 931 passing app test files, 259 native-helper checks, isolated packaged startup on both build and verification Macs, and a real-model engine 1.2.0 to 1.2.1 upgrade and rollback; the Studio matched all 121 compiled application files against the pinned build. The DMG, automatic-update ZIP and engine archive ship with signed update metadata and SHA-256 checksums. Windows and Linux builds remain previews; Cortheon remains disabled; OS automation and messaging depend on the operating system and configured services.

How to run it

On a Mac, 1.0 installs as a native binary with Homebrew:

brew install ashhart/tensorfold/tensorfold
tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
tensorfold serve TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit

The server listens on 127.0.0.1:8080/v1.

On a DGX Spark, the release archive carries the Linux binary for the GB10, and the README's runbook covers the CUDA runtime files. Since 1.0.3 a Linux x86_64 archive covers the RTX 30 series for Nemotron, and since 1.0.6 the RTX 50 series and RTX PRO 6000; Super VL needs a GB10 by the README, and we ran it on an RTX PRO 6000 above. For the models 1.0 has not yet qualified, and for RTX 40 cards, the Python engine stays on the project's python-0.6 line.

The Mac desktop app is a separate install: download the 1.0.0 DMG for Apple silicon on macOS 13.3 or later. It bundles engine 1.2.0 and can update to 1.2.1 inside the app; the Homebrew path above remains the server.

FAQ

What is TensorFold?

An open-source inference server for Apple Silicon and NVIDIA GPUs, Apache 2.0 since 0.6.0, that uses draft tokens to decode faster and accepts a draft only when it equals the token serial decoding would produce, so replies stay byte-identical. Since 1.0 it runs as a native Zig binary.

How fast is TensorFold on a Mac?

TensorFold's 1.2.0 release check puts Qwen3.8-27B at MLX 4-bit with DFlash2 on an M3 Ultra at 91.6 tok/s on one stream and 209.4 at sixteen; on an M5 Ultra it is 178.3 and 474.0. Earlier notes put a short code prompt at 141.3 to 158.4 tok/s against 38.2 to 39.3 for MLX serial, and chat around 74. In every Mac result TensorFold publishes, code runs faster than chat.

Is TensorFold faster than vLLM on a DGX Spark?

On TensorFold's published benchmark, with turboderp's EXL3 3.00 bpw pack, Qwen 3.8 27B decodes code at 83.4 tok/s against 23.4 for vLLM with MTP=3, and chat at 44.2 against 25.4. @WescheNex1q measured 102.9 against 34.9 on their own prompts, and noted the runs were not weight-matched. With many users at once it trailed vLLM until 0.6.1 fixed it: on one Spark, 16 requests went from 62 to 372 tok/s.

Is there a TensorFold Mac app?

Yes. TensorFold for Mac 1.0.0 is a separate desktop app for Apple silicon on macOS 13.3 or later, published on 2 November 2026 under the app-v1.0.0 tag. It bundles engine 1.2.0 and can update its engine to 1.2.1 inside the app; Windows and Linux builds are still previews.

Does TensorFold run on an RTX 4090 or an RTX 3090?

On an RTX 4090, through the Python engine on the python-0.6 line: its 0.6.0 notes give Qwen 3.8 27B with DFlash 2 a 40,182-token window and 64 to 86 tok/s. Up to 1.0.12 the native engine was qualified on CUDA for Nemotron 3.5 Lightning only, on the GB10 and, since 1.0.3, on Ampere cards: on an RTX 3090 its notes put it at 252 tok/s on prose and 313 on code, 1.44 to 1.69 times llama.cpp's faster mode. RTX 40 cards are not on the 1.0.12 list; RTX 50 cards and the RTX PRO 6000 joined it in 1.0.6, for Nemotron. 1.1.0 brings Qwen 3.8 27B, Qwen3.5-2B and Flash Next to CUDA through the native server, checked by its authors on a GB10, and its CUDA kernel contract now runs on sm_80 through sm_89, the compute capabilities of RTX 30 and RTX 40 cards; we have not run it on either. The 1.1.4 Qwen 3.8 27B figures are GB10 figures.

Is TensorFold faster than llama.cpp on an RTX PRO 6000?

For reading prompts yes, for plain decoding no. In our test of Nemotron 3.5 Super VL on one RTX PRO 6000 Blackwell, TensorFold 1.0.12 read a 113,108-token prompt at 6,408 tok/s against llama.cpp b11696's 3,766, and with drafts on decoded at 96.3 tok/s against 72.7. With drafts off it decoded at 50.1, so llama.cpp was 1.45 times faster there. The prompt was one repeated sentence, which favours drafting, and the README lists Super VL on a GB10, not on this card.

Which Mac can run Qwen 3.8 27B with TensorFold?

By file sizes and TensorFold's 70% memory limit, a 36 GB Mac holds the 27B and its draft model with about 56K of context at most, and a 48 GB Mac about 188K; both are ceilings, lower where macOS gives the GPU less than 70%. A 24 GB Mac does not fit it, and TensorFold's README says a 32 GB Mac needs more than the default budget.

Which models does TensorFold support?

TensorFold 1.2.1 and 1.2.0 build on the 1.1.4 list, which builds on the 1.0.12 list, which qualified Nemotron 3.5 Lightning (Macs from M1 to M5, the GB10, the RTX 3090 and the RTX PRO 6000, plus NVIDIA's NVFP4 checkpoint on the GB10 and oQ quants on an M3 Ultra), Nemotron 3.5 Super VL (one GB10 or two), Qwen 3.8 Flash Next (an M5 Ultra), GLM 5.3 Flash and the whole GLM 5.3 (two M5 Ultras, and by the 1.0.5 notes Flash on one with 256 GB), Qwen3.5-2B (an M5 Max), LFM2.5-24B-A2B (an M4 Pro) and, since 1.0.3, Qwen 3.8 27B with its DFlash2 drafter (an M5 Max and an M3 Ultra). 1.1.0 lifted the CUDA column onto the native line for Qwen3.5-2B, Qwen3.8-27B and Qwen3.8 Flash Next, GLM-5.3-Flash across two CUDA ranks on the EXL3 checkpoint, and Nemotron 3.5 Bolt from NVIDIA's 24 GB checkpoint. The same release brought Qwen3.6-35B-A3B to the native CUDA server and Gemma 4's 31B and 12B natively to Metal. The 1.1.4 notes add Gemma 4 12B and 31B on native CUDA, with their drafters. Ternary Bonsai 2 27B, Gemma 4 26B-A4B and DeepSeek V4 are not among them and stay on the Python 0.6.6 engine, kept on the python-0.6 line, and the 1.1.4 announcement says release binaries do not serve DeepSeek checkpoints yet. The 1.2.0 notes add MiniMax-M3.1-Flash on one 256 GB M5 Ultra, NVIDIA's NVFP4 Qwen3.6-35B-A3B on one GB10, dense EXL3 Qwen3.8-27B on CUDA, and Gemma 4 E4B on Intel's Arc Pro B70. The 1.2.1 notes add MiniMax-M3.1-Flash on two GB10s, Gemma 4 31B with MTP on a GB10, and Qwen3.6-35B-A3B at 1M on one GB10. The 1.2.3 notes add dots.mocr on HIP, Qwen3.5-MoE on Metal (Ornith-1.5-122B on one M5 Ultra), Qwen3-Embedding-8B and Qwen3-Reranker-8B on HIP, LFM3-1.7B on Intel Iris Xe, Nanbeige4.3-33B-INT4 on Metal, and PDF input. Kimi K3 and K3.1 are groundwork only, on a synthetic graph. The 1.2.4 notes add Gemma 4 E4B on CUDA, Clef-Flash's joint schema decisions and BF16 Qwen3.5 backbone on HIP, LFM3-1.7B text on Metal, and the HIP runtime on Windows. The 1.2.5 notes add TensorFold Web, LFM2.5-24B-A2B on CUDA, Nemotron 3.5 Parse and LightOnOCR-3 on HIP, Clef's joint schema head on CUDA, and a CPU projection and RMS norm in the kernel contract. The 1.2.6 notes add Tencent's DSpine-b16 as an opt-in drafter for Qwen3.8-27B on CUDA, --unix-socket, and POST /v1/images/generations. The 1.2.7 notes add LightOnOCR-3 image chat on CUDA, the Responses API over WebSocket, and a Qwen3.8-27B conformance profile for XPU on an Arc Pro B70.

What is new in TensorFold 1.2.7, 1.2.6, 1.2.5, 1.2.4, 1.2.3, 1.2.2, 1.2.1 and 1.2.0?

TensorFold 1.2.7 is mostly about resuming long conversations. On an M3 Ultra a follow-up in a 34K chat whose client drops earlier reasoning resumes at 34,100 tokens instead of 7,424, with its first token in about 0.3 s instead of 85 s. @niklaslenz_ai measured 0.22 s against 0.87 s without reasoning history on an M5 Ultra, and 0.10 against 0.51 with it. It also groups Flash Next's experts on M1 to M4, adds a Responses API WebSocket mode and LightOnOCR-3 image chat on CUDA, and fixes -Dgpu builds, damaged JPEGs and malformed GLM tool calls. We did not run it.

TensorFold 1.2.6 makes the prompt cache hold up under many agents: at 16 agents a follow-up's median first token drops from 546 s to 15.9 s on an M3 Ultra and from 105 s to 25.8 s on a GB10, with every reply equal to the previous version's. It adds Tencent's DSpine-b16 as an opt-in drafter for Qwen3.8-27B on CUDA, where agent-shaped requests are 2.4% to 19.8% faster than DFlash2 and chat and code are 10% to 19% slower. It also adds --unix-socket, an image-generation endpoint and reproducible tool-call ids. We did not run it.

TensorFold 1.2.5 adds TensorFold Web, a local browser app, plus LFM2.5-24B-A2B on CUDA, Nemotron 3.5 Parse and LightOnOCR-3 on HIP, and Clef's joint schema head on CUDA. On an M5 Ultra, Qwen3.8-27B MLX 8-bit cold prompts are 18% faster (649 to 768 tok/s at 2K) and decode 7.6% faster, with the same tokens. It also makes Flash Next BF16 cold prompts 3.7% to 4.4% faster on a GB10, HIP gfx12 rerank and embedding 2.9% to 4.8% faster, and refuses an explicit thinking off the template cannot honour. We did not run it.

TensorFold 1.2.4 is a serving and memory release. A 319K-token DeepSeek-V4.1 conversation restores from NVMe in 1.2 to 1.4 s instead of 155 to 178 s; a Flash Next chat at --tp 2 resumes in 1.5 s instead of 21.2 s; Flash Next prompts are 15% to 23% faster at --parallel 1, eight streams 3% to 6% faster, and a new short prompt's first token 19% sooner; two overlapping Flash Next clients on an M5 Ultra finish in 47 s instead of 127 s. It adds Gemma 4 E4B on CUDA, Clef-Flash on HIP, LFM3-1.7B text on Metal, and the HIP runtime on Windows. We did not run it.

TensorFold 1.2.3 is mostly memory and serving work. It makes plain decode on Flash Next NVFP4 over two GB10s at --tp 2 21% to 23% faster, with 2.62 GiB less per rank and a 65,536-token KV 40% smaller; a 1,024-token reply beside a 64K prompt on an M3 Ultra finishes in 8 s instead of 48 s. It adds PDF input, Qwen3.5-MoE on Metal (Ornith-1.5-122B on one M5 Ultra), dots.mocr on HIP, Qwen3-Embedding-8B and Qwen3-Reranker-8B on HIP, and LFM3-1.7B on Intel Iris Xe. Kimi K3 and K3.1 are groundwork only: a synthetic four-layer graph, no real weights, and K3.1 capped at 2,047 tokens. It also refuses more malformed checkpoints and does not reuse saved disk prompt states from earlier versions. We did not run it.

TensorFold 1.2.2 adds NVIDIA's Qwen3.8-Flash-Next NVFP4 checkpoint on CUDA at --tp 2, GLM-OCR on HIP, Qwen3.5-9B on Metal, and the first part of DeepSeek V4.1 on Metal. It also brings Qwen3.8-27B EXL3 prefill 38% to 42% faster on a GB10, HIP decode 7.5% to 18.9% faster on a gfx1151, Flash Next 36% to 39% faster decode on an M5 Max with --ple-on-ssd, a GB10 cold start down from about 74 s to 48 s, and loaders that release the checkpoint from the page cache. It is a CUDA and HIP feature release.

TensorFold 1.2.1 adds MiniMax-M3.1-Flash on two GB10s, Gemma 4 31B with Google's MTP assistant at 28.3 tok/s against 12.2 undrafted on a GB10, Qwen3.6-35B-A3B at 1M tokens on one GB10, Gemma 4 E4B prompts 2.0 to 2.6 times faster on an Arc B70, Nemotron 3.5 Lightning +13.3% at eight streams, and an opt-in GLM-5.3-Flash 64K resume in 1.7 seconds instead of 36. It also adds GET /v1/load. It is a CUDA and Metal feature release and does not replace the 1.2.0 Mac record.

Faster Qwen3.8-27B on Macs in 1.2.0, according to its release notes: on an M3 Ultra with DFlash2, 80.0 to 91.6 tok/s at one stream (+14.4%), 127.0 to 150.3 at four (+18.3%) and 145.2 to 209.4 at sixteen (+44.2%), with prefill +17%; on an M5 Ultra, 168.5 to 178.3 (+5.8%), 298.0 to 334.6 (+12.3%) and 378.5 to 474.0 (+25.2%). It adds MiniMax-M3.1-Flash on one 256 GB M5 Ultra, NVFP4 Qwen3.6-35B-A3B on CUDA, dense EXL3 Qwen3.8-27B on CUDA, and Gemma 4 E4B on Intel's Arc Pro B70. Every speedup keeps text replies token-equal to 1.1.9; Qwen-VL picture replies can differ slightly because pictures are prepared the way the checkpoint's processor does. We did not run it.

What changed in TensorFold 1.0?

The engine and server were rewritten in Zig and run as one native binary, installed on a Mac with Homebrew, with no Python or MLX needed to serve. Only qualified model and hardware pairs are admitted, and 1.0.2 added prompt caching kept on disk and faster GLM 5.3 Flash across two M5 Ultras, 199 tok/s across eight requests. 1.0.3 brought Qwen 3.8 27B to the native engine, RTX 30 cards for Nemotron, and Living Weights, an experimental mode in which the model learns facts by rewriting its own weights. 1.0.5 runs GLM 5.3 Flash on one 256 GB M5 Ultra and keeps prompt states between turns on Mac and CUDA. 1.0.6 and 1.0.7 let the 27B draft sampled requests and serve several at once, add RTX 50 and RTX PRO 6000 cards, and rebuild Living Weights around new neurons, learning on a GB10 too. 1.0.8 to 1.0.12 bring the whole GLM 5.3 to two Macs, oQ checkpoints, web research with --cortheon (off by default), and Nemotron 3.5 Super VL with images on CUDA.