NVIDIA published a local inference server called Flare
NVIDIA put a local inference server called Flare on GitHub on October 20, 2026, under the repository name flare-local-inference (Mike Gannotti's walkthrough, October 21, 2026). It runs on two machine families only: the GB10 chip in a DGX Spark, and the workstation Blackwell cards in the RTX PRO 6000 class. If you serve coding agents on either machine, this project deserves five minutes of your time.
The idea behind Flare sits opposite to llama.cpp or vLLM. Those engines try to run almost any model on almost any GPU. Flare runs only models NVIDIA has published a recipe for, and only on Nvidia's SM120 and SM121 chips (flare-local-inference README). The README calls Flare "a specialized option when its hardware and model coverage fit your machine", and it says plainly that Flare is not a replacement for production engines like vLLM. That is a rare sentence in a vendor README.
The approach, in plain English
A general purpose engine has to decide how to run your model while the server is already running. Flare moves those decisions to build time. The README's design notes say to "make memory and state explicit" and to plan buffers and execution capacities ahead of time. The heavy thinking happens once, when the package is built. What serves the request afterwards is a small, fast path.
The cost shows up in two places. Building the package needs the CUDA toolkit, and Gannotti's walkthrough calls that step the slow part. And the menu is short by design: the repository is open source, but it exists to run NVIDIA's own recipes on NVIDIA's own newest chips, nothing more (flare-local-inference README).
Conversation memory is the part aimed at coding agents. Many newer models carry a running internal state, a scratchpad the math updates on every token, alongside the usual KV cache. The README asks the server to treat that state as part of conversation memory, so reuse, branching and resuming preserve it (flare-local-inference README). Gannotti's walkthrough spells out the failure it prevents: a server that forgets the scratchpad either redoes the work or gets the continuation wrong. Long agent chats grow by one tool result at a time, so the reuse case is the common case.
Ignition, Helios and Orbit
The pipeline has three parts, and the names make it easy to hold in your head.
Flare's three parts, from the repository README and Mike Gannotti's walkthrough, October 2026
| Part | What it does | When it runs |
|---|---|---|
| Ignition | A Python program that loads the model, picks the GPU routines and records that work ahead of time. The output is a package: a small metadata file plus large companion files holding the weights and the recorded GPU work. Building needs the CUDA toolkit. | Once, before serving |
| Helios | A Rust program that serves a finished package. Python is not in the path when a request arrives. | On every request |
| Orbit | Multi machine mode. Each Spark runs a small agent, a controller starts each machine's piece, and the first machine answers the web requests over the ConnectX ports. | Only with two or more machines |
Packages fence the whole system, on both sides. You cannot point Helios at a normal Hugging Face model folder and expect it to start (Gannotti). Going the other way, the Qwen package card states in its first paragraph that Transformers, vLLM, SGLang and llama.cpp cannot load a Flare package either (Gannotti). A package built for the workstation cards will not load on a Spark, and a package built for four GPUs inside one computer cannot be turned into a package for four separate Sparks (the preparation and multi host guides, quoted in Gannotti's walkthrough).
What runs, and on which machine
The chip list is short. SM120 and SM121 mean the GB10 in a DGX Spark and the RTX PRO 6000 class workstation cards (flare-local-inference README; Gannotti). An AMD machine cannot run it. A Mac cannot run it. The Spark itself ships in 64 GB and 128 GB configurations, with CPU and GPU sharing that memory and the operating system taking a slice, and NVIDIA says up to four Sparks can be cabled together with ConnectX networking (NVIDIA's product page, quoted in Gannotti's walkthrough).
Models are still arriving. The repository's latest commit adds a Gemma 4 text recipe with Assistant MTP, a Qwen package card exists, and a community guide published on October 24 serves the package nvidia/GLM-5.3-Flash-FP4-Local with one rank on each of two DGX Sparks (PILCOTHINK, NVIDIA developer forums). That guide advertises a 3.4 million token KV cache across the pair. Treat it as one builder's result, not a benchmark.
Sizes need care before you plan a build. The recipe files list weight figures in GiB, the binary gigabyte, which is about 1.07 ordinary GB (Gannotti). Fitting the weights is not the same as fitting the chat beside them. A model that fills the memory leaves nothing for context, tools or the operating system.
Where TensorFold fits
The short answer: Flare and TensorFold aim at the same machines and jobs, and they package models in opposite ways.
TensorFold is an open source inference engine, at version 1.1.4 in its README on October 25, 2026, that serves language models from a Zig binary on Apple Silicon and NVIDIA GPUs (TensorFold README). It reads checkpoints, tokenizes requests and runs Metal or CUDA kernels directly, and serving needs no Python or MLX installation. The API listens at 127.0.0.1:8080/v1 by default and serves OpenAI chat completions, completions and Responses, plus Anthropic Messages (TensorFold README). Its qualified list covers Metal on M1 through M5 and CUDA on GB10, the RTX 3090 and the RTX PRO 6000 Blackwell, with CUDA support reaching back to Ampere cards such as the A100 and the RTX 30 series (TensorFold README). The site's front page names four model families: Qwen, Nemotron, GLM and LFM (tensorfold.dev).
Flare and TensorFold, from the two project READMEs and tensorfold.dev, October 25, 2026
| Flare | TensorFold | |
|---|---|---|
| Maker | NVIDIA, as a research prototype | ashhart, an NVIDIA Inception member |
| Version | First published October 20, 2026 | 1.1.4 on October 25, 2026 |
| Licence | Apache 2.0 server; model packages carry their own licences | Apache-2.0 engine |
| Hardware | SM120/SM121: GB10 and RTX PRO 6000 Blackwell only | Apple M1 through M5; CUDA on Ampere, GB10 and RTX PRO 6000 Blackwell |
| Model input | Flare packages only | Checkpoints, including MLX affine quants and NVFP4 for qualified models |
| Interface | Head machine answers web requests | OpenAI and Anthropic compatible endpoints |
Two differences decide which one to try. Packaging comes first: TensorFold reads model folders directly, while a Flare package must be built by Ignition before Helios can serve it, and no other engine can read the result. Hardware comes second: TensorFold also runs on Macs and on Ampere cards as old as the RTX 30 series, while Flare stops at SM120 and SM121.
Speed claims sit differently too. TensorFold's site records 874 tok/s decode for Nemotron 3.5 Lightning on an RTX PRO 6000 Blackwell at Release 1.0.6, greedy code decode with drafting enabled on an MLX affine 4-bit checkpoint, and the site labels its own figures "separate workloads and machines, not a like-for-like comparison" (tensorfold.dev, the vendor's own numbers). Our speed tool lists measured figures by model and card. The project's self reported measurements put decode at 1.6 to 3.1 times vLLM's speed with MTP=3 on DGX Spark across Qwen 3.8 27B, Qwen 3.8 Flash Next and GLM 5.3 Flash (TensorFold's own measurements, reported by flowtivity, September 27, 2026). Flare's README publishes no benchmark figures at all, and NVIDIA's own text tells you to compare engines on your model and your chats (Gannotti).
On the card class Flare targets, a measured head to head between TensorFold and llama.cpp now exists, and it is our own TensorFold test. We ran Nemotron 3.5 Super VL on one RTX PRO 6000 Blackwell with 96 GB through TensorFold 1.0.12 (the current release at the time) and llama.cpp b11696, the prebuilt CUDA 12.8 release, both engines on NVFP4 checkpoints, with a 113,108 token document as the prompt and temperature 0. Driver 580.82.07, ECC on. The TensorFold README does not list that model for that card, so this is outside its qualified list. Our full write-up, updated October 21, 2026, carries the method. At the 128K window, llama.cpp read the prompt at 3,766 tokens per second and decoded 72.7, while TensorFold with drafts on read it at 6,381 and decoded 96.3. The drafts changed no token. They carried the speed: with drafts on, TensorFold decoded 1.32 times as fast as llama.cpp at 128K and 1.24 times at 262K, and without them llama.cpp decoded 1.45 times as fast at 128K and 1.58 times at 262K. Memory ended in a tie within 0.2 GiB, and the drafter cost TensorFold about 1.9 GiB.
Organisationally the two are close but separate. TensorFold is part of NVIDIA's Inception startup program (tensorfold.dev), and it counted roughly 328 GitHub stars in late September 2026 (flowtivity). Flare is NVIDIA's own code. Both READMEs, read on October 25, 2026, describe their projects without naming the other. The licences have moved recently as well: TensorFold's engine switched to Apache-2.0 at version 0.6.0 (TensorFold repository), which postdates the MIT licence description in flowtivity's September review.
Verdict or what to do next
If you own a DGX Spark or an RTX PRO 6000 Blackwell and serve coding agents, watch the repository and read the preparation guide before building anything, because the package rules are strict. If you run a Mac, an RTX 3090 or anything else, Flare will not start on your machine, and TensorFold or llama.cpp remain the working routes today. Nobody should move production serving to a five day old prototype. The README itself says Flare is not a replacement for production engines. Test on your own model, your own chats and your own stopwatch.
Check the fit before you build. Package builds are slow and the weights are only half the memory story. Our GPU checker tests every model in the catalogue against your card, and the VRAM calculator adds quantization and context length.
Sources
- Mike Gannotti, 'NVIDIA Flare in plain English', October 21, 2026. https://x.com/i/article/2112866195078549825
- NVIDIA, 'flare-local-inference', GitHub repository README, accessed October 25, 2026. https://github.com/NVIDIA/flare-local-inference
- PILCOTHINK, 'Flare-local-inference: GLM-5.3-Flash Guide for 2x DGX Spark, 3.4M-Token KV Cache', October 24, 2026. https://forums.developer.nvidia.com/t/flare-local-inference-glm-5-3-flash-guide-for-2x-dgx-spark-3-4m-token-kv-cache/386738
- TensorFold, 'TensorFold: LLM Inference Engine for Metal and CUDA', GitHub README, accessed October 25, 2026. https://github.com/ashhart/TensorFold
- TensorFold, 'Run LLMs, Fast', tensorfold.dev, accessed October 25, 2026. https://tensorfold.dev
- AJ Awan, 'TensorFold: The Inference Engine That Makes Speculative Decoding Exact', September 27, 2026. https://flowtivity.ai/blog/tensorfold-inference-engine-review
- vramcalculator.com, 'TensorFold Against llama.cpp on a 96 GB RTX PRO 6000' (our own test), updated October 21, 2026. https://vramcalculator.com/tensorfold/
- NVIDIA, 'NVIDIA FLARE', accessed October 25, 2026. https://developer.nvidia.com/flare
FAQ
Is NVIDIA Flare the same as NVIDIA FLARE?
No. Flare, the subject of this article, is the local inference server for DGX Spark and RTX PRO 6000 Blackwell class machines that NVIDIA published on GitHub on October 20, 2026 (Mike Gannotti's walkthrough). NVIDIA FLARE, written in capitals, is the company's federated learning SDK, a separate product with its own documentation on NVIDIA's developer site. Same name, different jobs.
Can Flare run on a Mac or an AMD GPU?
No. The README restricts Flare to Nvidia's SM120 and SM121 chips, which appear in the DGX Spark and the RTX PRO 6000 class (Mike Gannotti's walkthrough). Mac owners who want a local server can look at TensorFold, whose qualified platforms include Metal on M1 through M5 (TensorFold README).
Can Flare load GGUF or a Hugging Face model folder?
No. Helios, the serving program, starts only from a package that Ignition built, and the Qwen package card states that Transformers, vLLM, SGLang and llama.cpp cannot load a Flare package either (Mike Gannotti's walkthrough). Conversion happens once, at build time, on a machine with the CUDA toolkit installed.
Does Flare publish benchmark numbers?
Not in the README, five days after publication. NVIDIA's text tells readers to compare engines on their own model and their own chats (Mike Gannotti's walkthrough, October 21, 2026). For a same hardware data point, TensorFold's site records 874 tok/s decode for Nemotron 3.5 Lightning on an RTX PRO 6000 Blackwell, a vendor published figure.
What is a Flare recipe?
A recipe is the per model definition that Ignition follows when it builds a package. The recipe files in the repository also list weight sizes, counted in GiB rather than GB (Mike Gannotti's walkthrough). Only models with a recipe run under Flare.
Is Flare free to use?
The server code is Apache 2.0 (Mike Gannotti's walkthrough; the repository). The model packages are published on Hugging Face separately, and Gannotti's walkthrough warns not to assume the weights carry the server's licence. Each package states its own terms.