LLM Tokens Per Second: Check What Your GPU Delivers

Two numbers sit on this page and they are not the same kind of number. The estimator gives you a bandwidth ceiling for one card and one model, derived from arithmetic you can check. The table under it holds measured LLM tokens per second figures other people published on real hardware, each one attributed to the person who ran it. Everything below explains how the first is built, how far it misses, and what the figure means once you have it.

How this estimate is calculated, and how wrong it gets

Decode is bandwidth bound before it is anything else. To emit one token, the model reads its weights out of memory once. So the ceiling is the bytes read per token divided by the bandwidth of the memory they are read from, and then scaled by the fraction of its rated bandwidth a real memory controller actually delivers. The panel gives you the answer and how much of the card it takes, rather than the working.

Three details decide whether that arithmetic lands anywhere near a real machine.

Active parameters, not total

A Mixture-of-Experts model keeps every expert in memory and reads a fraction of them per token, so traffic is set by the active parameters. A 180B model with 6B active reads roughly what a 6B dense model reads, which is why it answers quickly on hardware that has no business running 180B. Our catalogue records active and total separately for this reason. The total still has to fit, and that is what the model list on this page enforces before it offers you anything.

No memory controller delivers its rated figure

The default is 75 percent of rated bandwidth, which holds across GDDR and HBM. A device we have measurements for carries its own factor instead. The DGX Spark is the one in our data that needed it. Three MoE runs on that machine land at about 30 percent of its rated 273 GB/s, while a dense run on the same box reaches 76 percent, which is ordinary GDDR6 territory. The penalty belongs to the sparse gather an MoE does, not to LPDDR5X, so it is applied to MoE models only. That is four data points across two model classes. Calibration, not physics, and it is written down as such.

The fit check runs at 8K context

Every model offered for a card is checked at 8,192 tokens of context with an FP16 KV cache. Push the context further and the cache grows, so a pair that fits here can stop fitting at 128K. The GPU checker answers the memory question at any context length. This page answers the speed question at one.

Scored against the runs on this page

An estimate that never states its error is a guess with a font size. Ours is scored against every measured run in the table above that it is willing to answer for, by a script that drives the shipped page rather than a second copy of the formula. A scorer that reimplements the thing it scores agrees with itself by construction, and the first version of ours did exactly that, reporting healthy numbers for a formula that was out by a factor of twelve.

Tester	Hardware	Backend	Measured	Estimate	Error
@0xBakeer	1x DGX Spark	llama.cpp, unmerged PR	22.0	22	0%
@wikiwayne	1x DGX Spark 128 GB	llama.cpp qwen4exp	22.9	22	-4%
@splizard	1x 7900 XTX	llama.cpp Vulkan, MTP n=3	48.5	42	-13%
@0xBakeer	1x DGX Spark	vLLM, NVFP4	27.5	22	-20%
@filicroval	1x DGX Spark	llama.cpp, no speculation	27.7	22	-21%
@splizard	1x 7900 XTX	tinygrad, MTP K=3	71.0	42	-41%
@analogalok	1x RTX 4090	llama.cpp with DFlash 2	87.0	44	-49%
@ViC305	1x DGX Spark	llama.cpp with DFlash 2	24.5	12	-51%
@filicroval	1x DGX Spark	llama.cpp ngram-mod speculation	82.6	22	-73%
@pupposandro	1x AMD R9700 32 GB	Lucebox with DFlash 2	227.0	28	-88%

Mean absolute error across those ten is 36 percent, and every single miss is an underestimate. That pattern is the useful part. The size of the miss tracks how much acceleration the run was using, which is the one term the formula does not model at all.

Two of those testers published a baseline alongside their accelerated figure, and against the baseline the picture changes completely. @ViC305 reported 12.37 tok/s before turning DFlash 2 on, against our 12. @analogalok reported 60 before, against our 44. So the number this page gives you is a floor for an accelerated setup, not a midpoint, and it is close to right for a plain one.

What the colour means

Red under 30 tokens a second, amber to 50, green above. Those bands assume an agentic or reasoning workload, where the output runs to tens of thousands of tokens and nobody is reading along.

The published thresholds for chat are far more forgiving, and it is worth seeing why they do not transfer:

Rate	What it is	Whose criterion
4 to 7 tok/s	human reading speed, 3 to 5 words a second	arithmetic
15 to 20 tok/s	comfortable to read along with	consumer guides, chat
30 to 40 tok/s	feels instant	consumer guides, chat
20 to 120 tok/s	observed range for coding agents	Together.ai
60 to 90 tok/s	55 to 83 minutes for a 300K token task	Together.ai

Every one of the chat numbers is a claim about a person watching text arrive. That assumption dies on the workload this audience actually has. A Microsoft study of GitHub Copilot at production scale and a SWE-bench Verified analysis both put agentic work at roughly a thousand times the tokens of code chat, at 500K to 2M tokens per developer per day.

Reasoning models make it worse in a way the headline rate hides. Artificial Analysis measured GLM-5.3-Flash across their Intelligence Index and found 134 million of 149 million output tokens were reasoning, about 90 percent. The wait is not the answer arriving. It is the model thinking somewhere you cannot see it, at the same tokens per second.

So our bands are the chat “feels instant” line applied to a job with three orders of magnitude more tokens in it. The first version used 10 and 30, anchored to reading speed alone. That was the wrong anchor: the threshold belongs to the workload, not to a reader’s eyes.

What the estimate deliberately ignores

• Prefill. Reading your prompt is compute bound and behaves nothing like decode. @0xBakeer measured 500 to 660 tok/s of prefill on the same Spark that decodes at 22.
• Offload. Keeping part of a model in system RAM can make a configuration faster than not running it at all, which is the whole point of llama.cpp’s MoE offload flags. The speed then belongs to the offload path rather than to the card, and that is why the estimator does not cover it: offload speed is set by the CPU, the memory controller, the DIMM configuration and the lanes behind the slot, and those vary more between two machines with the same GPU than GPUs vary between themselves. The estimate covers models that fit entirely in VRAM. Offload runs are still recorded in the table below, labelled, because they happened. One exception is worth knowing: where a model’s architecture separates a lookup table rather than weights on the critical path, the cost is small and measurable. Qwen3.8-Flash-Next carries a 51B n-gram embedding table alongside its 125B network, and holding that table off the device cost @0xBakeer about 6% throughput. Compare 5 tok/s on a 4060 with experts in system RAM.
• The backend. @splizard ran the same model on the same 7900 XTX under two stacks and got 48.5 and 71 tok/s. That is a 1.46x spread from software alone.
• Speculative decoding. A drafter multiplies throughput by how well it predicts your workload, so the same setup can double on code and gain little on prose.

Why some cards and models are not offered

A model that does not fit the card you picked is not in the list at all, and a quantization that does not fit is not in its list either. That is deliberate. A model running mostly from system memory has a speed set by the PCIe bus and the DDR behind it, and this page models neither. Returning a confident bandwidth figure for a configuration that never touches that bandwidth is the worst thing a tool like this can do.

The same rule governs the corpus. Of the 29 measured runs collected here, the estimator declines 19: multi-device splits where the panel answers for one device, pairs where the model does not fit the card, and one A100, which is not in the card list. Refusing is the correct answer in all of them, and the page says which reason applies.

Where these measured LLM tokens per second figures come from

Twenty nine runs, fifteen testers, thirteen distinct backends, every row carrying the handle of the person who published it and the date they published it. Flags, context length, quantization and the tester’s own caveats travel with the number, because a decode figure without its method cannot be checked by anybody.

None of these are our measurements. Where a run came out of one of our own articles, the article is linked on the row. If you have published a run on hardware in our list and it is not here, the fastest way in is a post carrying the model, the quantization, the context length, the backend build and the figure as your tool reported it.

Frequently Asked Questions

Is 20 tokens per second good?

For chat, yes. It is faster than you read. For an agent working through a task unattended, it is the difference between a coffee and an afternoon: a 40,000 token job takes 33 minutes at 20 tok/s and 13 at 50.

Why is my model slower than the estimate?

Almost always because part of it is not on the card. Check that the whole model plus the KV cache fits at the context length you are actually running, which the GPU checker will tell you. After that the usual suspects are a backend with no tuned kernel for your card, and a context long enough that the cache is now competing with the weights for bandwidth.

Why is my model faster than the estimate?

Speculative decoding, batching, or a backend doing something clever with the memory traffic. Every miss in the table above runs in that direction. If you are getting more than the number on this page, the number on this page was a floor and your stack is doing its job.

Does more VRAM make a model faster?

Not directly. Capacity decides whether it runs at all; bandwidth decides how fast. Two cards with the same 24 GB and different memory generations will not deliver the same tokens per second. Capacity is the question the VRAM calculator answers, and it is the one to settle first.

Do Apple machines work the same way?

The arithmetic is the same and the memory is not. Unified memory gives the GPU a share of one pool rather than a fixed allocation, and the share is not published as a formula. The Mac checker handles that case separately.