LLM Tokens Per Second: Check What Your GPU Delivers

Speed estimate

How Many Tokens Per Second Will You Get?

ESTIMATENot a measurement

--tok/s Estimated decode speed
MEMORY--
QUANT--
CONTEXT--
PER WATT--

How this is calculated, and how wrong it gets

Measured runs are used privately to check this estimator; this page does not rank them.

Keep YOUR tool updated and accurate.ETH vramcalculator.eth 0x786f…1B09

Domains for sale: ainews247.com and ailogo365.com, both listed on Atom.com.

The number on this page is an estimate of decode LLM tokens per second for one card and one model: a bandwidth ceiling, derived from arithmetic you can check. It is not a measurement, and the answer card says so. We check it against measured runs other people published on real hardware, and those runs now stay off the page. Everything below explains how the estimate is built, how far it misses, and what the figure means once you have it.

How this estimate is calculated, and how wrong it gets

Decode is bandwidth bound before it is anything else. To emit one token, the model reads its weights out of memory once. So the ceiling is the bytes read per token divided by the bandwidth of the memory they are read from, and then scaled by the fraction of its rated bandwidth a real memory controller actually delivers. The panel gives you the answer and how much of the card it takes, rather than the working.

Three details decide whether that arithmetic lands anywhere near a real machine.

Active parameters, not total

A Mixture-of-Experts model keeps every expert in memory and reads a fraction of them per token, so traffic is set by the active parameters. A 180B model with 6B active reads roughly what a 6B dense model reads, which is why it answers quickly on hardware that has no business running 180B. Our catalogue records active and total separately for this reason. The total still has to fit, and that is what the model list on this page enforces before it offers you anything.

No memory controller delivers its rated figure

The default is 75 percent of rated bandwidth, which holds across GDDR and HBM. A device we have measurements for carries its own factor instead. The DGX Spark is the one in our data that needed it. Three MoE runs on that machine land at about 30 percent of its rated 273 GB/s, while a dense run on the same box reaches 76 percent, which is ordinary GDDR6 territory. The penalty belongs to the sparse gather an MoE does, not to LPDDR5X, so it is applied to MoE models only. That is four data points across two model classes. Calibration, not physics, and it is written down as such.

The fit check runs at 8K context

Every model offered for a card is checked at 8,192 tokens of context with an FP16 KV cache. Push the context further and the cache grows, so a pair that fits here can stop fitting at 128K. The GPU checker answers the memory question at any context length. This page answers the speed question at one.

Scored against measured runs

An estimate that never states its error is a guess with a font size. Ours is scored against the measured runs it is willing to answer for, by a script that drives the shipped page rather than a second copy of the formula. A scorer that reimplements the thing it scores agrees with itself by construction, and the first version of ours did exactly that, reporting healthy numbers for a formula that was out by a factor of twelve.

Checked privately against 63 measured runs, each at the quantization it used, the estimate typically lands 44 to 88 percent away from the measured figure: below it on 38 of the 45 runs that used speculative decoding such as DFlash 2, and above it on 12 of the 18 that did not.

Issue 2 of The Local LLM Sizing Guide

The Local LLM Sizing Guide. Issue 2, 31 October

Speed starts with fit. Issue 2 has every cell of our own Strata runs, and what one engine update did to them.

Decode, prefill, first token and memory for each build on our RTX PRO 6000. Every model sized for every tier from 8 GB to 128 GB, with the context each cache type holds.

Know what fits before you buy

$24.90 for three issues, or $9 a month. Card or crypto.

What the colour means

Red under 30 tokens a second, amber to 50, green above. Those bands assume an agentic or reasoning workload, where the output runs to tens of thousands of tokens and nobody is reading along.

The published thresholds for chat are far more forgiving, and it is worth seeing why they do not transfer:

RateWhat it isWhose criterion
4 to 7 tok/shuman reading speed, 3 to 5 words a secondarithmetic
15 to 20 tok/scomfortable to read along withconsumer guides, chat
30 to 40 tok/sfeels instantconsumer guides, chat
20 to 120 tok/sobserved range for coding agentsTogether.ai
60 to 90 tok/s55 to 83 minutes for a 300K token taskTogether.ai

Every one of the chat numbers is a claim about a person watching text arrive. That assumption dies on the workload this audience actually has. A Microsoft study of GitHub Copilot at production scale and a SWE-bench Verified analysis both put agentic work at roughly a thousand times the tokens of code chat, at 500K to 2M tokens per developer per day.

Reasoning models make it worse in a way the headline rate hides. Artificial Analysis measured GLM 5.3 Flash across their Intelligence Index and found 134 million of 149 million output tokens were reasoning, about 90 percent. The wait is not the answer arriving. It is the model thinking somewhere you cannot see it, at the same tokens per second.

So our bands are the chat “feels instant” line applied to a job with three orders of magnitude more tokens in it. The first version used 10 and 30, anchored to reading speed alone. That was the wrong anchor: the threshold belongs to the workload, not to a reader’s eyes.

What the estimate deliberately ignores

  • Prefill. Reading your prompt is compute bound and behaves nothing like decode. @0xBakeer measured 500 to 660 tok/s of prefill on the same Spark that decodes at 22.
  • Offload. Keeping part of a model in system RAM can make a configuration faster than not running it at all, which is the whole point of llama.cpp’s MoE offload flags. The speed then belongs to the offload path rather than to the card, and that is why the estimator does not cover it: offload speed is set by the CPU, the memory controller, the DIMM configuration and the lanes behind the slot, and those vary more between two machines with the same GPU than GPUs vary between themselves. The estimate covers models that fit entirely in VRAM. One exception is worth knowing: where a model’s architecture separates a lookup table rather than weights on the critical path, the cost is small. Qwen 3.8 Flash Next carries a 51B n-gram embedding table alongside its 125B network, and llama.cpp keeps that table on the CPU side and reads the few rows each token needs. @geldeki runs it with the table in system RAM on two RTX 3090s at about 85 tok/s. Compare 5 tok/s on a 4060 with experts in system RAM.
  • The backend. @splizard ran the same model on the same 7900 XTX under two stacks and got 48.5 and 71 tok/s. That is a 1.46x spread from software alone.
  • Speculative decoding. A drafter multiplies throughput by how well it predicts your workload, so the same setup can double on code and gain little on prose.

Why some cards and models are not offered

A model that does not fit the card you picked is not in the list at all, and a quantization that does not fit is not in its list either. That is deliberate. A model running mostly from system memory has a speed set by the PCIe bus and the DDR behind it, and this page models neither. Returning a confident bandwidth figure for a configuration that never touches that bandwidth is the worst thing a tool like this can do.

The same rule governs the scoring. A run split across several devices, where the panel answers for one, or a run of a model that does not fit the card, is declined rather than scored. Refusing is the correct answer for both.

Where these measured LLM tokens per second figures come from

From people who ran them and posted the result: every run we keep carries the handle of the person who published it and the date they published it. Flags, context length, quantization and the tester’s own caveats travel with the number, because a decode figure without its method cannot be checked by anybody.

Since 5 November the runs no longer appear on this page. Too many published runs use reworked quantizations rather than the standard files, so a public table ranked setups a reader could not reproduce. The runs still score the estimator, privately.

Every run stays attributed to the person who measured it. If you have published a run on hardware in our list, the fastest way into the corpus is a post carrying the model, the quantization, the context length, the backend build and the figure as your tool reported it.

Frequently Asked Questions

Is 20 tokens per second good?

For chat, yes. It is faster than you read. For an agent working through a task unattended, it is the difference between a coffee and an afternoon: a 40,000 token job takes 33 minutes at 20 tok/s and 13 at 50.

Why is my model slower than the estimate?

Almost always because part of it is not on the card. Check that the whole model plus the KV cache fits at the context length you are actually running, which the GPU checker will tell you. After that the usual suspects are a backend with no tuned kernel for your card, and a context long enough that the cache is now competing with the weights for bandwidth.

Why is my model faster than the estimate?

Speculative decoding, batching, or a backend doing something clever with the memory traffic. The estimate models none of them, so it misses in that direction. If you are getting more than the number on this page, the number on this page was a floor and your stack is doing its job.

Does more VRAM make a model faster?

Not directly. Capacity decides whether it runs at all; bandwidth decides how fast. Two cards with the same 24 GB and different memory generations will not deliver the same tokens per second. Capacity is the question the VRAM calculator answers, and it is the one to settle first.

Do Apple machines work the same way?

The arithmetic is the same and the memory is not. Unified memory gives the GPU a share of one pool rather than a fixed allocation, and the share is not published as a formula. The Mac checker handles that case separately.