What VRAM do you need to run local LLMs in 2026?

Running local LLMs in 2026 requires a minimum of 8GB VRAM for 4B–8B parameter models under Q4_K_M quantization. 16GB VRAM handles models up to 14B parameters comfortably, with Gemma 4 12B fitting natively including its multi-modal stack. Dense 30B+ models require 24GB to 32GB, making the RTX 4090, RTX 5090, or Apple M4 Max the practical baseline for unconstrained local inference at that scale.

The 2026 Reality: VRAM Is the Absolute Bottleneck

CUDA core count determines how fast you generate tokens. VRAM capacity determines whether you can generate them at all.

The constraint is binary: if a model’s weight tensors plus the KV cache required for your target context length exceed your GPU’s available VRAM, the runtime offloads layers to system RAM across a PCIe bus that cannot compete with GDDR7 bandwidth. Inference speed collapses to single-digit tokens per second. Interactive use becomes impractical.

How the RTX 50-Series Changed the Consumer Baseline

The RTX 5090 delivers 32GB of GDDR7 memory across a 512-bit memory bus, achieving approximately 1.79 TB/s of bandwidth — a 78% improvement over the RTX 4090’s 1.008 TB/s. LLM inference is almost entirely memory-bandwidth-bound: model weights must stream from VRAM to compute cores for every generated token, so bandwidth improvement translates directly to faster token generation at equivalent model size and quantization. The RTX 5080 brings the same architectural generation to the 16GB tier with comparable bandwidth gains.

Even with this hardware generation, VRAM capacity — not compute — remains the binding constraint that determines which models you can run.

VRAM Requirements Table: 2026 Models at Key Quantization Levels

KV cache overhead estimated at 8k context window. Context scales linearly: 32k context approximately quadruples the KV cache demand shown above.

VRAM Tiers: What Each Capacity Actually Runs in 2026

8GB VRAM — The Minimum Viable Tier

8GB remains the entry threshold for local inference in 2026, but the operating envelope is narrow. You are working with heavy quantization on small architectures where the trade-off between model quality and hardware headroom is constant.

Target hardware: RTX 4060, RTX 5060

What runs natively: Google’s Gemma 4 2B and 4B models are purpose-built for edge deployment. Both operate at Q4_K_M with comfortable VRAM headroom at standard context lengths. The Llama 3.1 8B is the practical ceiling at this tier — it requires strict Q4_K_M quantization and a conservative context window to avoid triggering RAM offload.

Where it breaks down: Any model beyond 8B parameters will trigger partial layer offload to system RAM at this tier, regardless of quantization. Multi-modal inputs, agentic loops with accumulated context, or multi-turn conversations that build KV cache toward the gigabyte range create hard constraints that no quantization technique can fully circumvent.

Practical reality: 8GB is functional for single-task deployment of sub-8B models. For complex reasoning, multi-turn productivity workflows, or code generation requiring large context, 8GB is a ceiling, not a foundation.

16GB VRAM — The Mainstream Production Standard

The 16GB tier is the practical sweet spot for the majority of local inference use cases in 2026. It supports capable multi-modal instruction-following models without requiring multi-GPU infrastructure, and handles the bulk of real-world workloads cleanly.

Target hardware: RTX 4080 Super, RTX 5080, M3 Pro with 18GB unified memory, standard workstation laptops

What runs natively: Gemma 4 12B is the headline model for this tier. Google engineered the 12B variant as an encoder-free multi-modal architecture — it processes text, images, audio, and video within a single model, without separate encoder models consuming additional VRAM. The entire stack runs on a 16GB VRAM laptop without offloading under standard context windows.

Qwen 2.5 14B at Q4_K_M also fits comfortably at this tier. At roughly 8.5GB of weight VRAM, it leaves adequate headroom for KV cache at 16k–32k context lengths while delivering strong multilingual instruction-following capability.

Where it breaks down: 24B+ dense models force aggressive RAM offloading. Qwen 2.5 14B at Q8_0 precision consumes approximately 15GB — filling nearly the full capacity and leaving insufficient headroom for meaningful context windows. Extended agentic sessions accumulating large KV caches will hit the ceiling mid-task.

24GB to 32GB VRAM — The Enthusiast and Research Baseline

This tier is the minimum for serious local inference: running 30B+ dense models without compromise, enabling 128k+ context windows without KV cache overflow, and supporting autonomous agent loops with substantial working memory.

Target hardware: RTX 3090 (24GB), RTX 4090 (24GB), RTX 5090 (32GB)

What runs on 24GB: The RTX 4090 handles Gemma 4 31B Dense at Q4_K_M with limited headroom for context — at approximately 18 GB of weight VRAM and 4 GB of KV cache at 8k context, the total approaches 23 GB, leaving roughly 1 GB of margin. Gemma 4’s 26B Mixture-of-Experts variant is more comfortable at this tier. MoE architectures activate only a subset of parameters per forward pass, meaning effective memory bandwidth pressure is significantly lower than the raw parameter count implies — while output quality tracks dense models of comparable active parameter size on reasoning benchmarks.

What the RTX 5090’s 32GB unlocks: The additional 8GB over the RTX 4090 is strategically significant rather than incrementally useful.

Supported Architectures: The 32GB RTX 5090 accommodates Google’s flagship Gemma 4 31B model and its 26B Mixture-of-Experts (MoE) variant natively without aggressive quantization constraints. Pushing the ceiling further, the RTX 5090 also fits Llama 3.3 70B at Q3_K_M quantization — consuming approximately 30GB and leaving 1–2GB of headroom.

This is a tight configuration. Context windows must remain moderate, and parallel GPU workloads (desktop compositor, other applications) risk triggering OOM. It is a functional 70B setup but not a comfortable one.

The quantization quality trade-off at 32GB: Llama 3.3 70B at Q3_K_M on an RTX 5090 delivers measurably lower output quality compared to Q4_K_M on a multi-GPU or Apple Silicon setup. Q3 quantization introduces compression artifacts particularly visible in complex multi-step reasoning, mathematical problem-solving, and long-form structured generation — tasks where 70B models are typically deployed. If output quality is the primary constraint rather than hardware cost, 32GB single-GPU is not the endpoint for 70B-class inference.

Apple Silicon Unified Memory — The Architectural Alternative

Nvidia dominates raw token throughput. Apple Silicon’s memory architecture bypasses the traditional discrete GPU VRAM bottleneck in ways that are uniquely relevant to large model inference.

How unified memory changes the constraint: Standard GPU inference requires model weights to reside entirely in discrete VRAM, physically isolated from system RAM. Apple’s M-series architecture shares a single high-bandwidth memory pool between CPU and GPU on the same die. The inference runtime sees the entire unified memory pool — no PCIe bus, no offloading penalty, no separate VRAM ceiling.

Target hardware: M4 Max (up to 128GB unified memory), M4 Ultra (up to 192GB unified memory)

What this unlocks: Llama 3.3 70B at Q4_K_M requires approximately 40–41GB of memory — a configuration requiring dual RTX 3090s with NVLink or a professional data center GPU on discrete hardware. A single M4 Max at 48GB absorbs the full model with comfortable context headroom. The M4 Ultra at 192GB runs 70B models at Q8_0 precision, or multiple 70B instances simultaneously for batch workloads.

The bandwidth trade-off: Apple Silicon’s unified memory bandwidth — approximately 800 GB/s on the M4 Ultra — is significantly below the RTX 5090’s 1.79 TB/s. This translates directly to lower tokens-per-second throughput on equivalent model configurations. The trade-off is capacity versus speed.

For latency-sensitive workloads (real-time chat, voice interfaces, interactive coding assistance), an RTX 5090 running a smaller model at higher precision often outperforms an M4 Max running a larger model in raw responsiveness. For batch inference, research, or workloads where model capability matters more than generation speed, Apple Silicon’s capacity advantage is decisive.

Q4_K_M vs Q8_0: The Quantization Decision Matrix

Quantization is not a binary quality-versus-VRAM switch. The format choice has distinct implications depending on model scale and task type.

Q4_K_M uses a mixed-precision scheme in which attention layers retain higher precision while feed-forward network weights compress aggressively, averaging approximately 0.5 bytes per parameter. Quality degradation is measurable in perplexity benchmarks but largely imperceptible in instruction-following, coding, and reasoning tasks for models above 13B parameters. This is the default format for most GGUF releases on llama.cpp and Ollama.

Q8_0 stores weights as 8-bit integers at approximately 1 byte per parameter. Output quality is effectively indistinguishable from FP16 on nearly all benchmarks. The practical limitation is whether your VRAM accommodates the weight size — Q8_0 for a 14B model requires approximately 15GB, leaving minimal headroom on a 16GB GPU.

Choosing the right format for your hardware:

• 8GB GPUs: Q4_K_M is the default and ceiling. Q8_0 is rarely viable beyond 7B models.
• 12GB GPUs: Q4_K_M for models up to 14B. Q8_0 viable for 7B–8B models.
• 16GB GPUs: Q4_K_M for 12B–14B models. Q8_0 for 7B–8B where quality is prioritized.
• 24GB GPUs: Q4_K_M for 31B models. Q8_0 feasible for 14B and below.
• 32GB GPUs: Q4_K_M for 31B dense. Q3_K_M is the only option for 70B models — with meaningful quality trade-offs. Note: 31B at Q8_0 (~33 GB weights) exceeds the 32GB ceiling and requires 48GB+ hardware.
• Apple Silicon 64GB+: Q4_K_M for 70B models comfortably. Q8_0 for 30B–40B models.

Q3_K_M should be treated as a last resort for fitting 70B-class models on 32GB hardware. Quality degradation is measurable and task-dependent — complex reasoning chains, precise instruction-following, and long-form structured output show the most pronounced degradation versus Q4_K_M.

KV Cache: The Context Window Tax

The VRAM requirements table above reflects weight sizes only. Running inference also requires memory for the KV (Key-Value) cache — the stored attention states that allow the model to reference prior tokens in the context window. KV cache demand scales with context length, batch size, number of attention heads, and model depth.

For Llama 3.3 70B at a 4k context window, KV cache overhead is approximately 2–3GB. Extending to 32k context raises that overhead to approximately 15–20GB — impossible on a single RTX 5090 alongside the model weights without KV cache quantization. llama.cpp supports KV cache quantization via the -ctk q8_0 and -ctk q4_0 flags, recovering significant headroom at a minor quality cost to attention precision.

Practical planning rule: When calculating VRAM requirements, add a minimum of 2–4GB headroom beyond the model weight size for short context workloads. For extended context tasks — document summarization, agentic loops, code review across large files — treat KV cache as a primary budget line rather than an afterthought.

Bare-Metal Optimization: Recovering Hidden VRAM

Hardware specification sheets represent theoretical maximums. The operating system layer consumes VRAM before a single model weight is loaded.

Consumer Windows 11 installations passively reserve 2GB to 4GB of combined GPU VRAM and system RAM for the desktop compositor (DWM), telemetry services, background application stack, and Windows Update. On an 8GB GPU, this overhead consumes up to 50% of available capacity before inference begins.

Reducing OS VRAM overhead in practice:

Lean Windows configurations — whether IoT Enterprise LTSC editions, debloated Windows 10 LTSC builds, or stripped Windows 11 Pro installations with disabled compositor and background services — recover 800MB to 1.5GB of GPU VRAM overhead compared to a standard Windows 11 Home baseline. For 8GB and 12GB GPUs, this headroom directly expands the viable KV cache budget and can be the difference between a model running cleanly and triggering OOM during a long generation.

On Linux, a headless server configuration (Ubuntu Server, Debian minimal) presents virtually the full GPU VRAM to the inference process. For dedicated inference machines where the OS exists only to support the model runtime, a minimal Linux environment is the highest-efficiency option.

GPU driver overhead also contributes a fixed reservation — typically 200MB to 400MB on Nvidia hardware regardless of OS configuration. Factor this into tight VRAM budgets.

Calculate Your Exact VRAM Requirements

Manually accounting for parameter count, quantization overhead, KV cache demand at your target context length, and OS reservation is error-prone. Use the VRAM calculator to input your target model and quantization level and get exact VRAM requirements for your hardware configuration — including KV cache overhead at your target context window.

Frequently Asked Questions

Can I run Llama 3.3 70B on a single RTX 5090?

Yes, with significant constraints. At Q3_K_M quantization, Llama 3.3 70B consumes approximately 30GB, leaving 1–2GB of headroom on the RTX 5090’s 32GB. Context windows must remain short, no other GPU-bound processes can run simultaneously, and Q3_K_M introduces measurable quality degradation on complex reasoning tasks. For unconstrained 70B inference at acceptable precision, dual RTX 3090s (48GB combined) or Apple M4 Max with 48GB+ unified memory are the practical alternatives.

What is the difference between VRAM and unified memory for LLM inference?

Discrete VRAM (Nvidia, AMD GPUs) is physically isolated from system RAM. Moving data between them requires PCIe bus transfer — a significant bandwidth bottleneck. Unified memory (Apple Silicon) shares a single pool between CPU and GPU at full memory bandwidth, with no PCIe penalty. For LLM inference, unified memory eliminates the VRAM ceiling: models too large for discrete VRAM can run in unified memory without the catastrophic speed penalty of PCIe offloading. The trade-off is throughput — Apple Silicon’s unified memory bandwidth peaks around 800 GB/s versus the RTX 5090’s 1.79 TB/s, producing lower tokens-per-second on equivalent configurations.

How much VRAM does Gemma 4 12B require?

Gemma 4 12B requires approximately 7–8GB of VRAM at Q4_K_M quantization, fitting on a 12GB GPU with headroom for standard context windows. At Q8_0 precision, it requires approximately 13GB, making a 16GB GPU the minimum. Google designed the 12B variant specifically for 16GB hardware deployment — including laptop configurations — and its encoder-free multi-modal architecture means no additional VRAM is consumed by separate vision or audio encoder models.

Does Q4 quantization noticeably degrade output quality?

For models above 13B parameters, Q4_K_M quantization produces output quality measurably indistinguishable from FP16 on most instruction-following, coding, and reasoning benchmarks. Perplexity scores show degradation, but task performance is well-preserved. Quality begins to degrade noticeably at Q3 and below, particularly on multi-step reasoning, structured generation, and instruction-following with precise formatting requirements. For sub-7B models, Q4 degradation is more pronounced due to lower absolute model capacity absorbing the precision loss less gracefully.

Is a dual RTX 3090 setup better than a single RTX 5090 for local LLMs?

It depends on the use case. Dual RTX 3090s provide 48GB of combined VRAM, enabling Llama 3.3 70B at Q4_K_M without quality compromise — something a single RTX 5090 cannot achieve. However, multi-GPU inference introduces tensor parallelism overhead and PCIe inter-GPU communication latency, reducing tokens-per-second efficiency compared to a single-GPU configuration at the same effective VRAM. For capacity (70B at Q4), dual 3090s win. For throughput, simplicity, and single-model inference on models that fit within 32GB, the RTX 5090 is faster and more practical.

What is the minimum VRAM for a local AI coding assistant?

A functional local coding assistant requires a minimum of 8GB VRAM running a 7B–8B code-specialized model (Qwen 2.5 Coder 7B, CodeLlama 7B) at Q4_K_M. For meaningful context windows that accommodate multi-file codebases (typically 16k–32k tokens), 16GB running a 14B code model is the practical minimum. Agentic coding workflows involving large repository context benefit significantly from 24GB, where Qwen 2.5 Coder 32B or Gemma 4 31B can run with context headroom sufficient for real codebase navigation.