How Much VRAM Do You Need for Local AI?

This guide explains why memory matters so much for local AI, gives ballpark figures by model size, and untangles the difference between VRAM, RAM and unified memory. All the numbers here are approximate and depend heavily on quantization, context length and the specific runtime you use — treat them as sizing guidance, not exact specifications.

Why memory is the bottleneck

A large language model is, at its core, a huge grid of numbers called weights. To generate even a single token of text, the computer has to read essentially the entire set of weights out of memory. Do that for every token, hundreds of times per reply, and you see why the job is punishing on memory rather than on raw arithmetic. This is why people who run models locally talk constantly about memory bandwidth — how fast weights can be streamed to the processor — and why fitting a model into the fastest available memory matters more than almost anything else.

There are two separate memory questions. The first is capacity: is there simply enough room to hold the model at all? If a model doesn't fit, it either won't load or it spills onto slower memory and crawls. The second is bandwidth: how quickly can those weights be read? A discrete graphics card wins on both counts for the models it can hold, which is why a GPU with enough VRAM is the gold standard for local inference.

The rough formula: parameters and quantization

Two numbers set a model's memory footprint: how many parameters it has (7B means seven billion, 70B means seventy billion) and the precision each parameter is stored at. Precision is where quantization comes in — the process of squeezing each weight into fewer bits to save space.

• FP16 / 16-bit stores each parameter in two bytes, so the weights alone take roughly 2 GB per billion parameters. A 7B model is about 14 GB of weights; a 70B model about 140 GB.
• 8-bit (INT8) roughly halves that to about 1 GB per billion.
• 4-bit (Q4 and similar) — the most common local format — lands near 0.5 GB per billion parameters for the weights, give or take, depending on the exact quantization scheme.

On top of the weights you need headroom for the KV cache (the running memory of your conversation, which grows with context length) plus general runtime overhead. A common practical allowance is to add roughly 10–20% on top of the weight size, more if you use very long contexts. So a 4-bit 7B model with a few thousand tokens of context tends to want somewhere in the 6–8 GB range in total rather than the bare ~3.5 GB the weights suggest.

Memory by model size (a rough guide)

The table below gives ballpark total memory for popular model sizes at 4-bit quantization, including weights plus a modest allowance for context and overhead. The FP16 column shows how much the same model needs at full 16-bit precision, to illustrate just how much quantization saves. All figures are approximate and depend on quantization and context length.

The pattern is clear: model size is the dominant factor, and quantization roughly cuts the requirement in half at each step down from 16-bit. That's why the standard advice for a memory-limited PC is to run the largest model you can fit at 4-bit rather than a smaller model at full precision — the extra parameters usually help more than the extra precision does.

GPU VRAM vs system RAM vs unified memory

Not all memory is equal, and the kind you have shapes what's realistic.

• GPU VRAM is the dedicated memory soldered onto a discrete graphics card. It's the fastest option by a wide margin and the one runtimes prefer. Its downside is that you get a fixed amount — 8, 12, 16 or 24 GB on typical consumer cards — and you can't add more.
• System RAM is your PC's main memory, used by the CPU. There's usually far more of it, and it's often upgradeable, but it has much lower bandwidth than discrete VRAM, so running a model here is slower.
• Unified memory is a single pool shared by CPU, GPU and, on newer machines, the NPU — the arrangement on Apple Silicon Macs and on many Windows laptops with integrated graphics. It lets you dedicate a large chunk to a model without a separate graphics card, but its bandwidth is typically well below that of high-end discrete VRAM, so a big unified-memory figure doesn't automatically mean fast inference.

This is the crucial nuance behind AI laptops: a machine can advertise a large shared memory pool yet still generate text slowly, because capacity and bandwidth are different things. For more on how the processors in these machines divide the work, see our explainer on GPU vs NPU for AI and what an NPU is.

CPU-only and offloading

You don't strictly need a GPU at all. Popular local runtimes — Ollama, llama.cpp and LM Studio — can run a model entirely on the CPU using ordinary system RAM. Ollama makes the split visible: as the Ollama FAQ explains, ollama ps reports "100% GPU" when a model is loaded entirely into the GPU, "100% CPU" when it sits entirely in system memory, and a percentage split when it is spread across both. This is a genuinely useful path for trying models bigger than your GPU could hold: an ordinary desktop with 32 GB of RAM can load a 13B model that would never fit in an 8 GB graphics card. The catch is speed. Because system RAM has far less bandwidth than dedicated VRAM, CPU-only inference is markedly slower, and larger models can drop to a token or two per second — usable for a considered task, frustrating for quick back-and-forth.

The middle ground is offloading, where a runtime keeps as many of the model's layers as fit in VRAM and puts the rest in system RAM. This lets you run a model that's slightly too big for your card at a speed between "all GPU" and "all CPU." It's a practical way to stretch an 8 or 12 GB GPU onto a somewhat larger model, and most tools handle the split automatically once you tell them how much VRAM to use. LM Studio, for one, documents that it will automatically reduce the GPU offload size to fit the weights in dedicated GPU memory and put the rest in system RAM — and reports that this beats falling back on shared GPU memory.

Examples by GPU tier

To make the table concrete, here's roughly what different amounts of memory unlock, keeping to 4-bit models. These are general guides, not promises — driver, runtime and context all move the line.

• 8 GB (entry discrete GPU or capable integrated graphics): comfortable for 7–8B models, the sweet spot for a first local assistant. Larger models need offloading or a drop in quantization.
• 12 GB: handles 13–14B models nicely and gives 7–8B models generous context headroom.
• 16 GB: a comfortable all-rounder — larger 13–14B models with long context, or smaller models running fast with room to spare.
• 24 GB (high-end consumer card): opens up 30–34B-class models and lets you run mid-size models with very long contexts.
• 48 GB and up (dual cards, workstation GPUs, or large unified memory): the territory where 70B models become practical to hold entirely in fast memory.

How much do you actually need?

For most people getting started, a machine that can run a good 7–8B model well is enough to be genuinely useful for drafting, summarising and coding help — and that means roughly 8 GB of VRAM, or a system with 16 GB or more of RAM if you're going CPU-only — which lines up with LM Studio's own system requirements, which recommend at least 16 GB of RAM and at least 4 GB of dedicated VRAM. Stepping up to 12–16 GB of VRAM is the difference-maker if you want the noticeably sharper 13–14B models. Chasing 70B models locally is a real commitment in hardware and cost, and for many people cloud access to a large model is the more sensible route than buying a workstation.

If you're choosing which model to install rather than which hardware to buy, our guide to the best local LLM for your PC walks through matching a model to your memory, and our overview of running local AI LLMs on Windows covers the tools that make it painless. You can also browse everything we've written on the subject in the Windows AI hub.