LLM Quantization and GGUF, Explained

What is quantization?

A large language model is, underneath, an enormous collection of numbers called weights — the values learned during training that determine how it responds. A model with, say, seven billion parameters has seven billion of these numbers. By default each one is stored as a 16-bit floating-point value, which is precise but takes two bytes of storage apiece. Multiply that out and even a "small" model is many gigabytes.

Quantization is the process of storing those numbers with lower precision — fewer bits each. Instead of a full 16-bit value, a weight might be rounded to an 8-bit, 5-bit or 4-bit representation. It is a bit like saving a photo as a slightly more compressed JPEG: the file gets much smaller, the picture looks almost the same, and only under close inspection do you notice the difference. The model still works; it just carries a touch less numerical detail in each weight.

Why quantization makes models smaller and faster

Because the weights are the bulk of a model, cutting the bits per weight cuts the file size almost proportionally. Roughly speaking, a 4-bit quant is around a quarter of the size of the original 16-bit weights. A model that would need a lot of memory at full precision can suddenly fit in the RAM of an ordinary laptop or the video memory of a mainstream graphics card.

Smaller also tends to mean faster, for a reason that surprises people: generating text is often limited less by raw arithmetic and more by memory bandwidth — how quickly the hardware can read all those weights for each token it produces. Fewer bits to move means less data to shuttle around, so the model can generate more words per second. On top of that, fitting the whole model in fast memory avoids the huge slowdown of spilling over onto disk. The practical upshot is that quantization is what makes capable local models run at all on everyday hardware, which is why it is central to any conversation about efficient on-device AI.

The levels: Q4, Q5, Q8 and the K-quants

Quantised models are labelled with a Q followed by a number — Q4, Q5, Q8 and so on. The number is, broadly, how many bits are used per weight, so Q4 is about 4-bit, Q5 about 5-bit and Q8 about 8-bit. Higher numbers preserve more of the original precision and quality, but produce a bigger file that needs more memory. Lower numbers are smaller and faster but shave off more detail.

Modern names carry extra letters, and they are worth decoding. The original llama.cpp schemes were simple round numbers like Q4_0, Q5_0 and Q8_0. The newer and now more common family is the K-quants, marked with a K — for example Q4_K_M. K-quants use a smarter "super-block" structure that spends bits more cleverly, giving better quality at a given size. The trailing letter is the variant: S, M and L stand for small, medium and large, meaning the quant spends slightly fewer or more bits on the layers that matter most. So Q4_K_M is a medium-sized 4-bit K-quant — the everyday sweet spot most people reach for first. The whole ladder, from Q2_K through the _S/_M/_L variants up to Q8_0, is enumerated in the llama.cpp quantize reference, which also tabulates the measured bits-per-weight and file size of each.

Bit-widths are approximate: K-quants mix precisions across layers, so the effective average differs slightly from the round number in the name.

What is GGUF, and why local models use it

GGUF (it succeeded an older format called GGML) is the file format used by llama.cpp — the open-source engine at the heart of most desktop local-AI tools — to store a ready-to-run model. Its defining feature is that it is self-contained: a single .gguf file holds the quantised weights plus the metadata a runtime needs to use them, including the model architecture, the tokenizer and default settings. The official GGUF specification in the ggml repository names single-file deployment as an explicit design goal and calls the format unambiguous precisely because it contains all the information needed to load a model. You download one file, point your app at it, and it runs. There is no separate config to wrangle and no assembly required.

That convenience is a big part of why GGUF dominates hobbyist and desktop local AI. It was designed for efficient inference — actually running a model — rather than for training, and it runs well on ordinary hardware, including pure-CPU machines with no dedicated GPU, while also using a GPU when one is available. It supports the whole range of quantization levels described above, so a single model is typically published as a menu of GGUF files at different quants, and you pick the one that fits your PC. Tools like LM Studio and Ollama are essentially friendly front-ends over this format: they help you find, download and chat with GGUF models. Our guide to running local LLMs on Windows walks through doing exactly that.

Other formats: safetensors, MLX and ONNX

GGUF is not the only way to store a model. It helps to know the neighbours, because you will meet them when browsing model repositories:

• safetensors is the standard format for distributing full-precision (or lightly quantised) model weights, especially on Hugging Face. It is a safe, fast way to store the raw tensors and is the usual starting point that other formats are converted from — the safetensors documentation bills it as a simple format for storing tensors safely, as opposed to pickle, while staying fast through zero-copy loading. It is common in Python and GPU-based workflows rather than lightweight desktop chat apps.
• MLX is Apple's framework and format for running models efficiently on Apple Silicon Macs — its own documentation describes it as a NumPy-like array framework for machine learning on Apple silicon, from Apple machine learning research. It is not a Windows format, but you will see MLX builds offered alongside GGUF for the same model.
• ONNX is an open, cross-platform format for machine-learning models, backed on Windows by the ONNX Runtime, which its documentation describes as a cross-platform machine-learning model accelerator with a flexible interface to hardware-specific libraries. It is often the route used to run models on a PC's AI hardware and NPUs, and it shows up in Microsoft's own on-device AI tooling.

For a typical Windows user who just wants to chat with a model locally, GGUF is the format you will use most, with the others appearing when you venture into development or platform-specific acceleration.

How to pick a quant for your hardware

The guiding principle is simple: choose the highest-quality quant that fits comfortably in your memory, with headroom to spare. "Fits" means the model file plus the working memory for your context (the conversation and any documents) both sit inside your RAM or, ideally, your GPU's video memory — spilling onto disk kills speed. Here is a practical way to decide:

1. Work out your budget. If you are running on the GPU, your limit is its video memory (VRAM); on the CPU, it is your system RAM. Leave a couple of gigabytes free for the operating system and context.
2. Start at Q4_K_M. For most models and most PCs this is the best all-round choice — a strong balance of quality, size and speed, and the same starting point LM Studio's own download guidance gives when it tells you to choose a 4-bit option or higher if your machine can handle it. If it fits with room to spare, you are done for casual use.
3. Trade up if you can. If you have spare memory, Q5_K_M, Q6_K or Q8_0 give a little more fidelity, which is most worth it for coding and careful reasoning.
4. Trade down only if you must. If Q4 does not fit, either pick a smaller model (a 7B instead of a 13B, say) at a decent quant, or drop to Q3_K_M on the same model and accept a more noticeable quality hit. As a rule, a smaller model at a higher quant often beats a bigger model crushed down to 2-bit.

The quality trade-offs, honestly

Does quantization make a model worse? Strictly, yes — you are throwing away precision, so it cannot improve quality. But the honest, practical answer is that in the common 4-bit to 8-bit range the loss is usually small, and often hard to notice for everyday tasks, while the savings in size and speed are large. That is why 4-bit quants are so popular: they capture most of a model's ability at roughly a quarter of the full-precision size.

The picture changes at the extremes. Push down to 2-bit or 3-bit and quality falls off more sharply — the model may become vaguer, make more mistakes, or lose the thread on complex tasks. The effects also show up unevenly: casual chat and simple drafting tolerate aggressive quantization well, whereas precise work like programming, mathematics and multi-step reasoning is more sensitive, and there a higher quant earns its extra megabytes. Vendor and community figures often quote only a few percent of measured quality lost at 4-bit, but treat any single number as a guide rather than a guarantee — the real impact depends on the model and your task, so it is worth trying two quants and comparing on your own work.

Putting it together

Quantization and GGUF are the two ideas that make local AI practical on an ordinary Windows PC. Quantization is the technique — storing a model's weights with fewer bits to make it smaller, faster and lighter on memory, at a modest and usually acceptable cost in precision. GGUF is the delivery format — a single, self-contained file that packages a quantised model so desktop tools can download and run it without fuss. Put them together and you can pull a capable model onto your machine, pick the quant that suits your hardware, and run it entirely offline.

If you want to see it in action, the natural next step is our guide to running local AI LLMs on Windows, which uses GGUF models in practice. For where these efficient local models are heading, read about Liquid AI and on-device models, weigh up local AI versus cloud AI, or start from the Windows AI hub for the wider picture.