How to Run Local AI Models (LLMs) on Windows

The short answer

You can run a real large language model — the same kind of AI that powers a chatbot — entirely on your own Windows PC, offline and free of any per-message cost. The easiest way is to install Ollama (a lightweight command-line and API runner) or LM Studio (a friendly desktop app with a chat window), download an open model, and start typing. How well it runs comes down mostly to one thing: how much memory your machine has for the model, especially GPU VRAM. This guide explains what to install, what hardware matters, and why you might want a local model at all.

Running models locally has gone from a hobbyist experiment to something genuinely practical on ordinary Windows hardware. Open models have got smaller and smarter, the tooling installs in a couple of clicks, and a mid-range gaming PC can now hold a capable assistant that answers instantly, works on a plane, and never sends a word to anyone else's server.

Why run an LLM locally at all?

Cloud AI services are convenient, so it is worth being clear about what going local actually buys you:

Privacy. Once the model is downloaded, your prompts and its replies stay on your device. Nothing is uploaded, so a local model is a strong fit for confidential documents, legal or medical notes, source code and anything you would rather not hand to a third party.
Offline use. After the one-time download, no internet connection is required. The model runs on a train, a plane or a site with no signal, exactly the same as at your desk.
No per-token cost. There is no subscription and no metered billing per message. You pay once, in electricity and the hardware you already own, then use it as much as you like.
Control and permanence. The model cannot be deprecated, rate-limited or changed under you. You keep the exact version you downloaded for as long as you want it.

The honest trade-offs: the very largest, most capable frontier models are still cloud-only, so a local model you can actually run will generally be smaller and less capable than the best hosted services. It will also run only as fast as your hardware allows. For many everyday tasks — drafting, summarizing, coding help, answering questions about your own files — a good local model is more than enough.

The hardware that matters: RAM, VRAM and NPUs

You do not need a supercomputer, but a little understanding of memory saves a lot of frustration. Three things decide what you can run and how fast.

System RAM

If you run a model on the CPU, it lives in ordinary system memory. More RAM lets you load a bigger model, but CPU inference is comparatively slow. As a general baseline, 16 GB of system RAM is a comfortable starting point for small models, and more headroom helps with larger ones and longer conversations.

GPU VRAM — the single biggest factor

This is the number that matters most. A dedicated graphics card runs models many times faster than a CPU, but only if the model fits in the card's own memory (VRAM). The amount of VRAM you have is the main ceiling on what you can run smoothly:

GPU VRAM	Comfortably runs (quantized)	Good for
6–8 GB	Small models around 7–8B parameters	Everyday chat, drafting, coding help
12–16 GB	Mid-size models in the low-teens of billions of parameters	Stronger reasoning, longer context
24 GB+	Larger models in the tens of billions of parameters	Higher quality, heavier local workloads

On Windows today the best-supported and most trouble-free path is an NVIDIA GPU, which both Ollama and LM Studio accelerate automatically once your graphics drivers are current. Modern AMD GPUs and the integrated graphics in newer chips are increasingly supported too. If a model is bigger than your VRAM, the tools can offload part of it to system RAM — it still runs, just more slowly.

NPUs and Copilot+ PCs

Newer Windows laptops marketed as Copilot+ PCs include a dedicated NPU (neural processing unit) — a chip built for AI work at low power. Microsoft's bar for that label is an NPU delivering 40+ TOPS (trillions of operations per second), paired with at least 16 GB of RAM. The NPU is what powers built-in Windows AI features efficiently and on battery.

A realistic note on NPUs and local chatbots

An NPU is excellent for the specific, optimized AI features Windows ships with, and support for running general chat models on the NPU is improving. For today's mainstream local-LLM tools, though, the GPU (its VRAM) and system RAM still do most of the heavy lifting. Do not assume that having a Copilot+ NPU alone will run large downloaded models quickly — check what your chosen tool actually accelerates on your hardware.

Quantization: how a big model fits a small PC

You will see model files labelled with things like Q4 or Q5. That is quantization — a compressed version that stores the model's numbers at lower precision, shrinking the file and the memory it needs, for a small drop in quality. A 4-bit quantized model is the usual starting point: it is the sweet spot between quality and fitting on modest hardware. Both tools below default to sensible quantized versions.

Approach 1: Ollama (lightweight, command line and API)

Ollama is a small, fast runner that installs as a native Windows app and exposes models through a simple command line and a local API. It is the favourite for people comfortable with a terminal, and for anyone who wants to plug a local model into other apps.

Download the Windows installer from the official Ollama site and run it. There are no extra dependencies to install.
Open Windows Terminal or PowerShell.
Pull and run a model in one command. Ollama downloads it the first time, then starts a chat:
```
ollama run llama3.1
```
Swap in another model name to try something else; the same command re-uses the download after the first time.
Type your prompt and press Enter. To leave the chat, type /bye — the same run-then-/bye flow given in the official Ollama quickstart.
List what you have downloaded, or remove a model to reclaim disk space:
```
ollama list
ollama rm llama3.1
```

Because Ollama also serves a local API (by default on your own machine only), other desktop apps and coding tools can talk to your local model without any of the conversation leaving the PC.

Approach 2: LM Studio (graphical, no command line)

Prefer to click rather than type commands? LM Studio is a desktop application with a built-in model browser and a chat window, available for Windows. It requires no Python and no terminal.

Download LM Studio for Windows from its official site and install it.
Open the app and use its search to browse models — LM Studio pulls listings from public model hubs and shows a fit hint telling you whether a given model and quantization will fit your machine's memory.
Click Download on a model that fits — a small 7–8B model in a 4-bit quantization is a good first pick.
Once it downloads, open a new chat, load the model, and start typing. Everything runs locally; nothing you type leaves your device.
Optional: LM Studio can also run a local server that speaks the common OpenAI-style API format, so your own scripts and apps can use the local model — its OpenAI compatibility API documentation lists the endpoints it answers.

Only download models from sources you trust

A model file is data, not a program you execute directly, but the ecosystem around it still deserves care. Stick to well-known models from reputable publishers on established model hubs, use the official installers for Ollama and LM Studio rather than random mirrors, and be wary of anything that asks you to disable security features. Large downloads also fill a disk fast — models can be several gigabytes each — so keep an eye on free space.

For developers: Windows ML, DirectML and Windows AI Foundry

If you want to build AI into your own Windows app rather than just chat with a model, Microsoft has a first-party stack worth knowing about, headed by Windows ML — Microsoft's local AI inferencing framework for Windows, which accelerates inference across NPUs, GPUs and CPUs.

Windows ML is the built-in, on-device inference runtime in Windows. Microsoft made it now generally available for developers targeting Windows 11, and it is designed to run models across CPUs, GPUs and NPUs from AMD, Intel, NVIDIA and Qualcomm, handling the hardware differences for you.
DirectML is Microsoft's hardware-accelerated machine-learning API that lets models take advantage of a wide range of GPUs through DirectX, and it underpins hardware acceleration in parts of this ecosystem.
Windows AI Foundry is the broader platform Microsoft has built around this — spanning model selection, optimization, fine-tuning and deployment, and exposing ready-made Windows AI APIs (for language and vision tasks) on top of Windows ML.

For most readers who simply want a private assistant on their desktop, Ollama or LM Studio is the direct route. The Windows ML and Windows AI Foundry stack is aimed at developers shipping AI features to other people.

Which should you choose?

A simple way to decide:

Want the simplest possible start with a graphical app? Use LM Studio. Its model browser and fit hints make it hard to pick something that will not run.
Comfortable in a terminal, or want to wire a model into other tools? Use Ollama for its clean command line and local API.
Building an app? Look at Windows ML / Windows AI Foundry and DirectML.

There is no wrong answer, and the two consumer tools coexist happily — many people keep both. Start small with a 7–8B quantized model, see how it feels on your hardware, and step up only if you need more.

For more current walkthroughs like this one, browse our How-To Guides. If you are weighing up hardware, our guides on Windows 11 Home vs Pro and speeding up Windows 11 performance are good companions, and the windows-now.com archive traces how home enthusiasts have been pushing their PCs for two decades.

Frequently asked

Do I need an expensive GPU to run a local LLM on Windows?

No — but it helps enormously. Small models will run on CPU and system RAM alone, just slowly. What makes a local model feel interactive is a GPU with enough VRAM to hold the whole model, and the biggest single factor in what you can run is how much VRAM that GPU has. A dedicated NVIDIA GPU is the best-supported path on Windows today; modern AMD GPUs and the integrated graphics on newer chips also work. If you have no dedicated GPU, stick to smaller models and expect slower responses.

How much RAM and VRAM do I need for a given model?

A rough rule of thumb: a model needs memory roughly equal to its file size on disk, plus headroom for context. A quantized 7-8B model is commonly a few gigabytes and fits comfortably on a GPU with 8 GB of VRAM or in system RAM. Larger models in the 13-14B range want more, and 30B-plus models need a lot of VRAM or will spill into slower system memory. Quantization (a 4-bit or 5-bit version of a model) shrinks the file and the memory it needs, at a small quality cost, which is why most people start there. Tools like LM Studio show a fit hint before you download.

Is running an LLM locally actually private?

Yes, that is the main reason to do it. Once the model file is downloaded, inference happens entirely on your machine — your prompts and the model's replies never leave the device and no account or internet connection is required to chat. That makes local models a strong fit for sensitive documents, confidential work and offline use. The one caveat is the initial download: you fetch the model itself over the internet, usually from a public model hub, before you go offline.

What is the difference between this and Copilot+ PC AI or Windows AI Foundry?

They solve different problems. Copilot+ PC features and the built-in Windows AI capabilities use your PC's NPU to run Microsoft's own optimized models for specific tasks such as live captions or image effects — you do not choose the model. Tools like Ollama and LM Studio let you download and run your own choice of open models, typically on the CPU or GPU. Windows AI Foundry and Windows ML are the developer platform underneath: APIs and an inference runtime for building AI features into your own apps across CPU, GPU and NPU. For simply chatting with a local model today, Ollama or LM Studio is the direct route.

More Windows help

Browse all how-to guides for more current, plain-English Windows walkthroughs, or dig into the windows-now.com archive for two decades of community Windows know-how.