How to run an LLM locally on your own computer

Running an LLM on your own computer means the AI model lives on your machine instead of a company’s server, so everything you type stays with you. No account, no subscription, no internet connection required, and no one on the other end logging your questions or training on them. The same kind of model behind ChatGPT, downloaded once and run on the laptop you already own.

It’s become genuinely easy, and it’s worth being clear upfront about why you’d bother. You do this for privacy, for working offline, for owning the tool outright, and for pointing it at your own files without them leaving the room. You don’t do it to save money or to get something smarter than ChatGPT, because a model that runs on your machine trails the big cloud tools, and if you had to buy hardware to run a good one, renting the same model online would usually cost less. Go in for the first list and it delivers. Go in expecting the second and you’ll be disappointed.

What running one locally actually gets you

The whole appeal comes down to nothing leaving your machine. When the model runs on your own computer, your prompts and any files you feed it never travel to a server, which means they can’t be leaked in someone else’s breach, subpoenaed from a vendor, or fed into a training run. That’s a different kind of private from ticking a no-training box on a cloud tool, where you’re trusting a company’s settings. Here there’s no one to trust, because the data has nowhere to go. For anything you’d hesitate to paste into a chatbot, client records, financials, unpublished work, that’s the point of the exercise.

That’s also why the businesses that genuinely need this are the ones holding data that legally can’t wander: a law firm, a clinic, an accountant. It’s a sharper reason than most people realise. Under GDPR, processing a prompt on a server outside the EU is itself a data transfer, and data sitting in a European data centre run by a US company is still reachable under American law, so “we use the EU region” isn’t full privacy the way a machine on your own desk is. It’s also the clean answer to the DeepSeek problem: its app sends your text to servers in China and a wave of governments banned it from official devices in 2025, but the open weights run on your own hardware and phone home to nobody, so you can have the capability without the data going anywhere.

Three practical things come with it. It works offline, so a plane, a dead connection or a locked-down office is no obstacle. It’s free to use, with no subscription and no per-question meter, so you can throw an entire book at it or ask ten thousand questions and the bill doesn’t move. And it’s yours, so nobody can retire the model you rely on, change the price, or add a weekly limit, which the cloud tools do more often than people expect. For a worked example pointed at one job, our guide to private transcription on your own laptop runs speech to text with nothing leaving the machine.

The two apps that make it easy

Two free apps turned this from a developer project into a ten-minute one: LM Studio and Ollama. You install one, it handles downloading and running the models, and you get a chat window. Underneath they’re both built on the same open-source foundation, llama.cpp, though Ollama has moved some newer models onto its own engine, so in practice the models behave and perform almost identically either way. Which one to start with is its own question, and our LM Studio vs Ollama guide settles it, along with the gotchas each one hides.

LM Studio is the one to start with if you want a normal app: a search box, a download button, a chat window, no command line anywhere. It became free to use, including at work, in July 2025 when it dropped its separate commercial licence, and it runs on Windows, Linux and Apple Silicon Macs (not the older Intel ones). Handily, it tells you which models will fit your computer before you download them, which takes the guesswork out of the next two sections.

Ollama is the one to reach for if you’re comfortable typing a command, or if you want other programs to use the model, which is what it’s really built for: it runs quietly in the background and lets your other tools talk to it, and it now has a simple desktop app alongside the command line. On safety, the thing to know is that Ollama listens only to your own machine by default, with no password, so your prompts stay on the device and nothing external can reach it. The alarming security stories are about people who deliberately exposed it to the internet, which researchers keep finding thousands of, so the rule is just: leave the defaults alone, keep it updated, and don’t open the port. If you mainly want to chat with your own documents rather than tinker, AnythingLLM or Msty are friendlier for that, which is the next section but one.

Get a model running in about ten minutes

Here’s the fastest path from nothing to a working local AI, using LM Studio because it needs no command line. Three steps:

1. Download LM Studio from lmstudio.ai and install it like any other app.
2. Inside the app, search for a model and download one it marks as a good fit for your machine. Start with something small, around 8 billion parameters, so your first go is quick.
3. Open a chat and start typing. Turn your Wi-Fi off first if you want proof it’s all happening on your computer.

That’s it. The first answer takes a moment as the model loads into memory, then it behaves like any chat tool, except it’s yours. Ollama gets you to the same place with one line in the terminal (ollama run qwen3 downloads a current model and drops you into a chat), or through its desktop app. On privacy, the app genuinely runs offline once a model is downloaded: the only time either app touches the network is to fetch a model or check for an update, and you can firewall the app to block even that.

Which model should you run

For a first run, Qwen3 is the safe default, and let the app tell you which sizes fit. Alibaba’s Qwen has quietly become the general-purpose local model most people reach for, taking the crown Meta’s Llama held for years, Llama 4 landed to a flat reception and Llama has slid down to a niche pick rather than the obvious start. Google’s Gemma is the other strong all-rounder and does a remarkable amount for its size. From there it’s worth knowing the standouts, with one caveat: these models get replaced every few months, so trust the list inside your app over any version number here.

The genuinely new thing is that OpenAI, the maker of ChatGPT, now gives away a model you can run at home. Its gpt-oss models arrived in August 2025 under an open licence, and the smaller one, gpt-oss-20b, runs on a machine with 16GB of memory while landing near OpenAI’s own o3-mini on common tests. If your machine has the room, it’s the strongest reasoning option you can run locally, and it’s free. Beyond that, a DeepSeek distill is the pick for logic and reasoning (a smaller model taught to imitate DeepSeek’s big reasoning model, since the real one is a 400GB server job, not a laptop one), Mistral leans towards code, and Microsoft’s Phi is the one for tight hardware. Don’t agonise: download two, ask them the same real question, keep the one you prefer.

Point it at your own documents

This is the use case where running locally most clearly pays off. Instead of just chatting, you can hand a model a folder of your own files, contracts, financials, a policy manual, and ask questions across the lot, with none of it leaving your computer. AnythingLLM is the easiest way in for a non-technical owner: point it at your documents, and it answers with the filename and passage it drew from, so you can check the source. Msty does something similar in a polished, no-terminal app. Both run the same local models underneath.

Know its limits before you trust it, because they’re specific. This kind of document search matches on similar wording, not on meaning, so the questions it quietly fumbles are the ones a contracts or accounts person asks first: what a document does not cover, an exact clause number like “section 4.2.1”, or anything about a threshold like “payments over $10,000”. Treat its answers as a fast way to find the right page, then read the page. And if answers come back thin, the usual culprit is the default memory setting being too small to hold the passage it found; bumping the context length up fixes most of it.

What your computer can handle

Memory is the one spec that decides what you can run. The rule of thumb is roughly 0.6GB of memory for every billion parameters, plus about a third on top for headroom. That’s because the app hands you a compressed version of the model, a process called quantisation that shrinks the file to about a quarter of full size for a barely noticeable drop in quality, as long as you take the standard “Q4” build the apps default to. Go smaller than that and quality falls off a cliff. One rule pros live by: a bigger model at that standard compression almost always beats a smaller model at higher quality, as long as both fit.

That translates into a clean ladder:

Your memory (RAM)	Comfortable model size	What that feels like
8GB	up to ~7-8 billion, tightly	A capable assistant for everyday tasks
16GB	up to ~14 billion, or gpt-oss-20b	Close to the free cloud tools
32GB	up to ~32 billion	Noticeably sharper, still comfortable
64GB+	up to ~70 billion	The serious end for a home machine

Two things save disappointment. First, any Apple Silicon Mac (the M1 from 2020 onwards) punches above its weight, because its memory is shared between the main chip and the graphics chip, so the whole lot is available to the model; on a Windows or Linux PC the fast path is a dedicated graphics card with its own memory, and the processor alone works but slowly. Second, speed is worth setting expectations on, because “slow” ranges from fine to unusable. You read at roughly 7 to 10 words a second; a decent Apple Silicon Mac or any dedicated graphics card comfortably beats that, so answers stream faster than you read, while a laptop with no graphics card crawls at a few words a second, fine for “ask it and come back”, painful for a live back-and-forth. And long conversations quietly eat memory as they grow, so a model that loaded fine can slow or stall deep into a big chat.

Where it loses to the cloud, and whether to buy a machine for it

A local model is a rung or two below the frontier cloud tools, and being honest about the gap saves you expecting the wrong thing. On hard reasoning, serious coding and long multi-step tasks, ChatGPT, Claude and Gemini are still clearly ahead, and how they stack up is our three-way comparison. Small local models are also shaky at the agentic, tool-using work where a model has to call other software and chain steps, and they lose the thread on very long documents sooner than the cloud apps. For summarising, drafting, rewriting, brainstorming and questioning your own files, a good local model is genuinely up to it. For your hardest work, keep a cloud tool.

On buying hardware for it, the honest answer for almost everyone is don’t. You can rent these same open models online for a few dollars per million words, so a box only pays off if you’ve got a privacy rule that forbids the cloud, steady high volume, or someone in-house who enjoys tinkering. And the “cheap local AI box” story has aged: memory prices spiked hard through 2025 and 2026 as AI data centres soaked up supply, so the sub-$2,000 mini-PCs with 128GB that made the rounds now cost more, and the specialist desktop boxes like NVIDIA’s DGX Spark (now around $4,700) are really developer kits, slower for everyday chat than a Mac. If you genuinely need a dedicated machine, a Mac with plenty of memory is the sensible buy, because it’s fast for this and it’s a real computer you’d use anyway. Where owning your AI setup is worth the trade for a business, and how it protects you when a vendor changes the terms, is the whole of our AI sovereignty guide.

What to actually do

Install LM Studio this afternoon, download a Qwen3 model around 8 billion parameters, and ask it something you’d normally ask ChatGPT. Ten minutes in you’ll have a private AI running with the Wi-Fi off, and a real feel for what it does well and where it taps out. If it sticks, match a bigger model to your memory using the ladder, point it at a folder of your own documents with AnythingLLM if that’s your reason for being here, and pick up Ollama the day you want your other software to use the model. If it graduates from your laptop to a box the whole team reaches, that’s the server version, and when it’s actually worth running one is the self-hosted AI guide. Treat it as the private, offline, free companion to your cloud tools rather than a replacement, and you’ve added a capability most people don’t know they already own the machine for. It’s one lane of the wider small-business AI shortlist, which covers the tool to reach for in every other job.