Self-hosted AI: what it is and when it's worth it

Self-hosted AI means running the model on hardware you control instead of renting it through someone else’s API. The same open models you can download and run, hosted on a server you own or a machine you rent, so the requests never touch a third party. In enterprise language it goes by “on-premise AI”, the same idea dressed for a procurement form.

For most businesses and most uses you don’t need to, and it’s worth saying that before anyone buys a graphics card. There’s a middle option that gives you almost all of the control for none of the running-a-server work, the two reasons people usually reach for self-hosting are narrower than they sound, and the cost saving is mostly a mirage. Here’s what it actually is, what it really costs, and how to tell if you’re one of the few it genuinely suits.

The three rungs

Self-hosting sits at one end of a ladder, and seeing the whole ladder is what makes the choice obvious. Picture three rungs. On the first you call a provider’s API: you send text to OpenAI, Anthropic or Google, they run the model on their computers, you get an answer back. Most convenient, least control. On the second you rent an open model from a host like OpenRouter, Groq or AWS Bedrock: someone else still runs the hardware, but you get an API and a contract that says they won’t train on what you send. On the third you run the model yourself, on a machine you own or a GPU you rent by the hour, and nothing leaves the box. That third rung is self-hosting.

The word trips people up because it covers two sizes of the same thing. Running a model on your own laptop for your own use is self-hosting for one person, and our guide to running an LLM on your own computer walks that path start to finish. “Self-hosted AI” as people usually search it means the bigger version: a model on a server that a team, or your own software, can reach, the way you’d host your own email or website instead of using Gmail. Same principle, a lot more moving parts. Underneath it all is one fact the AI sovereignty guide leans on: an open-weight model is a file, and a file can’t phone home, so whoever runs the computer it sits on controls where the data goes.

What running your own genuinely buys you

Strip out the things you can get more easily elsewhere, and self-hosting buys one real thing plus one double-edged one.

The real thing is that your data never touches another company at all. When the model runs on your own hardware, your prompts and files aren’t handed to anyone, so they can’t sit in someone else’s logs, be pulled into a training run, or surface in their next breach. That’s a genuinely stronger position than ticking a no-training box on a cloud tool, where you’re trusting a setting and a contract. Here there’s no one to trust, because the data has nowhere to go. That is the whole pitch of “on-premise AI”, and the next two sections are about who actually needs it.

The double-edged one is control of the version. The model you self-host is the one you picked, and no vendor can retire it, add a weekly limit, or reprice it out from under you, which the cloud tools do more often than people expect. The flip side is the part nobody mentions: you’re also frozen on it. The frontier models, GPT-5, Claude Opus, Gemini, are API-only and can’t be self-hosted, so a self-hoster is permanently a tier behind, running hardware sized to today’s open model, while API users get every new model the day it ships for no extra outlay. Owning your tap means owning an ageing tap.

The cost saving is a trap unless the GPU is busy

Most people reach for self-hosting to save money, and for most that’s exactly backwards. The reason is worth understanding, because it’s the whole game. A GPU is only cheap per answer when it’s kept busy, and a single business almost never has enough constant traffic to fill one. A provider does: it spreads one chip across thousands of customers, keeps it flat out, and passes the saving on. You can’t match that with one GPU serving your ten people, who between them leave it idle most of the day.

The numbers make it concrete. Rent an H100 and it bills around $2 to $3 an hour whether it’s flat out or sitting idle all weekend, and buying one outright runs to tens of thousands plus the power: left running around the clock it draws roughly 500 kilowatt-hours a month, sixty to a hundred and fifty dollars in electricity depending where you are, before anyone touches it. An API is the opposite: it charges nothing when you’re not using it. Practitioners who cost it out properly reckon the true bill lands several times the raw hardware price once you add the hours someone spends keeping it alive. And the thing you’re comparing against isn’t the pricey tool you’d assume: hosted open models have got so cheap that renting one often costs less than the electricity of running the same model yourself. In one worked example, serving 50 million tokens a day on your own GPUs came out at more than double the equivalent API bill. Self-hosting can undercut the expensive frontier models at high volume; it rarely undercuts a cheap hosted open model, and that’s the honest comparison for routine work.

A GPU that sits mostly idle costs more per answer than a premium API. You need to keep it running near half-full just to beat a mid-tier model, and most business workloads, spiky, business-hours, a few people chatting, never get there. Before you price hardware, work out how many hours a day it would genuinely be working. If the answer is “a couple”, the API wins and it isn’t close.

And you’d be running a server now

The other cost isn’t money, it’s the standing job you take on, and this is what catches teams out. A self-hosted model is infrastructure you now own: something to update, keep running, secure, and scale when more people use it. That’s not a one-off setup, it’s a permanent responsibility, and at a small company it’s usually nobody’s actual job, so it lands on whoever’s most technical until they leave and it quietly rots.

Three parts of that job are worth naming. Serving a team is harder than it looks: Ollama, the easy tool, comfortably handles a handful of people at once and starts falling over somewhere north of a dozen, so a whole team at once needs a purpose-built engine like vLLM that handles ten to twenty times the concurrent load, at the cost of being fiddlier to run. Security becomes yours: an inference server with no password, exposed to the internet, is a live door into your network, and researchers keep finding tens of thousands of exactly those left open, so the sane setup keeps it off the public internet entirely, behind a VPN or a login. And there’s no one to call when it breaks: one GPU going down at 2am takes your AI with it, with no support line and no failover unless you built one.

When it does genuinely pay

Two situations make self-hosting the right call, and both are narrower than the usual pitch.

The first is when data truly can’t touch any outside party, and the honest version of this rules out most of who thinks they qualify. Being in health, law or finance doesn’t force it: HHS explicitly allows health data in the cloud when there’s a signed agreement in place, and Azure OpenAI, AWS Bedrock and Google’s enterprise AI all sign business associate agreements and commit not to train on your data. So a compliant cloud clears the bar for the vast majority of regulated work, without any hardware. True on-premise is forced only by the narrow cases the cloud can’t cover: air-gapped or classified environments, export-controlled data, or a client contract that flatly forbids sending information to any third-party subprocessor at all. That last one is the realistic small-business trigger: not a law, but one big customer’s clause. Watch the trap here, because a rented GPU on RunPod or Vast is not on-premise, your data is still on someone else’s box, so the genuine “nothing leaves our building” case needs your own metal, not a cloud rental.

The second is high, steady, automated volume. If you’re running a model hard all day, feeding it a constant pipeline of work rather than a few people chatting, and an open model is good enough for that work, a busy GPU you control can beat paying a frontier API per token. The trigger is machines, not people. A handful of staff asking questions is cheaper and simpler on a business tier: ChatGPT Business runs about $20 a seat a month with a no-training contract and no server to run. It’s the relentless automated workload that tips the maths. And there’s a rung between renting a shared API and owning metal that suits a lot of these cases better than either: a dedicated managed endpoint from a provider like Fireworks, Baseten or Azure, a private instance they run for you, single-tenant, often able to scale to zero when idle, so you get isolation and steady capacity without becoming a sysadmin.

The genuine reasons show up in the regional fine print, too. A US-owned cloud provider can be compelled to hand over data under the US CLOUD Act no matter where the servers sit, and in 2025 a Microsoft France executive told a Senate inquiry under oath that he could not guarantee French data would never be passed to US authorities; for a government or defence contract that can’t tolerate that, self-hosted weights are the only setup that removes the reach. And when governments from Italy to Australia to US agencies restricted DeepSeek over its app routing data to China, the fix wasn’t to abandon the model but to run its open weights on your own hardware, where no packet leaves your network.

How to actually do it, and the realistic shape of it

The common stack has two parts and is less involved than it sounds: something to run the model and something to talk to it through. Most people pair Ollama to run the model with Open WebUI for the interface, a ChatGPT-style page in your browser with logins for different people, which is exactly what “self-hosted ChatGPT” usually means in practice. For a team hitting it at once, swap Ollama for vLLM underneath. Nearly all of these speak the same “OpenAI-compatible” language, so software written for ChatGPT’s API can usually be pointed at your own model by changing the web address it calls. If you want a genuine drop-in server, LocalAI is the all-in-one one; Jan is really a polished desktop app for one person that happens to expose a local API, not a team server, so don’t reach for it to serve a group.

Be realistic about what a small business actually stands up, because it’s rarely a data centre. For a small team it’s usually a single box, a Mac Studio with plenty of memory, or one machine with a good graphics card, running Ollama for internal use. That’s genuinely useful and it’s also a single point of failure with limited concurrency, fine for a few people querying their own documents, not a customer-facing service. To try the server version without buying anything, rent a GPU by the second from RunPod or Vast.ai, stand a model up for a few dollars an hour, and shut it down when you’re done, just remember that path isn’t on-premise for the compliance case above. And the sane pattern for most who go this route is hybrid, not either-or: run the routine, high-volume, sensitive work on your own model, and send the hard or occasional tasks to a frontier API, using each where it actually wins.

The one decision to make

Decide by two questions and it stops being a rabbit hole. First: does your data genuinely have to stay off every third party, the real air-gap or no-subprocessor kind, not the “we’re in healthcare” kind a compliant cloud already covers? If yes, self-host on your own hardware, and treat it as a compliance cost, not a saving. Second: are you running a model hard enough, all day and automated, that a per-token bill would seriously hurt? If yes, price a self-hosted or dedicated setup against that bill and see. If both answers are no, which they are for most businesses, use a hosted open model through a provider that signs a no-training contract: you keep the privacy and the freedom to switch models, and you skip running a server.

That middle rung is the quiet winner, and the sovereignty guide lays out how to use it, which models to try, and how to keep an exit from any one vendor. Self-hosting is the floor of that same ladder, worth understanding cold and worth reaching for exactly when those two questions say so, and not a step before. For the version that runs on the laptop in front of you, start with running an LLM on your own computer.