Running your own AI is usually a worse deal than it looks

There’s a lovely idea at the heart of self-hosted AI. An open model is just a file. You download it, run it on a machine you control, and nothing you type ever leaves the building. No one else’s logs, no training run, no breach at someone else’s company exposing your client notes.

That part is true, and for a few businesses it’s exactly what they need. For most it’s a lot more work than it’s worth, and the money it’s supposed to save doesn’t show up. Here’s why, and how to tell which camp you’re in.

Three ways to use a model

Picture three rungs. On the first, you send your text to OpenAI, Anthropic or Google, they run the model on their computers, and you get an answer back. Most convenient, least control. On the second, you rent an open model from a host like OpenRouter, Groq or AWS Bedrock. Someone else still runs the hardware, but you get a contract saying they won’t train on what you send. On the third, you run it yourself, on a machine you own or a chip you rent by the hour, and nothing leaves the box.

“Self-hosted” usually means that third rung at team size: a model on a server your staff or your own software can reach, the way you might run your own email instead of using Gmail. It’s the same idea as running a model on your laptop, with a lot more moving parts.

What you actually get

Strip out what you can get more easily elsewhere, and self-hosting buys you one real thing and one double-edged one.

The real thing is that your data never touches another company at all. On a cloud tool you’re trusting a setting and a contract. On your own hardware there’s no one to trust, because the data has nowhere to go.

The double-edged one is the version. The model you run is the one you picked, and no vendor can retire it, add a weekly limit or reprice it out from under you. But you’re also stuck with it. The best models, GPT-5, Claude Opus and Gemini, only run on their makers’ own servers and can’t be self-hosted, so a self-hoster is permanently a step behind, running hardware sized for today’s open model while everyone else gets each new model the day it ships.

The saving is a mirage unless the chip is busy

Most people reach for self-hosting to save money, and for most it works the other way. The reason is the whole game. The graphics chip a model runs on is only cheap per answer when it’s kept busy, and one business almost never has enough steady work to keep it busy. A provider does. It spreads one chip across thousands of customers, runs it flat out and passes the saving on. Your ten staff, who between them leave it idle most of the day, can’t compete with that.

The numbers make it concrete. A rented H100, the workhorse chip, bills around $2 to $3 an hour whether it’s flat out or idle all weekend. Buying one runs to tens of thousands, and left on around the clock it uses roughly 500 kilowatt-hours a month, sixty to a hundred and fifty dollars of electricity depending on where you live. A pay-per-use service charges nothing while you’re not using it. And the fair comparison isn’t the expensive frontier model you’d assume. Hosted open models have got so cheap that renting one often costs less than the power bill for running the same model yourself. In one worked example, serving 50 million tokens a day (tokens are the chunks of text a model reads and writes) on your own chips came out at more than double the equivalent pay-per-use bill.

Count the hours it would really work. A chip that sits mostly idle costs more per answer than a premium service. You need to keep it running about half-full just to beat a mid-range model, and most business workloads, spiky, business hours, a few people asking questions, never get there. Before you price any hardware, work out how many hours a day it would actually be working. If the answer is “a couple”, renting wins, and it isn’t close.

Congratulations, you run a server now

The other cost isn’t money. It’s the job you take on. A self-hosted model is infrastructure you now own: something to update, secure, keep running and scale when more people use it. At a small company that’s usually nobody’s actual job, so it lands on whoever’s most technical, until they leave and it slowly rots.

A few parts of that job are worth naming. Serving a whole team is harder than it looks. Ollama, the easy tool, handles a handful of people at once and starts to struggle somewhere past a dozen, so a busy team needs a purpose-built engine like vLLM, which handles ten to twenty times the load and is fiddlier to run. Security becomes yours, and researchers keep finding tens of thousands of these servers left open on the internet with no password. And there’s no one to call. If the one chip falls over at 2am, your AI goes with it.

When it does pay

Two situations make self-hosting the right call, and both are narrower than the usual pitch.

The first is when data truly can’t touch any outside party, and fewer businesses qualify than think they do. Being in health, law or finance doesn’t force it. The US health regulator explicitly allows health data in the cloud when there’s a signed agreement, and Azure OpenAI, AWS Bedrock and Google’s enterprise AI all sign those agreements and promise not to train on your data. What does force it is narrower: air-gapped or classified work, export-controlled data, or, the realistic one for a small business, a big client whose contract forbids sending their information to any outside company at all. And note that a chip rented from RunPod isn’t on-premise. Your data is still on someone else’s machine.

The second is high, steady, automated volume. If a machine is feeding a model a constant stream of work all day, and an open model is good enough for that work, a busy chip you control can beat paying per use. A few staff asking questions is cheaper and simpler on a business plan: ChatGPT Business is about $20 a seat a month, with a no-training contract and no server. There’s also a middle option worth knowing about, a private endpoint from a provider like Fireworks, Baseten or Azure, where they run a dedicated copy of the model just for you.

The CLOUD Act problem. There’s one more real reason, and it’s a bit unsettling. A US-owned cloud provider can be compelled to hand over data under the US CLOUD Act, wherever its servers physically sit. In 2025 a Microsoft France executive told a Senate inquiry under oath that he couldn’t guarantee French data would never be passed to US authorities. For work that can’t tolerate that, running the model yourself is the only setup that takes it off the table.

If you’re going to do it anyway

The usual setup has two parts: something to run the model and something to talk to it through. Most people pair Ollama with Open WebUI, which gives you a private, ChatGPT-style page in the browser with logins for your team. That’s what “self-hosted ChatGPT” usually means in practice. For a team hitting it at once, swap Ollama for vLLM underneath.

Be realistic about what a small business actually stands up. It’s rarely a data centre. It’s usually one box, a Mac Studio with plenty of memory or a machine with a good graphics card, for a few people querying their own documents. Useful, and also a single point of failure. To try it without buying anything, rent a chip by the second from RunPod, stand a model up for a few dollars, and switch it off when you’re done. For most who go down this road, the sensible pattern is both: routine, sensitive work on your own model, and the hard or occasional jobs sent to a frontier model.

Two questions

It stops being a rabbit hole once you ask two questions. Does your data truly have to stay off every outside party, the real air-gap kind, not the “we’re in healthcare” kind a compliant cloud already covers? And is a machine running a model hard enough, all day, that paying per use would really hurt? If either answer is yes, price it up properly. If both are no, which they are for most businesses, use a hosted open model through a provider that signs a no-training contract. You keep the privacy and the freedom to switch, and nobody has to babysit a server.

That middle option is the quiet winner, and the AI sovereignty piece covers how to use it.