Hardly a week goes by without a mid-sized company asking me the same question: should we run our own language model, Llama, Mistral or Qwen on our own hardware, or is the API from the big providers the smarter play? One camp dreams of "our own ChatGPT in the basement", the other dismisses self-hosting as expensive nonsense. Both are wrong. The answer is a calculation with three variables: volume, confidentiality and utilization. So let's actually do the math.
Why this question sounds different in 2026
Three years ago the answer was easy. Open models were nice toys, and serious work required an API. That is no longer true. Meta's Llama 4 generation includes Scout, a model with 109 billion parameters (17 billion active) that runs on a single H100 GPU. The French company Mistral released its entire Mistral 3 family in December 2025 under the permissive Apache 2.0 license, from a compact 3B model up to the flagship Mistral Large 3 with 675 billion parameters. And Alibaba's Qwen3 lineup of open-weight models regularly lands near the top of coding and math benchmarks.
More on this topic: AI in the Founder's Working Day – background, practice and every article in one place.
Translated into business terms: for most tasks in a mid-sized company, open models are simply good enough. Summarizing documents, classifying emails, pre-structuring quotes, making internal knowledge searchable. None of that needs a frontier model. So the question is no longer whether an open model can do what you need. The question is whether running it yourself actually pays off.
What a self-hosted LLM really costs
This is where the numbers usually get massaged. Hardware is only the beginning. A single data-center GPU rents in the cloud for roughly one and a half to two dollars per hour depending on provider and commitment, and a dedicated multi-GPU server quickly reaches five or six figures. Add power, cooling and redundancy on top.
The biggest line item never shows up on a hardware invoice: people. Someone has to set up the inference server, maintain drivers and model versions, tune batch sizes, build monitoring and get up at night when the thing falls over. That is not a project, it is permanent operations. If nobody on your team can do this and wants to do this, a self-hosted model mostly buys you a new dependency, either on a single colleague or on an outside contractor.
Utilization decides, not ideology
The real deciding factor is almost embarrassingly simple and still gets overlooked: what percentage of the time is your GPU actually working? A cost analysis by DigitalOcean ran the numbers cleanly. Self-hosting only beats pay-per-token APIs once your GPU is busy roughly 20 to 50 percent of the time, depending on how fast responses need to be. At 10 percent utilization, your own hardware costs more than twice as much as the API. At 5 percent, more than four times as much. Flip it around and a typical answer on a fully saturated GPU of your own costs a fraction of the API price.
My rule of thumb: a chatbot answering a few hundred requests a day belongs on the API. A pipeline grinding through millions of documents or tickets around the clock is a genuine candidate for your own model. In between lies a gray zone where you have to measure honestly instead of guessing. Latency matters too. If you need the model on the factory floor or at the edge, without a reliable internet connection, the decision is made for you.
The data protection argument and the GDPR
"We cannot send our data to the cloud" is the most common argument for self-hosting, and there is a true core to it. According to the Bitkom study on artificial intelligence in Germany, more than a third of companies now actively use AI, yet about half name legal uncertainty and strict data protection requirements as a central hurdle. The concern is real. The conclusion people draw from it is often wrong.
Because GDPR compliance is achievable both ways. The major API providers offer data processing agreements, EU data centers and contractual commitments not to train on your data. For most use cases that is sufficient, provided you review and document it properly. Your own model wins where the data is genuinely critical: patient records, engineering secrets, M&A documents, anything that contractually or legally must never leave the building. And it wins on sovereignty. Your prices, your model version and your availability no longer depend on the roadmap of a vendor in San Francisco. The fact that Mistral, a European company, releases its top models under an open license makes that path strategically even more attractive.
When each option pays off: my decision guide
Choose the API if you are just getting started, your volume is below a few million tokens per day, your use cases are still shifting, and no data is involved that legally must stay in-house. You stay flexible and only pay for what you use.
Choose your own model if you have permanently high, predictable volume (the pipeline that runs 24/7), if data absolutely must remain on-premise, or if you need to control latency and availability yourself, for example in production environments.
Choose a fine-tuned small model if you have one narrow task at massive scale, such as classification or extraction. A specialized 8B model often beats a generalist frontier model here, at a fraction of the cost.
Choose neither if you do not yet know your use case. Process first, then tooling. I wrote about why so many projects fail at exactly this sequencing in Why AI agent projects fail.
My conclusion: think hybrid, not dogmatic
In practice, nearly every serious company I know ends up with a hybrid architecture. A router decides per request: bulk work like classification and extraction runs on a cheap open model, locally or with an EU host, while the hard cases, complex reasoning and sensitive customer communication, go to the best available API. That gives you cost control and data sovereignty without giving up top-tier quality.
One more thing. The model is ultimately the smallest part of the value chain. The value appears when AI is wired into your processes, with access to your systems and clear ownership. What an organization looks like when agents take on real work is something I described in The autonomous organization. Whether Llama, Mistral, Qwen or an API runs underneath is one line in a config file. The architecture above it is what actually changes your company.
Frequently asked questions
At what volume does a self-hosted LLM pay off?
As a rule of thumb, only with permanently high hardware utilization, roughly 20 to 50 percent GPU usage around the clock, which means millions of tokens per day. For sporadic usage the API is almost always cheaper, because you only pay for what you actually consume.
Can LLM APIs be used in a GDPR-compliant way?
Yes, in principle. Major providers offer data processing agreements, EU data centers and commit not to train on business customer data. What matters is a proper legal review. Only for highly critical data that must never leave the building is a local model the safer choice.
Which open model is right for a mid-sized company?
For most tasks, mid-sized models like Qwen3 in its 8B to 32B variants or the smaller Mistral 3 models are enough, both freely usable under the Apache 2.0 license. Llama 4 Scout is interesting for very long documents. More important than the model choice is defining the use case cleanly first.
What is a hybrid LLM architecture?
A setup where a router sends each request to the right model: simple bulk tasks go to a cheap open model, local or EU-hosted, while complex tasks go to a powerful API. That combines cost control and data protection with top quality.
Warmly,
Dennis Weidner
Note: AI tools supported me in writing this article, and some images were edited with AI. I stand behind its content and every statement with my name.
