On-premise LLM in 2026: the hardware, the power and what it really costs
What it takes to run an LLM on your own hardware: GPU memory per model, 6.5 kW of power and heat, the software stack, data protection and the costs.

Running a large language model on hardware you control means choosing where the GPUs sit, how much memory they need and who keeps them running. This guide covers all three, with numbers from the 8-GPU server we run in a data centre near Sofia.
| If you... | Setup that usually fits | Why | Cost, excluding VAT |
|---|---|---|---|
| want to test whether an open model does the job, alone or in a small team, with a model up to 32B | A workstation or desktop GPU in the office | No data centre needed, and you find out early whether you need anything bigger | The GPU you have, or one card |
| need a 70B to 235B open model for a team or a product, 24/7, with the data in the EU and no data centre of your own | A dedicated GPU server in an EU data centre, such as our 8 × RTX PRO 6000 | 768 GB of GPU memory, no other tenants, full root access, power and cooling handled | From $1.80 per GPU-hour, about $10,510 a month for the whole server, power, cooling, internet and rack included |
| must own the hardware, will use it fully for three years or more and have staff to run it | Buy a server and colocate it | Full ownership, your own rack, your own rules | €168,524 to €241,261 for the server (October 2026), about €6,600 to €9,300 a month over three years with power and rack space |
| have low or uneven volume, or need the strongest closed models | An API | Nothing to run, you pay per token | Per token |
The server prices and the monthly cost of owning one come from our AI server cost guide, where every figure has a source and a date.
What "on-premise" means for an LLM: control, not the building
For an LLM, "on-premise" is mostly about control: who can reach the model, where the prompts and documents are stored, and who has physical and administrative access to the machine. There are three common places for the hardware:
- Your office. You hold the keys to everything, and you also carry the power, cooling, noise and repairs.
- Your own server in a colocation rack. You own the machine and the data centre provides power, cooling and the network. You still handle the hardware.
- A dedicated server rented in a data centre. The machine is reserved for you alone, with root access, and the provider owns and repairs the hardware.
In all three the model runs on GPUs that serve nobody else, and no prompt goes to a shared public API. The difference is who carries the hardware. Our local LLM vs private LLM server guide covers the step from a laptop to a server in more detail.
Sizing: the model sets the minimum, the users set the rest
Two things decide how much GPU memory you need.
The model. At 4-bit quantization the Ollama library lists Llama 3.3 70B at 43 GB, gpt-oss-120b at 65 GB and Qwen3-235B-A22B at 142 GB (checked on 13 October 2026). The first two fit on one 96 GB card. The 235B model needs at least two.
The users. Every open conversation keeps a key-value cache next to the model. For Llama 3.3 70B that is about 320 KB per token at 16-bit, so one conversation of 8,000 tokens adds about 2.5 GB. Twenty people at once can need more memory for the cache than for the model itself. Serving engines such as vLLM use that memory more efficiently, but the cache is still what turns a model that "fits" into one that runs out of memory at 10 a.m. on a Monday.
A practical rule: pick the model by quality first, then size the GPUs for the busiest hour, not the average. The VRAM requirements guide has the calculation for other models.
The office is the hard part: 6.5 kW of power and heat
A single GPU in a workstation fits under a desk. An 8-GPU server does not. Our 8 × RTX PRO 6000 server draws about 1.1 kW at idle, about 6.5 kW with all GPUs at full load and up to 6,960 W in a synthetic stress test, measured at the wall (power and temperatures).
What that means in a building:
- Power. A standard 230 V circuit with a 16 A breaker delivers at most 3.68 kW. One 8-GPU server needs more than that on its own, so it needs dedicated circuits or a three-phase feed, sized for the 7 kW peak.
- Heat. Almost all of that power becomes heat, about 23,750 BTU per hour at the peak, all day. An office air conditioner is not built to remove it.
- Noise. To keep eight passive cards at 600 W without throttling, our server runs its fans at full speed all the time. It belongs in a server room with the door closed.
- Idle cost. At 1.1 kW the server uses about 790 kWh in a 30-day month before anyone sends a prompt.
This is the main reason many companies that want an on-premise LLM put the hardware in a data centre instead of the office.
The software stack: an engine behind an OpenAI-compatible API
The model itself is only one part. A setup a team can rely on usually has four layers:
- A serving engine. Ollama is the simplest and suits a few users. vLLM and SGLang are built to serve many requests at once. Our vLLM vs Ollama vs SGLang comparison covers the trade-offs.
- An OpenAI-compatible endpoint. Ollama and vLLM both offer one, so tools and code written for a cloud API usually need only a new address and key.
- A gateway in front of it. User accounts or API keys, limits per user, and a clear decision on whether prompts are logged. Prompts often contain personal data, so the log is data you have to protect too.
- Monitoring. GPU temperature, power and memory. Measure power at the wall or the rack PDU: the management controller on our server reports only about half of the real draw.
Running it: what went wrong on our own server
These are things we found on our own server between July and October 2026, none of which a spec sheet mentions:
- The default fan mode cut GPU power. In the factory "Optimal" mode the cards throttled to 372 W at 88°C under load, about 62% of their rated power. Switching the fans to full speed fixed it.
- The power reading was half the truth. The management controller counted two of the four power supplies, so it showed about half the power the data centre's meter billed.
- A drive dropped out under load. A power-saving feature of one NVMe drive made it drop out of the storage array. Turning that feature off fixed it.
- Idle power costs money. About 1.1 kW runs day and night, whether or not the GPUs are busy.
Each of these took hours to find, and the burn-in test guide shows how we test a server before it carries real work. If you buy and colocate, plan for this kind of work, plus spare parts, remote hands in the data centre and out-of-band access through the server's management controller. With a rented dedicated server, the provider handles the hardware and you keep root access to everything above it.
Data protection: on-premise is a start, not compliance
Keeping the model on your own hardware removes one question: no prompt goes to a third-party model provider. The rest of the work stays:
- GDPR applies to what goes into the model. If prompts or documents contain personal data, your company is still the controller. A hosting provider that runs the hardware for you is a processor, which needs a data processing agreement under Article 28.
- Location still matters. Hardware in the EU keeps you out of the rules for transfers to third countries, as long as nobody outside the EU has access to it.
- Tell people they are talking to AI. From 2 August 2026, the EU AI Act requires that people are informed when they interact with an AI system such as a chatbot, unless it is obvious (Article 50).
Our private LLM hosting page describes how we handle data on our server.
Questions
What is the difference between on-premise and self-hosted LLM? In practice they mean the same thing: you run an open model on hardware that you control, instead of sending prompts to a shared API. "On-premise" sometimes means the hardware is in your own building. A dedicated server in a data centre gives the same control over the model and the data, without the power and cooling work.
Can I rent an LLM server instead of buying one? Yes. A dedicated GPU server is rented by the month for a fixed term, with root access and no other tenants. You install the model and the serving engine yourself, as you would on your own hardware.
How many GPUs do I need for a company chatbot? Start from the model. A 70B model or gpt-oss-120b fits on one 96 GB card at 4-bit. Then add memory for the busiest hour: each open conversation needs its own cache. Eight cards can run several copies of a mid-size model, one per card, or one large model across all of them.
Is on-premise cheaper than an API? Only at steady, high volume. A server costs the same every month, busy or idle. At low or uneven volume, an API usually costs less.
Sources
- Ollama library, 4-bit file sizes: llama3.3, gpt-oss, qwen3, checked on 13 October 2026.
- Key-value cache per token for Llama 3.3 70B: our calculation from the model's configuration (80 layers, 8 KV heads, head size 128, 16-bit).
- vLLM: documentation and OpenAI-compatible server. Ollama: OpenAI compatibility.
- Power, temperatures, fan mode and the power reading: measured on our server, July to October 2026, see 8x RTX PRO 6000 server power and temperatures.
- Server prices and monthly cost of ownership: AI server cost in 2026, with sources dated October 2026.
- GDPR: Regulation (EU) 2016/679, Article 28. EU AI Act: Regulation (EU) 2024/1689, Article 50 and Article 113 (application dates).
- NEOXIS: pricing, from $1.80 per GPU-hour, checked on 17 November 2026.


