How much VRAM does an LLM need?

VRAM = model weights + KV cache + overhead. The formula, worked numbers for Qwen3-8B, Qwen3-32B, Llama 3.3 70B and gpt-oss-120b, and what fits on 96 GB.

Bare circuit board of an NVIDIA RTX 4070 Super graphics card with the cooler removed, the GPU chip in the centre and GDDR memory chips around it
Photo: Video über die RTX 4070 Super und Vergleichskarten (极客湾Geekerwan) 12 by 极客湾Geekerwan, Wikimedia Commons, CC BY 3.0

A large language model needs GPU memory for three things: its weights, a cache for every conversation it is serving (the KV cache), and some working space for the inference engine. The weights are easy to estimate from the parameter count. The cache is the part people forget, and with long prompts or many users it can be larger than the model itself. Below is the formula, the numbers for four popular open models, and what fits on one and on eight 96 GB cards.

Which setup for whom

  • Models up to about 14B, one user, short prompts → a single consumer card with 16 to 24 GB. Qwen3-8B in 8-bit needs about 8 GB for weights plus a few GB of cache.
  • 30B-class models, or 70B in 4-bit, a few users → one 48 GB card, or one 96 GB card for room to grow.
  • 70B in FP8 with long context, gpt-oss-120b, or a team of users on one model → 96 GB per card. One RTX PRO 6000 Blackwell has 96 GB of GDDR7.
  • 70B in full 16-bit precision, several models at once, or hundreds of parallel requests → several cards in one server. Our server in Sofia has eight RTX PRO 6000 cards, 768 GB of GPU memory in total. It rents from $1.80 per GPU-hour, or about $10,510 a month for the whole machine. It is rented into early 2027 and we take reservations for the period after. See the server.

The formula

VRAM needed ≈ weights + KV cache + overhead

Weights = parameters × bytes per parameter. 16-bit (FP16 or BF16) is 2 bytes, FP8 or INT8 is 1 byte, and 4-bit formats are about half a byte plus a little for scaling factors. So a 70B model is about 140 GB in 16-bit, about 70 GB in FP8 and roughly 35 to 40 GB in 4-bit.

KV cache per token = 2 × layers × KV heads × head size × bytes. The 2 is for keys and values. The layer count, KV heads and head size are in the model's config.json on Hugging Face. Multiply by the number of tokens in all conversations served at the same time: prompt plus answer, for every user.

Overhead covers activations, the CUDA context and the engine's own buffers. vLLM, for example, takes 90% of GPU memory by default (gpu_memory_utilization = 0.9) and fills what is left after the weights with KV cache. Plan on keeping about 10% of the card out of your sums.

KV cache for four open models

These are computed from each model's published configuration, with a 16-bit cache:

Model Layers KV heads × head size KV cache per token 32,768 tokens
Qwen3-8B 36 8 × 128 144 KiB 4.5 GiB
Qwen3-32B 64 8 × 128 256 KiB 8 GiB
Llama 3.3 70B 80 8 × 128 320 KiB 10 GiB
gpt-oss-120b 36 8 × 64 up to 72 KiB (1) up to 2.25 GiB

(1) gpt-oss uses a 128-token sliding window in some of its layers, so its real cache for long prompts is smaller than this upper bound.

All four use grouped-query attention: only 8 KV heads, not one per attention head. Without it the cache would be several times larger. That is why an older model with the same parameter count can need much more memory for the same context.

Two things multiply the cache:

  • Context length. Llama 3.3 70B accepts up to 131,072 tokens. One conversation at full length needs about 40 GiB of cache, more than half of the FP8 weights.
  • Parallel users. Ten users with 8,000 tokens each is 80,000 tokens of cache, the same as one user with 80,000.

Most engines can also store the cache in FP8, which halves these numbers at a small cost in accuracy. Test it on your own prompts before relying on it.

Worked examples

Qwen3-8B on a 24 GB card. In 16-bit the weights are about 16 GB. With 10% overhead, roughly 5 GB is left for cache, about 34,000 tokens. Enough for one user with long documents or a handful of short chats.

Qwen3-32B on a 96 GB card. In FP8 the weights are about 32 GB. Of the 86 GB that vLLM uses at its default, about 54 GB is left for cache: over 200,000 tokens, enough for a small team.

Llama 3.3 70B on a 96 GB card. In FP8 the weights take about 70 GB, leaving roughly 16 GB for cache. That is nearly 50,000 tokens, so one long conversation or several short ones at a time. With an FP8 cache it doubles to around 100,000. In 4-bit the weights drop to under 40 GB, and the rest of the card holds the cache.

gpt-oss-120b. OpenAI ships the MoE weights in the 4-bit MXFP4 format and says the model fits on a single 80 GB GPU. On a 96 GB card that leaves noticeably more room for cache than on 80 GB. The model has 117B parameters but only 5.1B are active per token, so it answers quickly for its size. The memory still has to hold all 117B.

Llama 3.3 70B in 16-bit on eight cards. The weights, about 140 GB, are split across the cards with tensor parallelism. Of the roughly 690 GB the engine uses, about 550 GB is left for cache: well over a million tokens. This is the setup for many users on one model at full precision, or for running a 70B model next to several smaller ones. More on serving engines in vLLM vs Ollama vs SGLang.

Common mistakes

  • Sizing for weights only. The model loads, then fails or slows down once real prompts arrive, because there is no room for cache.
  • Mixing GB and GiB. Model files are listed in GB (10⁹ bytes), GPU memory tools often show GiB (2³⁰ bytes). A 96 GB card shows about 89 GiB.
  • Counting MoE models by active parameters. A mixture-of-experts model needs memory for all of its experts, not only the ones used per token.
  • Forgetting the vision part. Multimodal models carry an image encoder and turn every image into hundreds or thousands of tokens of cache.
  • Splitting over too many small cards. Two 48 GB cards hold the same weights as one 96 GB card, but every layer then talks over PCIe. One large card keeps the model in one place.

For when a laptop is enough and when a server pays off, see local LLM vs private LLM server. For hosting your own model in the EU, see private LLM hosting.

Questions

Is there a simple rule of thumb? For the weights: about 2 GB per billion parameters in 16-bit, 1 GB in 8-bit and 0.5 to 0.6 GB in 4-bit. Then add 20 to 50% for cache and overhead, more with long prompts or many users. The table above is more precise.

Does a bigger context window cost memory if I do not use it? Not with vLLM or SGLang: the cache is filled by the tokens you actually send. But the engine checks at start-up that at least one request at the maximum length fits, so a too large max_model_len can stop it from starting.

Can I use system RAM instead of VRAM? Tools like Ollama and llama.cpp can keep part of the model in system memory. It works, but every token then waits for data over PCIe, and speed drops sharply. For serving several users, keep the model and cache in GPU memory.

How much VRAM for fine-tuning? Much more than for serving. Full fine-tuning keeps gradients and optimizer states next to the weights. LoRA and QLoRA cut this a lot, but they still need more than inference of the same model.

Sources

  • Qwen, Qwen3-8B config.json: 36 layers, 8 KV heads, head size 128, checked 15 November 2026.
  • Qwen, Qwen3-32B config.json: 64 layers, 8 KV heads, head size 128, checked 15 November 2026.
  • Meta, Llama 3.3 70B Instruct configuration (public mirror of the model files): 80 layers, 8 KV heads, hidden size 8,192 with 64 heads (head size 128), maximum 131,072 positions, checked 15 November 2026.
  • OpenAI, gpt-oss-120b model card and config.json: 117B parameters, 5.1B active, MXFP4 MoE weights, fits a single 80 GB GPU; 36 layers, 8 KV heads, head size 64, sliding window 128, checked 15 November 2026.
  • vLLM, engine arguments: gpu_memory_utilization default 0.9, checked 15 November 2026.
  • NVIDIA, RTX PRO 6000 Blackwell Server Edition: 96 GB GDDR7, checked 13 November 2026.
SHARE