vLLM vs Ollama vs SGLang: which one to serve open models

Ollama, vLLM and SGLang compared: how each handles parallel requests and several GPUs, what published benchmarks show, and which fits your workload.

A single NVIDIA H100 PCIe data centre GPU with a tan metal shroud, standing on a white surface against a black background
Photo: NVIDIA H100 (极客湾Geekerwan) 014 by 极客湾Geekerwan, Wikimedia Commons, CC BY 3.0

All three run open models and answer over an OpenAI-compatible API. They differ in how many people they are built to serve at once, and that difference is what this guide measures against published tests.

If Choose Why
You work alone or with a few colleagues and want a model running in minutes Ollama One command, a large model library, modest hardware
A team or a product sends many requests at once, and you want the widest model support vLLM Continuous batching, 200+ model architectures, a commercially supported distribution
Your requests share long prefixes: agents, long chats, the same system prompt or documents SGLang Built around reusing cached prefixes, with a gateway that routes by cache
You need the hardware for vLLM or SGLang, with no other tenants and root access in the EU A dedicated GPU server, such as our 8x RTX PRO 6000 from $1.80 per GPU-hour, about $10,510 a month 768 GB of GPU memory. It is rented into early 2027 and we accept reservations for the period after that

We have not benchmarked these engines ourselves: our server is with a customer, and we do not run tests on a rented machine. Every number below comes from the published source named next to it.

How the three differ

Ollama vLLM SGLang
Started as A simple way to run models locally A research project at UC Berkeley's Sky Computing Lab A project hosted by LMSYS, a non-profit
Key technique Runs on llama.cpp, with quantized model files PagedAttention, continuous batching, prefix caching RadixAttention, a prefix cache kept in a radix tree
Requests at once 1 per model by default, raised with OLLAMA_NUM_PARALLEL No fixed limit of that kind 100% success at 128 at once in the test below
Several GPUs Uses one card if the model fits on it, otherwise spreads across all --tensor-parallel-size and --data-parallel-size --tp-size and --dp-size, plus a routing gateway
License MIT Apache 2.0 Apache 2.0

Ollama: simple first, parallel by setting

Ollama binds to 127.0.0.1 port 11434 by default and answers one request per model at a time. Three defaults in its FAQ decide how it behaves as a shared service:

  • Parallel requests. OLLAMA_NUM_PARALLEL is 1, and required memory scales with that value multiplied by the context length. Further requests wait in a queue of up to 512, then get rejected.
  • Unloading. A model stays in memory for 5 minutes after use. The next request after a quiet period has to load it again, unless you change OLLAMA_KEEP_ALIVE.
  • Several GPUs. If a model fits on one card, Ollama loads it on one card. On an 8-GPU server, one small model will not use the other seven by itself.

None of this is a flaw. Red Hat's guide calls the primary goal of Ollama "accessibility, not scalability". For testing a model before you commit to it, as we describe in local LLM vs private LLM server, that is the right goal.

vLLM: built for many requests at once

vLLM describes itself as "a high-throughput and memory-efficient inference and serving engine for LLMs". PagedAttention stores the key-value cache in blocks, so memory goes only to tokens that exist. Continuous batching adds new requests to the running batch rather than waiting for it to finish. The prefix cache hashes each cache block together with the tokens before it, so requests that share a beginning reuse the work.

Its documentation lists 200+ model architectures, on NVIDIA and AMD GPUs and on CPUs. It has 96.5k stars on GitHub, and Red Hat sells a supported distribution of it. Before you commit, check that your model is on its supported list.

One default to plan for: vLLM takes 0.92 or 1.0 of the GPU's memory at start, depending on the cache mode, so sharing a card with another process means lowering --gpu-memory-utilization.

SGLang: built around reusing prompts

SGLang's central idea is that many LLM calls begin with the same text: a system prompt, few-shot examples, earlier turns of a chat, or the steps of an agent. RadixAttention keeps those prefixes in a radix tree, evicts the least recently used, and schedules requests with the cache in mind.

Its Model Gateway extends this across copies of a model. The default policy, cache-aware routing, combines cache locality with load balancing. vLLM's built-in balancer for data parallel copies uses the running and waiting queues, and its documentation notes that it does not yet account for cache state.

The project says it is "powering 400k+ GPUs" and names no companies in the text of its page, so treat that as its own claim. Its hardware list includes the RTX PRO 6000 and the B200 and H100 families.

What the published benchmarks show

Test Hardware and model vLLM SGLang Ollama
Red Hat, 8 August 2025 One A100 40 GB, Llama 3.1 8B, concurrency 1 to 256 793 tokens/s, 80 ms P99 latency at peak Not tested 41 tokens/s, 673 ms P99 latency at peak
Joshua8.AI, 10 January 2026 One RTX PRO 6000 Max-Q 96 GB, Llama 3.1 8B 8,033 tokens/s (NVFP4) 6,395 tokens/s (GPTQ-INT4) 484 tokens/s (Q4_K_M)
Same test, first token at one request 13.8 ms (NVFP4), 10.7 ms (GPTQ-INT4) 14.2 ms 65.0 ms

Ollama falls far behind once requests overlap. Both tests agree on the direction: about 19 times less throughput than vLLM in Red Hat's test and about 17 times in the Joshua8.AI test. In the second test, vLLM and SGLang kept a 100% success rate at 128 concurrent requests, while Ollama "broke down". Red Hat tuned Ollama to OLLAMA_NUM_PARALLEL=32, which it calls the highest stable value on that card.

vLLM against SGLang is close and depends on the setup. The Joshua8.AI headline number favours vLLM, but the two ran different formats. On the same GPTQ-INT4 format, SGLang won 6 of 8 concurrency levels, by up to 17%. SGLang ran there with --disable-cuda-graph because of a Blackwell compatibility issue, and the author could not get its NVFP4 working. Both may have changed since January.

Read these tests with care. Red Hat sells a vLLM product, and its test uses versions from mid-2025 (vLLM 0.9.1, Ollama 0.9.2). Joshua8.AI is one company's lab, one card, one model and one prompt pattern, and it does not state software versions. Its prompt is a single sentence with one changing word, so it says little about workloads with long shared prefixes.

For those workloads the main figure is LMSYS's own from January 2024: "up to 5 times higher throughput" for SGLang against the systems it compared, including vLLM 0.2.5. vLLM has added prefix caching since, so that gap should not be assumed today. We found no neutral, reproducible 2026 test of all three on the same large model. Run your own with your prompts; GuideLLM, the tool Red Hat used, is one way.

Laying them out on an 8-GPU server

On a machine with eight 96 GB cards, both server engines offer the same two levers:

  • Copies of the model. Each copy serves requests on its own, which multiplies throughput for a model that fits. SGLang's documentation shows --dp-size 8 with --tp-size 1, and vLLM has --data-parallel-size. gpt-oss-120b at 65 GB fits on one card with room for the cache.
  • One model across several cards. Qwen3-235B-A22B at 142 GB needs two. vLLM's documentation gives --data-parallel-size 4 --tensor-parallel-size 2 as an example for one 8-GPU node: four copies, each on two cards.

Sizes here are the 4-bit files in the Ollama library. vLLM and SGLang work with Hugging Face weights, so the quantization you tested on a laptop is not the same file you will run on the server. Check the answers again after the move. Details of the hardware are on the RTX PRO 6000 server page.

Keep all three off the open internet

Ollama's FAQ shows how to change the bind address and put Nginx in front, and it describes no access control of its own. vLLM has --api-key, but its documentation warns not to rely on that alone, because it covers only some paths and /invocations stays unauthenticated. SGLang also has --api-key.

SGLang had a CERT/CC note in March 2026 (VU#665416) on unsafe deserialization that could lead to code execution when certain features are enabled. CERT/CC says version 0.5.10 addresses it and advises against exposing the service interfaces to untrusted networks. That advice suits all three engines: use a private network or VPN, with an authenticating proxy in front, and update regularly. How we handle data and access is on the private LLM hosting page.

Questions

Can Ollama run in production? For a few internal users and light traffic, yes, with OLLAMA_NUM_PARALLEL raised and memory planned for it. The tests above show where it stops: heavy overlapping requests.

Do I have to rewrite my application to switch? Usually not. All three offer an OpenAI-compatible endpoint, so the change is mainly the address and the model name. Tool calling and structured output settings can differ, so test those.

Does Ollama support JSON output? Yes. Its documentation says structured outputs let you enforce a JSON schema on model responses, through the format parameter.

How much context should I allow? Only what your real documents need. More context means more cache per user. Ollama's default is 4,096 tokens, while vLLM derives the limit from the model config unless you set --max-model-len.

Sources

SHARE