Skip to content
,

vLLM vs SGLang vs TensorRT-LLM vs llama.cpp vs Ollama: LLM Inference Engines Compared

Which LLM inference engine should you use? An in-depth comparison of vLLM, SGLang, TensorRT-LLM, llama.cpp, Ollama, LM Studio and MLX: how PagedAttention, continuous batching and prefix caching work, hardware support, throughput vs latency, and which to pick for local use or production serving.

6 min read

LLM Inference Engines Compared — cover image from TensorCore.si

Short answer

An LLM inference engine is the software that loads a model onto your hardware and turns prompts into tokens as efficiently as possible. For production serving on GPUs, the leading open-source choices are vLLM and SGLang (high throughput, many users), with NVIDIA's TensorRT-LLM for maximum performance on NVIDIA hardware. For local and personal use, llama.cpp and the apps built on it, Ollama and LM Studio, run quantized models on almost any computer, and MLX is optimised for Apple Silicon. The right choice depends on whether you care most about throughput, latency, ease of use or hardware flexibility.

Why the engine matters

The same model on the same GPU can run several times faster or slower depending on the engine. Serving LLMs is hard because:

  • Decode is memory-bound: each token requires reading the full weights; see prefill vs decode.
  • The KV cache is large and grows unpredictably: every request's KV cache expands as it generates, and requests finish at different times.
  • Requests vary wildly: short chats, long documents, many users at once.

Good engines solve these with smarter memory management, batching, caching and optimised GPU kernels.

Key techniques inside modern engines

Continuous (in-flight) batching

Naive batching waits for a whole batch to finish before starting new requests, leaving the GPU idle as short requests complete. Continuous batching adds and removes requests at every decoding step, keeping the GPU full. It is the single biggest throughput improvement for multi-user serving.

PagedAttention

Introduced with vLLM in 2023, PagedAttention stores the KV cache in small fixed-size blocks, like virtual memory pages in an operating system, instead of one large contiguous buffer per request. This nearly eliminates wasted memory from fragmentation and over-reservation, so many more requests fit on a GPU at once. Most serious engines now use a paged KV cache.

Prefix caching

Many requests share the same beginning: a long system prompt, a document in a RAG pipeline, or earlier turns of a conversation. Prefix caching reuses the KV cache for that shared prefix instead of recomputing it, cutting time to first token dramatically. SGLang's RadixAttention organises cached prefixes in a tree for automatic reuse.

Optimised kernels

Fused attention kernels such as FlashAttention and FlashInfer, CUDA graphs, and hand-tuned matrix multiplications for FP8 and FP4 squeeze more from each GPU.

Quantization support

Engines differ in which formats they run fast: GGUF for llama.cpp, AWQ, GPTQ, FP8 and NVFP4 for GPU servers. See LLM quantization and NVFP4 explained.

Speculative decoding

Draft-and-verify methods speed up single-stream generation 2–3×. See speculative decoding explained.

Parallelism and disaggregation

Large models are split across GPUs with tensor, pipeline and expert parallelism; see distributed training and parallelism explained. Advanced deployments disaggregate prefill and decode onto separate GPU pools, because the two phases stress hardware differently.

The engines compared

EngineBest forHardwareModel formatsEase of use
vLLMProduction serving, many users, broad model supportNVIDIA, AMD, TPUs, others via pluginsHugging Face models, AWQ, GPTQ, FP8, othersModerate (Python, OpenAI-compatible server)
SGLangHigh-throughput serving, agents, structured output, prefix reuseNVIDIA, AMDHugging Face models, FP8, othersModerate
TensorRT-LLMMaximum performance on NVIDIA GPUsNVIDIA onlyConverted engines, FP8, NVFP4Harder (build and tuning steps)
llama.cppLocal inference on any hardware, CPUs, Macs, edgeCPU, NVIDIA, AMD, Apple Metal, VulkanGGUFModerate (command line)
OllamaEasiest local setup, one-command model downloadsSame as llama.cppGGUF (from its library)Very easy
LM StudioLocal use with a graphical interfaceWindows, macOS, LinuxGGUF, MLXVery easy
MLX / mlx-lmFast local inference and fine-tuning on Apple SiliconApple M-seriesMLX formatEasy for Python users

vLLM

Started at UC Berkeley, vLLM popularised PagedAttention and continuous batching and has become the most widely deployed open-source serving engine. It supports a very wide range of models, including mixture-of-experts and multimodal models, offers an OpenAI-compatible API, and runs on NVIDIA and AMD GPUs and more. A good default for teams serving open models.

pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 16384

SGLang

SGLang combines a fast runtime with RadixAttention prefix caching and strong support for structured output and multi-call programs, making it popular for agent workloads and large MoE models. It competes closely with vLLM on throughput, and the two projects borrow ideas from each other.

TensorRT-LLM

NVIDIA's library compiles models into optimised engines with the latest kernels, FP8 and NVFP4 support, in-flight batching and speculative decoding. It often achieves the highest throughput on NVIDIA data center GPUs such as the H100, H200 and B200, at the cost of more setup and less flexibility. It is often used together with NVIDIA's Triton Inference Server or Dynamo for deployment.

llama.cpp

A C/C++ engine that runs GGUF-quantized models on almost anything: laptops, desktops, Macs, Raspberry Pi-class boards and GPUs from every vendor. It can split models between GPU VRAM and system RAM, which lets you run models larger than your GPU memory, more slowly. It is the foundation of much of the local AI ecosystem.

Ollama and LM Studio

Both wrap llama.cpp-style inference in friendly tools. Ollama gives you a one-line command (ollama run llama3.1:8b) and a local API; LM Studio adds a graphical model browser and chat interface. Ideal for personal use, prototyping and private document chat. They are not designed for high-concurrency production serving. See how to run an LLM locally.

MLX

Apple's array framework for its own chips. mlx-lm runs and fine-tunes models efficiently on M-series Macs, taking advantage of large unified memory.

Throughput vs latency

GoalWhat to optimiseTypical choice
Serve many users cheaplyTokens per second per GPU (throughput)vLLM, SGLang, TensorRT-LLM with large batches
Fast replies for one userTime to first token and tokens per second per userSpeculative decoding, smaller or quantized models
Run on a laptopFits in memory, simple setupOllama, LM Studio, llama.cpp, MLX
Long shared context (RAG, agents)Prefix cache hit rateSGLang or vLLM with prefix caching

You can explore how batch size, context and hardware affect speed in our LLM inference simulator.

Hardware sizing tips

How to choose

1. Just want to chat with models on your PC or Mac? Ollama or LM Studio. 2. Building a local app or running on unusual hardware? llama.cpp. 3. On a Mac and want speed? MLX or LM Studio's MLX backend. 4. Serving an open model to users from GPU servers? Start with vLLM or SGLang. 5. Need every last token per second on NVIDIA hardware? Evaluate TensorRT-LLM. 6. Always benchmark with your own prompts, context lengths and concurrency.

Frequently asked questions

Is Ollama good for production?

Ollama is excellent for local and small internal use, but engines built for high concurrency, such as vLLM and SGLang, achieve much higher throughput for many simultaneous users.

Is vLLM faster than llama.cpp?

On GPU servers with many concurrent requests, vLLM is usually much faster thanks to continuous batching and paged KV cache. For a single user on a laptop or with CPU offloading, llama.cpp is often the better fit.

Do these engines change model quality?

The engine itself does not, but the quantization format and sampling settings you choose can. Compare outputs when switching formats.

Can I use the OpenAI API format with these engines?

Yes. vLLM, SGLang, Ollama, LM Studio and llama.cpp's server all offer OpenAI-compatible endpoints, so most apps can switch with a URL change.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *