Short answer
An LLM inference engine is the software that loads a model onto your hardware and turns prompts into tokens as efficiently as possible. For production serving on GPUs, the leading open-source choices are vLLM and SGLang (high throughput, many users), with NVIDIA's TensorRT-LLM for maximum performance on NVIDIA hardware. For local and personal use, llama.cpp and the apps built on it, Ollama and LM Studio, run quantized models on almost any computer, and MLX is optimised for Apple Silicon. The right choice depends on whether you care most about throughput, latency, ease of use or hardware flexibility.
Why the engine matters
The same model on the same GPU can run several times faster or slower depending on the engine. Serving LLMs is hard because:
- Decode is memory-bound: each token requires reading the full weights; see prefill vs decode.
- The KV cache is large and grows unpredictably: every request's KV cache expands as it generates, and requests finish at different times.
- Requests vary wildly: short chats, long documents, many users at once.
Good engines solve these with smarter memory management, batching, caching and optimised GPU kernels.
Key techniques inside modern engines
Continuous (in-flight) batching
Naive batching waits for a whole batch to finish before starting new requests, leaving the GPU idle as short requests complete. Continuous batching adds and removes requests at every decoding step, keeping the GPU full. It is the single biggest throughput improvement for multi-user serving.
PagedAttention
Introduced with vLLM in 2023, PagedAttention stores the KV cache in small fixed-size blocks, like virtual memory pages in an operating system, instead of one large contiguous buffer per request. This nearly eliminates wasted memory from fragmentation and over-reservation, so many more requests fit on a GPU at once. Most serious engines now use a paged KV cache.
Prefix caching
Many requests share the same beginning: a long system prompt, a document in a RAG pipeline, or earlier turns of a conversation. Prefix caching reuses the KV cache for that shared prefix instead of recomputing it, cutting time to first token dramatically. SGLang's RadixAttention organises cached prefixes in a tree for automatic reuse.
Optimised kernels
Fused attention kernels such as FlashAttention and FlashInfer, CUDA graphs, and hand-tuned matrix multiplications for FP8 and FP4 squeeze more from each GPU.
Quantization support
Engines differ in which formats they run fast: GGUF for llama.cpp, AWQ, GPTQ, FP8 and NVFP4 for GPU servers. See LLM quantization and NVFP4 explained.
Speculative decoding
Draft-and-verify methods speed up single-stream generation 2–3×. See speculative decoding explained.
Parallelism and disaggregation
Large models are split across GPUs with tensor, pipeline and expert parallelism; see distributed training and parallelism explained. Advanced deployments disaggregate prefill and decode onto separate GPU pools, because the two phases stress hardware differently.
The engines compared
| Engine | Best for | Hardware | Model formats | Ease of use |
|---|---|---|---|---|
| vLLM | Production serving, many users, broad model support | NVIDIA, AMD, TPUs, others via plugins | Hugging Face models, AWQ, GPTQ, FP8, others | Moderate (Python, OpenAI-compatible server) |
| SGLang | High-throughput serving, agents, structured output, prefix reuse | NVIDIA, AMD | Hugging Face models, FP8, others | Moderate |
| TensorRT-LLM | Maximum performance on NVIDIA GPUs | NVIDIA only | Converted engines, FP8, NVFP4 | Harder (build and tuning steps) |
| llama.cpp | Local inference on any hardware, CPUs, Macs, edge | CPU, NVIDIA, AMD, Apple Metal, Vulkan | GGUF | Moderate (command line) |
| Ollama | Easiest local setup, one-command model downloads | Same as llama.cpp | GGUF (from its library) | Very easy |
| LM Studio | Local use with a graphical interface | Windows, macOS, Linux | GGUF, MLX | Very easy |
| MLX / mlx-lm | Fast local inference and fine-tuning on Apple Silicon | Apple M-series | MLX format | Easy for Python users |
vLLM
Started at UC Berkeley, vLLM popularised PagedAttention and continuous batching and has become the most widely deployed open-source serving engine. It supports a very wide range of models, including mixture-of-experts and multimodal models, offers an OpenAI-compatible API, and runs on NVIDIA and AMD GPUs and more. A good default for teams serving open models.
pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 16384SGLang
SGLang combines a fast runtime with RadixAttention prefix caching and strong support for structured output and multi-call programs, making it popular for agent workloads and large MoE models. It competes closely with vLLM on throughput, and the two projects borrow ideas from each other.
TensorRT-LLM
NVIDIA's library compiles models into optimised engines with the latest kernels, FP8 and NVFP4 support, in-flight batching and speculative decoding. It often achieves the highest throughput on NVIDIA data center GPUs such as the H100, H200 and B200, at the cost of more setup and less flexibility. It is often used together with NVIDIA's Triton Inference Server or Dynamo for deployment.
llama.cpp
A C/C++ engine that runs GGUF-quantized models on almost anything: laptops, desktops, Macs, Raspberry Pi-class boards and GPUs from every vendor. It can split models between GPU VRAM and system RAM, which lets you run models larger than your GPU memory, more slowly. It is the foundation of much of the local AI ecosystem.
Ollama and LM Studio
Both wrap llama.cpp-style inference in friendly tools. Ollama gives you a one-line command (ollama run llama3.1:8b) and a local API; LM Studio adds a graphical model browser and chat interface. Ideal for personal use, prototyping and private document chat. They are not designed for high-concurrency production serving. See how to run an LLM locally.
MLX
Apple's array framework for its own chips. mlx-lm runs and fine-tunes models efficiently on M-series Macs, taking advantage of large unified memory.
Throughput vs latency
| Goal | What to optimise | Typical choice |
|---|---|---|
| Serve many users cheaply | Tokens per second per GPU (throughput) | vLLM, SGLang, TensorRT-LLM with large batches |
| Fast replies for one user | Time to first token and tokens per second per user | Speculative decoding, smaller or quantized models |
| Run on a laptop | Fits in memory, simple setup | Ollama, LM Studio, llama.cpp, MLX |
| Long shared context (RAG, agents) | Prefix cache hit rate | SGLang or vLLM with prefix caching |
You can explore how batch size, context and hardware affect speed in our LLM inference simulator.
Hardware sizing tips
- VRAM sets the limit: weights plus KV cache for all concurrent requests must fit. See how much VRAM you need.
- Bandwidth sets single-user speed: see GPU memory bandwidth.
- Compute sets batched throughput: more tensor core TFLOPS lets you serve more users per GPU.
- Rent before you buy: benchmark your workload on cloud GPUs first; compare prices in our GPU cloud price tracker.
How to choose
1. Just want to chat with models on your PC or Mac? Ollama or LM Studio. 2. Building a local app or running on unusual hardware? llama.cpp. 3. On a Mac and want speed? MLX or LM Studio's MLX backend. 4. Serving an open model to users from GPU servers? Start with vLLM or SGLang. 5. Need every last token per second on NVIDIA hardware? Evaluate TensorRT-LLM. 6. Always benchmark with your own prompts, context lengths and concurrency.
Frequently asked questions
Is Ollama good for production?
Ollama is excellent for local and small internal use, but engines built for high concurrency, such as vLLM and SGLang, achieve much higher throughput for many simultaneous users.
Is vLLM faster than llama.cpp?
On GPU servers with many concurrent requests, vLLM is usually much faster thanks to continuous batching and paged KV cache. For a single user on a laptop or with CPU offloading, llama.cpp is often the better fit.
Do these engines change model quality?
The engine itself does not, but the quantization format and sampling settings you choose can. Compare outputs when switching formats.
Can I use the OpenAI API format with these engines?
Yes. vLLM, SGLang, Ollama, LM Studio and llama.cpp's server all offer OpenAI-compatible endpoints, so most apps can switch with a URL change.
Sources and further reading
- Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM, 2023)
- Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs (2023)
- Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (continuous batching, OSDI 2022)
- vLLM documentation
- NVIDIA TensorRT-LLM on GitHub
- llama.cpp on GitHub




Leave a Reply