Edited by 
Short answer
The easiest way to run an LLM locally is to install Ollama and type ollama run llama3.1:8b, or use LM Studio if you prefer a graphical app. Both download a quantized model and run it on your GPU, Apple silicon or CPU. For more control, llama.cpp runs GGUF models directly. A GPU with 8 to 12 GB of VRAM, or a Mac with 16 GB or more of unified memory, comfortably runs 7B to 8B models at 4-bit. All three tools offer an OpenAI-compatible API, so your own apps and agents can use the local model.
Why run an LLM locally?
- Privacy: your prompts and documents never leave your computer.
- No usage fees: once you have the hardware, generating text is free.
- Offline use: works without an internet connection.
- Control: choose any open-weight model, quantization and settings, and build your own tools on top.
The trade-off is that local models are usually smaller than the best cloud models, and speed depends on your hardware.
What hardware do you need?
The key number is memory: the model has to fit in your GPU's VRAM (or a Mac's unified memory). Use our guide on how much VRAM you need for an LLM for exact figures. As a quick rule:
| Your hardware | Comfortable model size (4-bit) |
|---|---|
| CPU only, 16 GB RAM | 1B to 4B (slow but usable) |
| GPU with 8 GB VRAM | up to 8B |
| GPU with 12 to 16 GB | 8B to 14B |
| GPU with 24 GB (RTX 3090, 4090) | up to about 32B |
| GPU with 32 GB (RTX 5090) | 32B with long context |
| Mac with 32 to 64 GB | 14B to 70B |
NVIDIA GPUs have the best support. AMD GPUs work through ROCm or Vulkan, and Apple silicon Macs work very well thanks to Metal and unified memory. For buying advice, see the best GPU for local LLMs.
Option 1: Ollama (easiest from the command line)
Ollama is a free tool for Windows, macOS and Linux that downloads, manages and runs models with one command.
Install and run your first model
Download Ollama from ollama.com, or on Linux run the install script from the site. Then open a terminal:
ollama run llama3.1:8bThe first run downloads a 4-bit quantized model (about 5 GB); after that you get a chat prompt. Type /bye to exit. Other useful commands:
ollama pull qwen3:8b # download a model without starting a chat
ollama list # show downloaded models
ollama ps # show running models and GPU memory use
ollama rm llama3.1:8b # delete a modelBrowse the model library on ollama.com for options such as Llama, Qwen, Gemma, Mistral, DeepSeek and gpt-oss. Tags like :8b or :27b choose the size.
Use Ollama from your own code
Ollama runs a local server on port 11434 with an OpenAI-compatible endpoint:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
model="llama3.1:8b",
messages=[{"role": "user", "content": "Explain tensor cores in one sentence."}],
)
print(reply.choices[0].message.content)Because the API matches OpenAI's, most tools and agent frameworks can use a local model by changing the base URL. Our Python programming volume shows how to build a full AI agent this way.
Option 2: LM Studio (easiest with a graphical app)
LM Studio is a free desktop app for Windows, macOS and Linux:
- Install LM Studio and open the Discover tab.
- Search for a model, for example "Qwen3 8B" or "Gemma 3 12B".
- Pick a quantization; LM Studio marks which files fit your hardware. Q4_K_M is a good default.
- Download, load the model and start chatting.
LM Studio can also start a local server with an OpenAI-compatible API (by default on port 1234), so you can use it from code exactly like Ollama.
Option 3: llama.cpp (most control)
llama.cpp is the open-source engine that many local tools build on. It runs GGUF model files and supports NVIDIA (CUDA), AMD, Apple Metal, Vulkan and plain CPUs.
Install a prebuilt release from its GitHub page or with a package manager, then download a GGUF file from Hugging Face. To chat in the terminal:
llama-cli -m ./Qwen3-8B-Q4_K_M.gguf -ngl 99To start an OpenAI-compatible server with a built-in web chat page:
llama-server -m ./Qwen3-8B-Q4_K_M.gguf -ngl 99 -c 16384 --port 8080-ngl 99 puts all layers on the GPU (lower it to offload some layers to the CPU if the model does not fit), and -c sets the context length. llama.cpp can also download models directly with the -hf option and a Hugging Face repository name.
Which model should you start with?
- General chat on 8 to 12 GB: an 8B model such as Llama 3.1 8B or Qwen3 8B.
- Better quality on 16 to 24 GB: a 12B to 32B model such as Gemma 3 12B/27B or Qwen3 14B/32B.
- Coding: coding-tuned models such as Qwen Coder variants.
- Reasoning: models with a "thinking" mode, which give better answers to hard questions but produce more tokens.
New open-weight models appear constantly, so check current leaderboards and the Ollama library before you choose. Start with 4-bit (Q4_K_M) and move to 5-bit or 8-bit if you have spare memory.
Tips for better speed
- Keep the whole model on the GPU. Offloading to system RAM can make generation many times slower.
- Use a smaller quantization if you are short on memory; speed depends mainly on memory bandwidth.
- Do not set the context longer than you need; the KV cache uses extra memory.
- Update your GPU drivers and the tools themselves, as performance improves often.
Frequently asked questions
Can I run an LLM without a GPU?
Yes. Small models (1B to 4B) run acceptably on a modern CPU with 16 GB of RAM, but larger models are slow.
Is running an LLM locally free?
The software and open-weight models are free to download. Check each model's licence for commercial use.
Is Ollama or LM Studio better?
Ollama is simpler for terminal users and developers; LM Studio is easier if you prefer a graphical interface. Both use llama.cpp-based engines under the hood, and LM Studio also supports Apple's MLX on Macs.
Are local models as good as ChatGPT?
The best open models are strong, but the largest ones need data center hardware. Models that fit on a single consumer GPU are very capable for many tasks but generally behind the top cloud models.
Leave a Reply