Short answer
Temperature controls how random a language model's output is. At each step the model scores every possible next token; a low temperature (0 to 0.3) makes it almost always pick the most likely token, giving focused, repeatable answers, while a high temperature (0.8 to 1.2) spreads the choice more widely for more varied, creative text. Top-k and top-p limit which tokens are allowed at all. For most uses, adjust temperature or top-p, not both.
How a model chooses the next word
A large language model does not output a word directly. At each step it produces a score (a "logit") for every token in its vocabulary, often 100,000 or more. These scores are turned into probabilities with the softmax function. Then one token is sampled from those probabilities, appended to the text, and the process repeats.
Sampling settings decide how that choice is made. You can see this live in our LLM sampling simulator.
Temperature
Before softmax, every score is divided by the temperature:
probability(token) ∝ exp(score / temperature)- Temperature below 1: differences between scores grow, so the top token dominates.
- Temperature = 1: the model's original probabilities.
- Temperature above 1: probabilities even out, so unlikely tokens get chosen more often.
- Temperature = 0: in practice means "always pick the most likely token" (greedy decoding).
Example: suppose the next word after "The GPU has 80 GB of" has these probabilities at temperature 1: "memory" 70%, "VRAM" 20%, "HBM3" 8%, "cheese" 2%. At temperature 0.3, "memory" rises to about 98%. At temperature 1.5, "cheese" more than doubles to about 5%, which is how high temperatures produce odd or incoherent text.
Top-k sampling
Top-k keeps only the k most likely tokens and samples among them. With top-k = 40, the model can never pick a token outside the top 40. It prevents rare nonsense, but a fixed k can be too strict when many words fit, or too loose when only one does.
Top-p (nucleus) sampling
Top-p keeps the smallest set of tokens whose probabilities add up to p. With top-p = 0.9, the model samples only from tokens that together make up 90% of the probability. When the model is confident, that might be one or two tokens; when many continuations are plausible, it might be hundreds. This adaptivity made top-p a popular default.
Min-p sampling
Min-p keeps tokens whose probability is at least a fraction of the top token's probability (for example 5% of it). It is popular in local LLM tools because it stays coherent even at higher temperatures.
Other settings you may see
- Frequency and presence penalties: reduce repetition by penalising tokens that already appeared.
- Repetition penalty: the local-model equivalent, common in llama.cpp and Ollama.
- Max tokens: caps the length of the response.
- Seed: fixes the random choices so results can be reproduced.
- Stop sequences: strings that end generation.
Recommended settings
| Task | Temperature | Top-p | Why |
|---|---|---|---|
| Coding, maths, data extraction | 0 to 0.2 | 1.0 | Accuracy and consistency matter most |
| Factual Q&A, summaries, RAG | 0.2 to 0.5 | 0.9 to 1.0 | Stay close to the sources |
| General chat and assistants | 0.6 to 0.8 | 0.9 to 0.95 | Natural but sensible |
| Creative writing, brainstorming | 0.8 to 1.1 | 0.95 | More variety and surprise |
| Generating many diverse options | 1.0 to 1.3 | 0.95 to 1.0 | Maximum diversity, check quality |
Always start from the model provider's recommended defaults; some models are tuned for specific values. Many reasoning models fix or ignore temperature because their training assumes particular settings; see What is a reasoning model?.
Common mistakes
- Lowering temperature to stop hallucinations: low temperature makes answers more consistent, not more correct. A model can confidently repeat the same wrong fact at temperature 0. Use RAG, sources and verification instead; see AI hallucinations.
- Changing temperature and top-p at the same time: their effects overlap and become hard to reason about.
- Expecting temperature 0 to be perfectly deterministic: small numerical differences on GPUs and batching can still change outputs.
- Using high temperature for code: it introduces typos and logic errors.
Where to change these settings
- ChatGPT, Claude and Gemini apps: usually fixed; you control style through your prompt.
- APIs and playgrounds: set
temperature,top_pand sometimestop_kin each request. - Local tools: Ollama, LM Studio and llama.cpp expose temperature, top-k, top-p, min-p and repetition penalty. See how to run an LLM locally.
Does sampling affect speed or hardware?
Very little. Sampling is a tiny calculation compared with running the model itself, whose speed depends mainly on memory bandwidth and model size. Generating several alternative answers, however, multiplies the compute.
Frequently asked questions
What is the best temperature for ChatGPT-style chat?
Around 0.7 is a common balance between natural and reliable. Use lower values for factual or technical work.
Is temperature 0 always best for accuracy?
It is best for consistency. For some reasoning tasks, generating several answers at a moderate temperature and picking the most common one can be more accurate.
What is the difference between top-k and top-p?
Top-k keeps a fixed number of tokens; top-p keeps a variable number based on cumulative probability. Top-p adapts better to how confident the model is.
Why is it called temperature?
The term comes from statistical physics, where higher temperature means particles move more randomly, just as higher sampling temperature makes token choices more random.
Sources and further reading
- Holtzman et al., The Curious Case of Neural Text Degeneration (nucleus sampling, 2019)
- Fan et al., Hierarchical Neural Story Generation (top-k sampling, 2018)
- Nguyen et al., Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs (2024)
- OpenAI, API reference: chat completions




Leave a Reply