LLM Sampling Parameters Explained
What temperature, top-p, top-k and the penalties actually do to the probability distribution — and which combinations to avoid.
Sampling Parameters Guide
Every parameter that controls how LLMs generate text — temperature, top-p, top-k, penalties, and more.
Quick Presets
Parameter Reference
Scales the logits (raw model output scores) before applying softmax to convert them into a probability distribution. The most fundamental sampling parameter.
Controls randomness and creativity. Low values make output deterministic and focused; high values make output diverse and creative.
Near 0: deterministic — always picks the highest-probability token. Good for code, facts, structured data.
Near 2: very random — equal probability for many tokens. Risk of incoherence or hallucination.
- →0.0–0.3: factual Q&A, code generation, classification, data extraction
- →0.5–0.8: balanced conversations, summarisation, translation
- →0.9–1.2: creative writing, brainstorming, varied outputs
- →1.5–2.0: poetry, experimental text — monitor for quality degradation
- →Temperature and top-p interact: reduce one if you increase the other
Everything operates on one distribution
At each step the model produces a score for every token in its vocabulary. Softmax turns those scores into a probability distribution. Every sampling parameter is a transformation of that distribution before one token is drawn from it.
Temperature divides the scores before softmax. Below 1 it sharpens the distribution, concentrating probability on the top candidates. Above 1 it flattens it, giving unlikely tokens more chance. At 0 it degenerates to always taking the maximum — greedy decoding, fully deterministic.
Top-k keeps only the k highest-probability tokens and renormalises. A blunt instrument: k=40 keeps 40 candidates whether the distribution is confidently peaked on one token or genuinely spread across hundreds.
Top-p (nucleus sampling) keeps the smallest set of tokens whose cumulative probability reaches p. This adapts to the distribution's shape — where the model is confident the set is tiny, where it is uncertain the set is large. That adaptivity is why top-p is generally preferred over top-k.
Min-p, where available, keeps tokens above a fraction of the top token's probability. It behaves better than top-p at high temperature and is worth knowing about.
Settings by task
| Task | Temperature | Top-p | Reasoning |
|---|---|---|---|
| Classification, extraction | 0 | — | You want the single most likely answer, reproducibly |
| Code generation | 0–0.2 | 0.95 | Correctness matters far more than variety |
| Factual Q&A / RAG | 0–0.3 | 0.9 | Minimise fabrication; the answer should come from the context |
| Summarisation | 0.3–0.5 | 0.9 | Slight variation reads better without inventing content |
| General conversation | 0.7–0.8 | 0.9–0.95 | The common default; natural without being erratic |
| Creative writing | 0.8–1.0 | 0.95 | Variety is the point |
| Brainstorming variants | 1.0–1.2 | 0.98 | Deliberately widening the net; expect some noise |
Change one of temperature and top-p, not both. They interact multiplicatively and tuning both at once makes the effect of either impossible to reason about. Most practitioners fix top-p at 0.9–0.95 and adjust only temperature.
Sampling mistakes
Temperature 0 still produces different outputs
Cause:Greedy decoding is deterministic in principle, but batched GPU inference is not bit-reproducible — floating-point reduction order varies with batch composition. Mixture-of-experts routing adds further nondeterminism.
Fix:Treat temperature 0 as "as deterministic as available", not as a guarantee. If you need reproducibility, cache results rather than relying on the model to repeat itself.
High temperature produces incoherent output
Cause:Flattening the distribution enough gives genuinely bad tokens real probability, and one bad token derails everything after it.
Fix:Keep temperature at or below 1 for anything that must stay coherent. If you want variety, raise top-p slightly or sample several completions at a moderate temperature and pick.
Repetition penalty degrades quality
Cause:It penalises tokens for having appeared, indiscriminately. In code and structured output that punishes necessary repetition — keywords, indentation, JSON syntax.
Fix:Avoid repetition penalties for code and structured output entirely. Frequency and presence penalties are gentler; if the model is looping, the prompt is usually the real problem.
Reasoning model ignores temperature
Cause:Several reasoning models fix or restrict sampling parameters internally, since their training assumes a particular decoding regime.
Fix:Check the provider docs. Passing unsupported parameters is sometimes an error and sometimes silently ignored, which is worse.
About
This reference covers every sampling parameter that controls how LLMs generate text. Temperature scales the probability distribution over the vocabulary (higher = more random). Top-p (nucleus sampling) truncates to the smallest set of tokens whose cumulative probability exceeds p. Top-k limits sampling to the k most-probable tokens. Repetition Penalty reduces the probability of tokens that already appeared. Frequency Penalty is OpenAI's log-probability penalty proportional to token frequency. Presence Penalty is a flat penalty for any token that appeared at all. Min-P is a newer alternative that sets a minimum probability relative to the top token. Mirostat dynamically adjusts sampling to maintain target perplexity. Six quick presets (Precise, Balanced, Creative, Code, Chat, Story) let you jump to sensible configurations.
How to use
- 1 Click a Quick Preset to see recommended settings for your use case.
- 2 Expand any parameter to read its detailed description, effect, and tips.
- 3 Use the Low / High value boxes to understand each end of the parameter's range.
- 4 Note which providers support each parameter (OpenAI, Anthropic, Google, Ollama, etc.).
- Should I use temperature or top-p?
- Typically you adjust one or the other, not both. Temperature scales all probabilities; top-p truncates low-probability tokens. OpenAI recommends changing temperature OR top-p but not both simultaneously. For deterministic outputs (code, data extraction) lower temperature (0.0–0.3) is more reliable. For creative tasks, higher temperature (0.7–1.0) produces more variety.
- What temperature should I use for coding tasks?
- For code generation and factual Q&A, use temperature 0.0–0.2 and top-p 0.9. Low temperature makes the model pick high-probability (likely correct) tokens consistently. Very low (0.0) is deterministic — useful for reproducible outputs.
- What is the difference between repetition, frequency, and presence penalties?
- Repetition Penalty (Hugging Face / Ollama): divides logit by the penalty factor if the token appeared before — multiplicative. Frequency Penalty (OpenAI): subtracts a value proportional to how many times the token appeared — additive, linear. Presence Penalty (OpenAI): subtracts a flat value for any token that appeared at all — additive, binary.
- What is top-p vs top-k vs min-p?
- Top-p keeps only the smallest set of tokens summing to probability p (dynamic k). Top-k always keeps exactly k tokens regardless of probability gaps. Min-p keeps tokens whose probability ≥ p × (probability of top token), adapting to the model's confidence — more aggressive pruning when the model is certain.
In-depth guides
More in AI Tools
See all ai tools.