An agent makes six model calls to answer one question. Asking what temperature the agent runs at assumes there is a single answer. There are six, and a well-built agent may deliberately use different values for different calls.
The Short Answer
Temperature is a parameter of a model call, not a property of an agent.
The agent chooses the value and sends it with each request. It can send a different value every time. So an agent does not “have” a temperature any more than a program has a single function argument — it supplies one per invocation.
What Temperature Does
At each generation step the model produces a score for every possible next token. Those scores are converted into a probability distribution, and one token is sampled from it — the mechanism covered in how an LLM answers a query.
Temperature scales those scores before the conversion.
| Temperature | Effect on the distribution | Behaviour |
|---|---|---|
| 0 | Collapses to the single highest score | Always picks the most likely token |
| Low (0.1–0.3) | Sharpened — likely tokens dominate | Consistent, predictable |
| Around 1 | Unmodified | The model’s natural distribution |
| High (1.5+) | Flattened — unlikely tokens gain ground | Varied, increasingly incoherent |
Two points people get wrong. Temperature does not make a model more creative in any meaningful sense — it makes it more willing to select tokens it considers less likely. And it does not change what the model knows or believes; it changes how conservatively it samples from what it already computed.
Temperature 0 Is Not Deterministic
This is the part that surprises people, and it matters for anyone building reproducible systems.
Temperature 0 means greedy decoding — always take the highest-scoring token. That removes sampling randomness entirely. Yet identical requests can still produce different outputs.
| Source | Why it causes variation |
|---|---|
| Floating-point arithmetic | Parallel reductions sum in varying order; results differ in the last bits |
| Batch composition | Your request is batched with others; batch shape can change kernel selection |
| Hardware variation | Different GPU types produce slightly different numerics |
| Expert routing | In mixture-of-experts models, routing can depend on batch composition |
| Near-ties | When two tokens score almost identically, a tiny numeric difference flips the choice |
The last row is the mechanism that makes the others visible. Most of the time the top token wins by a clear margin and tiny numeric differences change nothing. Occasionally two candidates are nearly tied — and a difference in the final decimal places decides it. From that token onward the outputs diverge, sometimes substantially.
Temperature 0 gives you high consistency, not a guarantee. Systems that require exact reproducibility need to cache outputs rather than assume regeneration will match.
The Other Sampling Parameters
| Parameter | What it does |
|---|---|
| top-p | Sample only from the smallest set of tokens whose probabilities sum to p |
| top-k | Sample only from the k highest-probability tokens |
| frequency penalty | Reduces probability of tokens already used, scaled by count |
| presence penalty | Reduces probability of any token already used, regardless of count |
| seed | Requests reproducible sampling — best-effort, not guaranteed |
| stop sequences | Ends generation when specified text appears |
Temperature and top-p do related jobs — one reshapes the distribution, the other truncates it. Providers generally advise adjusting one, not both. Tuning them together makes effects hard to reason about, and a common outcome is two changes that partly cancel.
The seed parameter is worth understanding precisely: it makes the sampling step reproducible, but it does not eliminate the numeric sources of variation listed above. It improves reproducibility; it does not deliver it.
Why Agents Usually Run Low
Agent work is mostly not creative writing. It is selecting the right tool, constructing valid arguments, and producing parseable output.
| Agent task | Effect of higher temperature |
|---|---|
| Tool selection | More likely to pick a plausible-but-wrong tool |
| Argument construction | More malformed or invented parameter values |
| Structured output | More schema violations |
| Following instructions | More drift from stated format |
| Loop stability | Higher chance of erratic or repeating behaviour |
For the mechanical parts of an agent loop, variation is not a feature. Low temperature is the sensible default, and many production agents run at or near zero throughout.
Varying Temperature Across the Loop
The more interesting answer to the original question: a single agent can and often should use different values at different steps.
| Step | Suggested | Reasoning |
|---|---|---|
| Tool selection | 0 to 0.2 | One right answer; consistency is everything |
| Argument construction | 0 to 0.2 | Must satisfy a schema |
| Planning | 0 to 0.3 | Reliable decomposition beats novel decomposition |
| Extraction and classification | 0 | Deterministic-as-possible is the goal |
| Summarising retrieved content | 0.2 to 0.4 | Slight flexibility in phrasing, grounded in source |
| Drafting user-facing prose | 0.5 to 0.8 | Readable output; variation acceptable |
| Generating alternatives | 0.8 to 1.0 | Variation is the point |
The last row is the genuine case for high temperature in agents. When you deliberately want several different candidate approaches — to compare, rank or let a user choose — low temperature produces near-identical options and defeats the exercise.
This also pairs with the reflection pattern from agent prompt patterns: generate candidates at higher temperature, evaluate at low.
Reasoning Models Behave Differently
Models that perform extended internal reasoning often treat sampling parameters differently. Some fix temperature internally, some ignore the parameter, and some accept it only within a narrow range.
Passing temperature 0 to such a model may have no effect, or may be rejected. The practical implication is that sampling configuration is not portable across model families — a value tuned for one model should be re-validated rather than carried over.
Where Temperature Is Set
In an agent stack the value can be specified at several layers, which is a common source of confusion when the observed behaviour does not match the configuration.
| Layer | Note |
|---|---|
| Framework default | Frameworks often set a default you did not choose |
| Agent configuration | Applies to all calls unless overridden |
| Per-call override | What actually takes effect |
| Prompt template metadata | Some template systems store sampling config alongside the prompt |
Storing temperature with the prompt version is a sound pattern. A prompt and its sampling settings were evaluated together, so versioning them together means a rollback restores both.
When behaviour seems wrong, log the parameters actually sent rather than the ones you believe are configured. Framework defaults overriding intended values is a routine cause.
Anti-Patterns
- Raising temperature to fix poor output. If the model is getting things wrong, more randomness produces different wrong answers. The issue is usually the prompt, the tools or the retrieved context.
- Tuning temperature and top-p together. Effects interact confusingly and often partly cancel.
- Assuming temperature 0 is reproducible. It is consistent, not deterministic. Cache when you need exactness.
- Using one value for the whole loop when steps have genuinely different needs.
- Carrying settings across model families. Especially to and from reasoning models.
- High temperature with structured output. Constrained decoding keeps output valid, but higher temperature still degrades the quality of choices within the constraint.
Key Takeaways
- Temperature belongs to the model call, not the agent — the agent supplies it per request
- It scales token scores before sampling; it does not change what the model knows
- Temperature 0 is not deterministic — floating-point, batching and hardware cause variation
- Adjust temperature or top-p, not both
- Seed improves reproducibility but does not guarantee it
- Agents should run low for tool selection, arguments and extraction
- Vary it across the loop — high only where variation is the objective
- Reasoning models may fix or ignore the parameter; settings are not portable
Frequently Asked Questions (FAQ)
Q: Does an AI agent have a temperature setting?
Not as a property of the agent. Temperature is a parameter sent with each model call, so an agent making six calls can use six different values. What people usually mean is the default the agent sends, which is configurable and often overridden per call.
Q: What does temperature actually do in an LLM?
It scales the scores the model assigns to candidate next tokens before they are converted into probabilities. Lower values sharpen the distribution so likely tokens dominate; higher values flatten it so less likely tokens become reachable. It does not change what the model knows.
Q: Is temperature 0 deterministic?
No, though it is highly consistent. Greedy decoding removes sampling randomness, but floating-point arithmetic in parallel computation, batch composition, hardware differences and expert routing can still alter results — most visibly when two candidate tokens are nearly tied.
Q: What temperature should I use for an agent?
Low, typically zero to 0.2, for tool selection, argument construction, extraction and classification. Moderate values suit summarising and user-facing prose. High values are justified only where variation is the goal, such as generating alternative candidates to compare.
Q: Should I adjust temperature or top-p?
One, not both. Temperature reshapes the probability distribution while top-p truncates it, and tuning them together makes the combined effect difficult to reason about. Most providers recommend holding one at its default.
Q: Will a seed make my outputs reproducible?
It makes the sampling step reproducible, which helps considerably, but it does not remove numeric variation from parallel computation, batching or hardware. Treat it as improving reproducibility rather than guaranteeing it, and cache outputs when exactness is required.
Q: Can I use one temperature for the whole agent loop?
You can, and many agents do run low throughout. But steps have different needs — tool selection wants consistency while generating alternatives wants variation. Setting temperature per call rather than per agent is usually the better design.
Q: Do reasoning models support temperature?
Often differently. Some fix it internally, some ignore the parameter, and some accept only a narrow range. Sampling configuration is not portable across model families, so values tuned for one model should be re-validated rather than carried over.
Q: My agent gives poor answers — should I change the temperature?
Rarely the right fix. Higher temperature produces different wrong answers rather than better ones. Poor output usually traces to the prompt, tool descriptions or retrieved context. Check what was actually sent to the model before adjusting sampling.
Related Reading: