You tell an assistant your name, and twenty messages later it still uses it. That looks like memory. It is not. The model forgot your name the instant it finished replying, and remembers it now only because your entire conversation was sent to it again, from the beginning, along with your latest message.
The Short Answer
No. An LLM is stateless. It holds nothing between requests.
Each time you send a message, the model receives a block of text, produces a response, and retains nothing. The next message is a completely fresh request to a model with no recollection of the previous one.
Conversation is an illusion maintained by the application, not a capability of the model.
How the Illusion Works
The application stores your conversation and resends all of it every single turn.
| Turn | What is actually sent to the model |
|---|---|
| 1 | System instructions + your message 1 |
| 2 | System instructions + message 1 + reply 1 + your message 2 |
| 3 | System instructions + messages 1–2 + replies 1–2 + your message 3 |
| 10 | System instructions + everything that came before + your message 10 |
By turn ten, the model is reading the whole transcript afresh. It appears to remember your name because your name is sitting there in the text it was just handed.
The practical test: start a new chat and ask about something from an earlier conversation. The model has no idea, because that transcript was not included.
What Happens When You Send a Query
The sequence inside a single request:
| Step | What happens |
|---|---|
| 1. Assembly | The application builds the full input — system instructions, conversation history, any retrieved documents, your message |
| 2. Tokenisation | Text is split into tokens, roughly word fragments, and converted to numbers |
| 3. Embedding | Each token becomes a vector representing meaning and position |
| 4. Forward pass | Vectors move through the network’s layers, where attention lets each token be interpreted in light of the others |
| 5. Prediction | The model produces a probability distribution over every possible next token |
| 6. Sampling | One token is selected from that distribution |
| 7. Repeat | The chosen token is appended and steps 4–6 run again, one token at a time |
| 8. Stop | Generation ends at a stop condition or length limit |
Step 7 is the part that surprises people. The model does not compose a response and then write it out. It generates one token at a time, each prediction conditioned on everything produced so far. There is no plan — the answer emerges token by token.
This is why streaming output looks the way it does. You are watching the actual generation process, not a pre-written answer being revealed gradually.
Why the Same Question Gives Different Answers
Step 6 involves choice. The model produces probabilities, and a sampling method picks from them.
| Setting | Effect |
|---|---|
| Temperature 0 | Always pick the highest-probability token — near-deterministic |
| Low temperature | Mostly predictable, occasional variation |
| High temperature | More varied, more creative, less reliable |
| Top-p sampling | Sample only from the most probable tokens up to a cumulative threshold |
For financial applications — extracting figures, summarising policy — low temperature is appropriate. Variation is not a feature when the answer should be the same every time.
The Context Window Is the Real Constraint
The context window is the maximum amount of text a model can process in one request. Critically, it holds everything at once:
- System instructions
- The entire conversation history
- Any retrieved documents
- Your current message
- The response being generated
Because history is resent each turn, a long conversation consumes the window. When it fills, something must give — and applications handle this in different ways.
| Strategy | What happens | Consequence |
|---|---|---|
| Truncation | Oldest messages dropped | The model genuinely forgets the start of the conversation |
| Summarisation | Earlier turns compressed into a summary | Gist retained, specifics lost |
| Sliding window | Only recent turns kept | Early context gone |
| Retrieval over history | Relevant past turns fetched as needed | Better, but requires RAG infrastructure |
This explains a familiar experience: an assistant that seemed to understand your requirement early on begins contradicting it later. The early instruction was dropped or compressed away.
Cost Grows With Conversation Length
A direct consequence of statelessness that catches teams out when they see their first bill.
API pricing is per token processed. Since the full history is resent every turn, the input grows continuously.
| Turn | Input tokens processed |
|---|---|
| 1 | Small |
| 5 | All four previous exchanges plus the new message |
| 20 | The entire transcript so far |
| 50 | Very large — most of the cost is re-reading old messages |
Cost scales roughly with the square of conversation length, not linearly. A fifty-turn conversation costs far more than five ten-turn conversations covering the same ground.
Some providers offer prompt caching, which reduces the cost of repeatedly sending an identical prefix such as a long system prompt. This is a billing and latency optimisation — it does not make the model stateful.
What “Memory” Features Actually Are
Products increasingly advertise memory across conversations. This is application engineering, not a model capability.
| Implementation | How it works |
|---|---|
| Stored facts | Details extracted and saved externally, then inserted into future prompts |
| Conversation summaries | Past sessions compressed and supplied as context |
| Retrieval over history | Past conversations indexed and searched when relevant |
In every case the mechanism is the same: something outside the model stores information and puts it back into the input. The model remains stateless throughout. This is the same architectural pattern as RAG — retrieval feeding the prompt.
Why This Matters in Finance
- Data is re-transmitted every turn. If a customer’s details appear in message two, they are sent again in messages three through thirty. For a regulated entity assessing where data flows, per-turn accumulation matters more than a single disclosure.
- Long conversations lose instructions. A compliance constraint stated at the start can be truncated away. Critical constraints belong in the system prompt, which is resent every turn, not in a passing message.
- The model does not learn from corrections. Correcting an error in one conversation changes nothing for the next. Persistent behaviour change requires prompt changes or fine-tuning.
- Reproducibility needs low temperature. Where an auditor may ask why a system produced a particular output, sampling variation is a liability.
- Logging is your responsibility. Since the model retains nothing, any audit trail must be built by the application. Under RBI’s framework, accountability sits with the institution — and you cannot evidence what you did not record.
Key Takeaways
- LLMs are stateless — nothing is retained between requests
- Conversation works because the application resends the full history every turn
- Queries are answered by generating one token at a time, with no plan formed in advance
- Different answers to the same question come from sampling, controlled by temperature
- The context window holds everything at once — instructions, history, documents, response
- When it fills, early messages are dropped or summarised
- Cost grows with the square of conversation length
- Memory features are external storage feeding the prompt, not model state
Frequently Asked Questions (FAQ)
Q: Does an LLM remember previous messages?
Not by itself. The model is stateless and retains nothing after responding. The application stores the conversation and resends all of it with each new message, which creates the appearance of memory while the model reads the transcript fresh every time.
Q: How does an LLM answer a query?
The application assembles the full input, which is tokenised and converted to vectors. Those pass through the network’s layers, producing a probability distribution over possible next tokens. One token is sampled, appended, and the process repeats — generating the response one token at a time.
Q: Why do I get different answers to the same question?
Because token selection involves sampling from a probability distribution rather than always picking the most likely option. Temperature controls this — at zero the output is near-deterministic, while higher values produce more variation.
Q: What is a context window?
The maximum text a model can process in one request. It holds everything simultaneously — system instructions, the full conversation history, any retrieved documents, your message and the response being generated. When it fills, earlier content must be dropped or compressed.
Q: Why does a chatbot forget what I said earlier in a long conversation?
Because the context window filled and the application dropped or summarised the earliest messages to make room. The model is not forgetting — that text is simply no longer being sent to it.
Q: Why does a long conversation cost more per message?
Because the entire history is resent every turn and pricing is per token processed. By turn fifty, most of what you are paying for is re-reading earlier messages. Cost scales roughly with the square of conversation length.
Q: How do AI products offer memory across conversations then?
Through external storage. Facts are extracted and saved, or past conversations summarised or indexed, then inserted into future prompts. The mechanism sits outside the model, which remains stateless — it is the same pattern as retrieval augmented generation.
Q: If I correct the model, does it learn from that?
Only within that conversation, and only while the correction remains in the context window. It changes nothing for future conversations. Persistent behaviour change requires modifying the prompt or fine-tuning the model.
Related Reading: