a conversation is something you resend
multi-turn feels native to build because it is list.append. every genuinely hard part of it lives somewhere else.
- query decontextualisation
- coreference in follow-up turns
- the stateless request model
- externalised session state
- partition key design
- prompt caching economics
- super-linear context growth
- audit trail vs conversational dependence
- latency budgeting under a serial hop
- semantic cache scope validation
- stopping rules
- Memory, multi-turn persistent memory, is perhaps very hard for retrieval systems. I cannot begin to think of ideas on how it is perfected at massive scale.
- A conversation has to outlive the process that answered it. Mine lives in one container’s memory, so it dies on restart and breaks the moment a second replica exists.
- A follow-up gives a retriever nothing to match on, and it makes answer N depend on answers 1 through N-1, which is bad when someone has to defend a number.
- And what a person expects is just the most natural, human sense of discussion. That expectation is the actual spec, and none of the parts underneath were built for it.
the appearance of continuity
Just because the UI shows the same thread, the appearance of continuity when we scroll past, one cannot assume perfect multi-turn awareness. The thread is a thing the client is doing. Each request is stateless, and the continuity you feel comes from resending history. Every time, in full.
turn 3, as it actually goes out
┌─────────────────────────────┐
│ user q1 │
│ assist a1 │ resent
│ user q2 │ resent
│ assist a2 │ resent
│ user q3 <- the new bit │
└─────────────────────────────┘
most of this request is a repeat- The retriever, unlike the model, does not get that array. It gets one string.
- Everything that made the follow-up intelligible was in the part you did not send it.
the follow-up has nothing in it to match on
Here is the pair that started this, and it is a completely ordinary way for a person to talk.
Analyst: “What was Company Y’s, Z’s cloud revenue in FY2025, and how do they describe it in ITEM 1B, ITEM 7 of the filing?”
Analyst: “And how did that compare to the year before? What do their competitors say?”
The first one is a retrieval engineer’s dream. It has companies in it, a fiscal year, a metric, and two named sections to anchor on. The second one, read on its own, is nearly content-free.
| what the second query offers a retriever | |
|---|---|
| named entities | none |
| metric | none, just “that” |
| time anchor | relative, “the year before” |
| section anchor | none |
| domain words | “competitors” |
It will still find something. Paragraphs about competition in general, from any company, any year. What bothers me is that nothing errors. You get a fluent, sourced, confident answer to a question nobody asked.
Therefore multi-turn systems truly need some powerful query contextualization pattern. A smaller LLM call that rewrites the follow-up into a standalone query, before the retriever ever sees it.
But is it even worth it?
Mostly yes, and the shape of the rewrite decides whether it helps or hurts.
One production team reports that over 60% of follow-up messages carry unresolved coreference or implicit context, so the degraded case is the majority of turns rather than a rare tail. But in the SIGIR 2025 LiveRAG Challenge, the DoTA-RAG team tried hypothetical-document expansion and reported flatly that “HyDE decreased both Correctness and Faithfulness.” They dropped it and spent the effort on hybrid dense-plus-lexical retrieval instead.
Rewrites that invent keywords or imagined documents tend to distort intent. The ones that pay are narrower, and only resolve what the pronouns point at. So the useful version is boring on purpose, and its whole job is to put the nouns back.
For filings I would add one constraint, and it is what I would get wrong first. A rewriter that summarises will cheerfully drop ITEM 1B and FY2025, because beside the semantic content those look like noise. They are the payload. The hop needs a must-preserve list. Identifiers, section labels, fiscal years, all passed through untouched.
the thread is list.append, and that is the easy half
messages = [
{"role": "user", "content": [{"text": q1}]},
{"role": "assistant", "content": [{"text": a1}]},
{"role": "user", "content": [{"text": q2}]},
]Multiple turns feels very native to build. Append the turn, send the array back. There is no hidden state on the model side. Each request is stateless and the illusion of continuity comes from the client resending history. LangChain’s ConversationBufferMemory is a wrapper around list.append.
Which is why “we support multi-turn” is nearly a free claim, and why it stopped meaning much to me. What decides whether it is true is where that list lives once the process holding it goes away.
As a beginner LLM systems developer, I have not cracked this yet. State lives in one Fargate task’s memory (well, i’m using this) so it dies on task restart and breaks the moment i run two tasks without session affinity.
resending has two bills. one is state, one is tokens
Luckily its a very small addition to fix this, using DynamoDB fixes this.
Small change, and the reasoning under it is the part worth keeping.
Picture the running task as an office with a filing cabinet in it. Turn one walks in and you write the conversation down. Turn two arrives, the load balancer sends it to whichever office is free, and that office has never met this user. Which conversations break is decided by routing, so it looks random and reproduces badly. Restart the task and the cabinet goes in a skip.
Sticky sessions pin a user to one office. That quietly turns a stateless service stateful. Load goes lopsided, deploys start dropping conversations, and one task dying takes its users’ history with it. Replicating state between tasks means writing a consensus protocol by accident.
The move is to take the cabinet out of the offices and put it in a records room all of them can reach. Once no task owns a conversation, tasks are interchangeable again and “restart” stops meaning “forget.” The state becomes somebody else’s job, which is the whole point of keeping a service stateless.
Why this particular database suits the shape:
- The access pattern is one key-value lookup. Given a session id, get its turns. No joins, no scans. That is the shape DynamoDB is best at, at any size.
- Partition key
session_id, sort keyturn_index. A conversation’s turns sit in one partition, in order, so reading the thread is a singleQuery. And it costs the same at ten sessions or ten million, because the partition is the unit of scaling rather than the table. - One item per turn, not one document per conversation. Appending is a single
PutItemwith no read-modify-write, so concurrent turns cannot clobber each other, history is never rewritten, and no item creeps toward the 400 KB ceiling. - TTL does the forgetting. Set an expiry attribute and expired turns are removed for you. No cleanup job to own.
Then the part I did not expect. Every turn re-sends the accumulated blocks, so the usual mitigation is a sliding window, where only the last k turns go to the model. With this layout the window is just Limit=k on the query you were already making. The shape the database wanted and the shape the token budget wanted are the same shape.
That curve is where prompt caching starts to pay, and the arithmetic is unusually easy. A cache write costs roughly 25% more than plain input; a read costs roughly 10% of it. So a cached prefix is behind after one use and ahead from the second. In a conversation that happens straight away, because the prefix you are re-sending is by definition the thing you already sent.
tokens billed at full price, per turn
no cache with cache
turn 1 ████ ████+ write
turn 2 ████████ ▌
turn 3 ████████████ ▌
turn 4 ████████████████ ▌
grows every turn prefix readsthe audit trail wants the opposite of a conversation
The good news is, i have something similar and stronger in value. An absolute audit trail. I assumed it was very cheap and useful for interesting statistics, so i developed stuff that writes every query, context, and response to a Parquet log in object storage. Independently auditable. Every answer traces to specific sentence IDs in specific filings.
A conversational thread makes answer N depend on answers 1 through N-1, which is what i feel like, what you don’t want when someone has to defend a number. Right?
Right, and the tension is real. A single-turn answer is a closed proof. Question, retrieved sentences, answer, one row of the log. A turn-eight answer that leaned on turn three drags a dependency chain behind it, and if turn three was subtly wrong, the log faithfully records eight confident answers sharing one defect. The CALMem work avoids recursive abstraction over history for the same reason: summarising erases the verbatim detail that made the record worth keeping.
There is a cheap way to keep both, and it happens at the logging layer. Log the rewritten standalone query beside the raw one, and the hop you added for retrieval accuracy becomes the thing that turns a context-dependent turn back into a self-contained, auditable question.
What is genuinely non-trivial is that follow-up questions break dense retrieval, so real multi-turn RAG needs a query-rewriting hop, and I wasn’t willing to add a serial LLM call to a pipeline I’m already optimizing for latency.
That instinct holds up, and the fix turned out to sit somewhere I had not looked. One team measured the rewrite step at more than 80% of their total RAG latency. They kept the hop and took the waiting out of it: race several small models at once, and if none answers within one second, fall back to the user’s raw message. Median latency went 326 ms to 155 ms. The fallback is the honest bit. A slightly worse query beats a stalled thread.
the shape they share
| the piece | looks like | actually is | what making it true costs |
|---|---|---|---|
| the thread | continuous | one array resent per turn | nothing. It is the interface |
| the follow-up | a question | a string with the nouns removed | a rewrite hop, narrow on purpose |
| session memory | list.append | state nobody durably owns | a key-value table and a partition key |
| the token bill | linear | super-linear in turns | a cache, and a Limit on a query |
| the audit trail | unaffected | a dependency chain | logging the rewrite beside the raw turn |
Every row is the same trick. Something reads as continuous because a client is doing the work of making it look that way, and the moment you need it to be true rather than apparent, you are buying a specific, separate system.
LLM Design perfection feels endless at that point. Once you do invest in multi-turn query hop fetches, conversation and history persisted perfectly, prompt caching, exact-hash query caching or threshold based caching (cosine similarity), end-to-end streaming, retrieval variants, concurrent retrievals, lexical + dense, citation groundings and data drilldowns, log observability, iterative self-improving gold tests, LLM-as judge + eval metrics, non-determinism analysis.. where do LLM projects 'end'? where, how do we justify the utility and timing budgets?
They end where you can name the decision the next feature changes.
That list is about five projects wearing each other’s clothes, and they justify themselves differently. Caching is arithmetic, and you can work out the break-even before writing any code. Streaming is perception, justified by what a person will tolerate. Evaluation is a claim about correctness, and the only item that tells you whether the others worked.
It feels endless because the items get sequenced by which is most interesting to build, rather than by which decision is currently unsupported. Interestingness tracks novelty, and novelty usually means a thing is not load-bearing yet.
The stopping rule I would defend is plain. Write down the decision the system exists to support. Someone has to defend a number in a meeting. Then ask which single missing piece most changes the odds that the number is right. Build it. Re-ask. When the honest answer is “none of them meaningfully,” you are done, and the leftovers become things you chose not to need.
Threshold-based semantic caching is the one I would be most careful with. It is cheap and clever, and on filings it can serve last year’s answer to this year’s question. That is a correctness risk taken on for a latency gain you may never have measured.
I do not have a finished version of any of this. What changed while writing it down is that “we support multi-turn” stopped sounding like a feature and started sounding like a bill with several line items, most of which are not the model. The thread itself is list.append. Everything hard is what you build so that appending to a list stays true after the process holding it dies, after the follow-up loses its nouns, and after somebody asks you, six turns later, where exactly that number came from.