Choosing Where to Store Conversation State

The situation

A team runs a customer-facing chat assistant on Amazon Bedrock. Each user turn is a stateless model call. The runtime holds no memory between requests, so the application gathers the running transcript and any session facts (the user’s plan, the current basket, what the assistant already asked) and hands the whole context back to the model on every turn. Right now that state lives in a process-local dictionary keyed by session id. It worked in the prototype. It falls over the moment there is more than one container behind the load balancer, because the next turn lands on a different instance and the transcript is gone.

The traffic is spiky. A quiet afternoon is a few sessions; a promotion takes it to thousands of concurrent conversations. Some run thirty or forty turns, with users sending messages a couple of seconds apart. Conversations should survive a container restart mid-chat, but they do not need to live forever. After a day of inactivity the session is dead, and keeping it is a privacy liability, because the transcript holds names, addresses, and order details. Product also asks that the assistant remember a returning user across sessions (“last time you asked about the annual plan”). That is a different kind of memory from the in-flight transcript.

Running a database for its own sake has no appeal here, and neither does hand-building a memory layer AWS already offers as a service. The decision underneath is the same either way. Where should conversation state live, given how durable it has to be, how fast it has to be read and written, and how much of it the team is prepared to operate?

What actually matters

Conversation state is not one thing, and splitting it is the first useful move. Short-term, in-flight state is the current transcript and the session’s working facts. It is read and written on every turn, and worthless once the conversation ends. Long-term memory is durable facts or summaries about a user that outlive the session and get retrieved when they return. The two have nearly opposite storage profiles, and serving both from one store is where these designs go wrong.

For the in-flight state, the dividing property is durability against latency. Every turn is a read-modify-write of the session: fetch the context, append the new turn, call the model, write it back. Tens of milliseconds on that round trip disappear next to the model’s own latency. An in-memory store is the fastest option and, without durability configured, the least safe, because a node failure can take the live conversation with it. So ask how bad losing a conversation mid-flight actually is. For a casual chat it is an annoyance. For a booking or a support case with a transaction attached it is a real failure, and that argues for a durable store even at some latency cost.

Scale and cost shape form the second axis. The load is spiky and unpredictable, so a store that scales with traffic and bills per request fits better than one provisioned for peak and paid for at trough. Both shapes are now available in both families. DynamoDB on-demand charges per request unit and scales to zero. ElastiCache Serverless charges for data stored per GB-hour plus ElastiCache Processing Units, while a node-based cluster is billed on the nodes whether or not anyone is talking to it.

Expiry and privacy are the same concern from two directions. The state is transient by nature and sensitive by content. It should expire without a cleanup job that might not run, and it should be encrypted and access-controlled the whole time it exists. One detail decides how far that gets you. DynamoDB’s time to live is a background process, not a scheduled delete: AWS documents removal as happening within a few days of the expiry timestamp, and expired items still appear in reads and still count toward storage until the sweep reaches them. TTL keeps the table from growing without bound. Reads have to filter on the expiry attribute, and a firm retention deadline needs an explicit delete.

The last axis is build against buy. Everything above assumes the team assembles the memory layer: pick a store, key it by session, manage expiry, and write the read-modify-write loop. The managed alternative is AgentCore Memory. It stores turn-by-turn events within a session and extracts long-term records across sessions, with no store to stand up, around a reasoning loop that stays the team’s own. The trade is control and portability. It suits an agent-shaped assistant whose memory needs match what the service extracts. It fits badly when the state model is unusual, or when governance requires the data to sit in the team’s own stores.

And the cross-cutting one: long-term memory is a retrieval problem, not a session problem. Durable facts and summaries are stored to be searched later, often by meaning rather than by key. So long-term memory usually lands in a separate durable or vector store, queried when the user comes back, rather than in the hot per-session table.

What we’ll filter on

1. State lifetime: in-flight transcript that dies with the session, or durable memory that outlives it?
2. Durability: how bad is losing a live conversation on a node failure?
3. Latency and turn rate: relentless high-frequency chat, or occasional bursts?
4. Scale and cost shape: does the bill track spiky per-request traffic, or a provisioned cluster?
5. Expiry and privacy: how does the state get deleted, and how firm is the deadline?
6. Build against buy: does managed agent memory fit, or does the team need to own the store?

The landscape

Amazon DynamoDB

A serverless, fully managed key-value and document store. Key the table by session id, hold the transcript and session facts in an item, and read-modify-write it each turn. AWS replicates the data across three Availability Zones by default and documents single-digit millisecond performance at any scale. On-demand capacity bills per request unit and scales to zero, which suits spiky traffic. The default per-table ceiling is 40,000 read and 40,000 write request units per second, adjustable on request.

Time to live deletes items whose timestamp attribute has passed, without consuming write throughput. That deletion is asynchronous, typically within a few days, so filter expired items out of reads rather than trusting the sweep to be prompt. DynamoDB Accelerator (DAX) sits in front as a read cache where one path needs microsecond reads.

Amazon ElastiCache

A managed in-memory store running Valkey, Redis OSS, or Memcached, as either a serverless cache or a node-based cluster. Reads are microseconds because the data sits in RAM, which suits high-turn-rate chat. Valkey and Redis OSS carry native key expiry, so per-session TTL comes with the engine.

Durability used to be the dividing line, and that line has moved. Node-based ElastiCache for Valkey now supports durability through a Multi-AZ transactional log. Synchronous writes are designed for zero data loss at single-digit millisecond write latency; asynchronous writes keep microsecond write latency and risk up to ten seconds of uncommitted data on a failure. Both options keep microsecond reads. Durability is a node-based feature, so a serverless cache, or a cluster with durability switched off, remains a cache that can lose the live conversation.

Amazon MemoryDB

A Valkey- and Redis OSS-compatible in-memory database, durable by design. AWS documents microsecond read and single-digit millisecond write latency, with data stored across multiple Availability Zones in a Multi-AZ transactional log for fast failover, recovery, and node restart. It is built to serve as a primary database rather than a cache, so one cluster covers both roles. The same engines bring the same native key expiry. It costs more than a plain cache.

AgentCore Memory

The managed option, part of Amazon Bedrock AgentCore. Short-term memory stores raw turn-by-turn events under a session id, retained for a duration set when the memory resource is created, up to 365 days. Long-term memory comes from the strategies attached to that resource; the built-in ones cover summarisation, user preferences, and semantic facts. Define no strategy and the resource keeps raw events only, extracting nothing. Extraction and consolidation run asynchronously in the background. Namespace templates such as /users/{actorId}/preferences/ scope records so one subscriber’s memory stays separate from another’s, and a customer-managed KMS key encrypts the store.

The trade is control and portability. The memory model is what the service extracts, and the data lives inside it rather than in your own tables.

A separate durable or vector store for long-term memory

Whatever holds the live transcript, durable cross-session memory (facts, preferences, running summaries) is usually kept apart. It is read on return rather than on every turn, and often searched by meaning. That can be a DynamoDB table of per-user facts, or a vector store such as an Amazon OpenSearch Serverless vector search collection or Aurora PostgreSQL with pgvector. Keeping it separate holds the cold, occasionally-read memory off the hot per-session path.

Evaluation

Side by side

Read against this scenario, the table splits three ways. The live transcript needs a durable per-session store with automatic expiry, which is DynamoDB unless the turn rate is high enough and the loss harsh enough to justify a durable in-memory cluster. The “remember me next time” feature is long-term memory, so it belongs in a separate store or in the platform’s managed memory. And the whole thing collapses into far less code if the assistant is agent-shaped and AgentCore Memory covers what it needs.

The solution

DynamoDB is the default for the in-flight transcript here, and the scenario lines up with it point by point. The traffic is spiky, so on-demand capacity that bills per request beats a cluster sized for peak. The conversation must survive a container restart, so a store replicated across three Availability Zones beats a cache that can lose the session with a node. Single-digit millisecond latency sits well under the model call, so durability does not show up in perceived latency. Key the table by session id and keep the item small, trimming or summarising long transcripts rather than letting one item grow without bound. Encryption at rest is on by default. Set the TTL attribute, filter expired items out of reads, and issue an explicit delete where the retention deadline is firm, because the sweep runs within days rather than on the second.

The in-memory pick needs more care than it used to. A node-based ElastiCache for Valkey cluster with synchronous durability holds microsecond reads and a Multi-AZ log, which removes the old objection that a cache drops your session. MemoryDB offers the same durability as a database rather than a cache, sized and billed as a cluster. Either is worth reaching for only when the turn rate is high enough that millisecond writes show, and losing a mid-flight conversation is a real failure. If turns are seconds apart, DynamoDB’s latency already sits under the model call, and the in-memory speed changes nothing a user sees.

AgentCore Memory is the build-against-buy pick, and it deserves a look before any store goes up. Short-term context within a session and long-term recall across sessions both come from configuration. That removes the store, the expiry management, and the read-modify-write loop. Attach at least one memory strategy, because a resource with none keeps raw events and extracts nothing; the failure is silent, and the assistant greets a returning subscriber as a stranger. Set the event retention to match the privacy stance too, since the default is not a day. Build it yourself instead when the assistant is not agent-shaped, when the memory model needs something the service does not extract, or when governance requires the personal data to sit in the team’s own stores under their existing retention and audit tooling.

Long-term memory is really a separate decision. The returning-user feature is not the live-transcript problem, and stapling it to the hot per-session table is a mistake. It is read on return, not every turn, and it is often searched by meaning rather than by exact key. So it lands in its own durable store: a per-user DynamoDB table for plain facts, or a vector store when recall is semantic and the assistant needs the most relevant past context rather than all of it. It is still personal data, so the same encryption, access control, and a real retention limit apply.

Across every option, the privacy posture is not optional. Conversation state holds PII by default. Encrypt it at rest and in transit, scope access tightly with IAM, and set a retention limit on the long-term memory as well as on the transcript. The store decides latency and cost. How it is governed decides whether a transcript full of names and addresses becomes a breach.

Worked example

The team splits the assistant’s memory in two and routes each half to the store that fits.

In-flight transcript. Turns arrive a couple of seconds apart, spiking to thousands of concurrent sessions during a promotion, and a conversation with a booking attached must survive a container restart. Seconds-apart turns mean DynamoDB’s single-digit millisecond latency already sits under the model call, so an in-memory store adds nothing a user would notice. The spiky load makes per-request billing the right cost shape. The pick is a DynamoDB table keyed by session id, with the transcript as an item, encryption at rest, and a TTL attribute set a day after the last turn:

Table: chat_sessions
  session_id   (partition key)
  transcript   (list of turns, trimmed to the last N + a summary)
  session_facts(map: plan, basket, pending_question)
  expires_at   (number, epoch seconds; TTL attribute)

Each turn: GetItem(session_id) -> append turn -> PutItem with
expires_at = now + 86400.

Reads filter on expires_at > now. The TTL sweep removes expired
items within a few days of the timestamp rather than at it, and
until then an unfiltered read still returns them.

If load-testing shows one read path needs microsecond latency, DAX goes in front without changing the durable table. If turns arrived milliseconds apart and losing a live booking were unacceptable, a durable in-memory cluster would be worth evaluating. That is not this case.

Cross-session memory. “Last time you asked about the annual plan” is long-term memory, read only when the user returns and best matched by relevance. It goes in a separate store: a per-user vector collection holding short summaries of past conversations, queried on the user’s return to pull the most relevant prior context into the opening turn. It never touches the hot per-session table. It carries its own encryption and retention policy, and it is populated by summarising a session as the in-flight transcript expires.

Had the assistant been built on AgentCore from the start, both halves could have come from managed memory: short-term events within the session, long-term records across sessions, with no table and no TTL to operate. The team kept their own stores because governance required the transcript’s PII to stay where their audit and retention tooling already reaches. That is a build-against-buy call made on a real constraint, not a reflex.

What’s worth remembering

1. Split conversation state before choosing a store: the in-flight transcript and the long-term memory have nearly opposite storage profiles.
2. DynamoDB is the common default for the durable per-session transcript, with three-AZ replication, on-demand billing that tracks spiky traffic, and a TTL attribute that clears dead sessions.
3. DynamoDB’s TTL is a background sweep, documented as deleting within a few days of expiry, so filter expired items out of reads and delete explicitly when a retention deadline is firm.
4. If turns are seconds apart, DynamoDB’s millisecond latency already sits under the model call, and an in-memory store changes nothing a user sees.
5. ElastiCache for Valkey now supports durability on node-based clusters through a Multi-AZ transactional log, so “it is a cache, so it loses data” no longer separates it from MemoryDB on its own.
6. AgentCore Memory extracts long-term records only through the strategies you attach; with none attached, the resource keeps raw events and extracts nothing.