Designing Short-Term and Long-Term Memory for a Bedrock Chat Assistant

The situation

A product team is building an AI support assistant for a mid-sized SaaS company. The assistant handles first-line queries, billing, account access, feature questions, refund requests, and escalates to human agents when it can’t. Measured over six weeks of closed beta:

• Average conversation length: 15 turns, ranging from five to past thirty.
• Return rate: 40% within 30 days. Median return gap eleven days; roughly half reference something from a previous thread, “did the refund you mentioned go through?”, “I’m still seeing the login error you helped me with last week”.
• Tool useTool useLetting an LLM call structured functions you’ve defined – search, calculator, database query, API call – instead of trying to do everything in text.: three or four per conversation. Account lookups, subscription checks, ticket creation.
• Platform: Bedrock. Nothing self-hosted.
• Team: two backend engineers, one front-end, no dedicated ML-ops.
• Compliance: GDPR. Conversation content is personal data; deletion-on-request has to be clean, retention has to be bounded.

What actually matters

“Memory” is two problems, not one. The first is keeping a single conversation coherent: turn fifteen has to carry what happened at turn two. The second is recognising a returning user: someone who comes back eleven days later should land on a bot that already has their open refund in context rather than one that asks them to retype it. Build both with one mechanism and you usually get one that does neither well, because the two pull in different directions. In-conversation memory has to be correct on every turn and fails loudly when it isn’t, which makes it backend plumbing. Cross-visit memory can be approximate, but it has two failure modes that are worse than approximate, which makes it product policy with engineering behind it.

Those two cross-visit failures are worth naming, because they set the privacy bar. Surfacing someone else’s conversation as if it were this user’s is a wrongful-disclosure incident: a stranger’s refund thread pulled up against this user’s login question. Failing to surface this user’s own open refund when they ask about it is milder, a trust dent rather than a breach, but still a product bug. Avoiding the first means per-user isolation has to be airtight, and prompt injectionPrompt injectionAn attack where untrusted text the model is processing tries to override the instructions you actually gave it. must not be able to move it. Avoiding the second means retrieval has to work on short, fragmented conversation text, which is exactly what document-retrieval tooling is bad at.

GDPR sets the next bar. When a user asks to be forgotten, every trace of their conversations has to go, cleanly and provably. A design where deletion cascades across four stores is one that eventually fails an audit. Aim instead for one delete call per store, each scoped to an identifier the application already holds. Records addressed by a per-user identifier delete cleanly; per-turn vectors scattered through a shared index behind metadata filters can be made to work, but they’re far harder to stand behind when someone asks you to prove the data is gone.

Then there’s the team: two backend engineers, no ML-ops. Anything that scales with conversation volume is a liability by year two. A summarisation cron firing an LLMLLMA neural network trained to predict the next token in a sequence, large enough that it generalises to tasks it wasn’t explicitly trained for. call on every session close brings its own eviction policy, retention TTL, and retry logic, all of it infrastructure to own and operate. A managed option that does the same job behind a config flag frees that attention for the product. The thing you give up is flexibility, and this product never needs it. One seam is worth leaving open, though: a billing agentAgentA system that wraps an LLM with tools, memory, and a loop, so it can take multi-step actions toward a goal rather than just answering one prompt. and a support agent may one day need the same record of the same user, so memory keyed to user identity rather than to a single agent instance is the easier thing to grow into.

What we’ll filter on

Five things the design has to deliver.

1. In-session coherence. Turn fifteen must have turn two in its context. The agent needs the relevant history of this conversation in the prompt that generates the next response.
2. Cross-session recall. A user returning eleven days later should land on a bot that can reasonably answer “what was the last thing we talked about?” without asking them to retype context. Not perfect replay, a usable summary.
3. Orchestration included. Fifteen turns with three tool calls per conversation means the assistant is planning, calling tools, observing results, and deciding what to do next. The memory solution has to live next to the orchestration, not compete with it.
4. Retrieval quality for conversational context. Pulling the correct fact from a past conversation is a different retrieval problem from pulling the correct paragraph from a product manual. Conversation data is short, interleaved, and context-dependent.
5. Operational overhead low enough for two backend engineers. No bespoke orchestration loop, no custom summarisation pipeline, no self-hosted vector database. GDPR erasure has to be a short list of scoped API calls against one service.

The landscape

Four plausible ways to build this.

Bedrock AgentCore’s managed memory. AgentCore is the operational layer for an agent whose reasoning loop you own, and memory is one of the capabilities it supplies. It covers both halves of the problem directly: short-term memory stores the turn-by-turn events of a single session, and long-term memory extracts facts, preferences, and summaries out of those events so a returning customer is recognised. Neither half needs a datastore or a retrieval engine the team designs and operates; retention for raw events is one required number. A memory is its own resource with its own identifier, so it is not welded to one agent.

DynamoDB-backed session store (build-your-own). Roll the memory layer yourself. A Lambda receives the user turn, reads conversation-so-far from DynamoDB (partition key sessionId, sort key turn timestamp), builds the prompt, calls the model, writes the response back, returns it. Cross-session recall is a second table keyed by user ID holding rolled-up state. Summaries come from a model call you write and schedule.

Bedrock Knowledge Bases for long-term recall. Dump transcripts or summaries into S3 and query at runtime for “what’s this user’s history?”. Chunking strategies assume a prose document; conversations are short, fragmentary, and relevance is keyed to who spoke and when. A chunk from someone else’s refund thread retrieved as “relevant” to this user’s login question is a correctness problem with a compliance problem stapled to it.

Custom vector store with conversation embeddings. Embed each conversation (or turn, or summary) with a Bedrock embedding model such as Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0, 8K-token context, configurable output dimensions), store in OpenSearch Serverless or pgvector with per-user metadata, at session start query for the current user’s top-k most relevant past interactions. Full control of chunking granularity, metadata filtering, ranking. Also a second stateful system to own alongside DynamoDB.

Evaluation

Side by side

Option	In-session coherence	Cross-session recall	Retention and scoped delete built in	Retrieval for conversation	Low ops
AgentCore managed memory	✓	✓	✓	✓	✓
DynamoDB session store (DIY)	✓	✓	✗	✓	✗
Knowledge Bases for past transcripts	✗	✗	✗	✗	✗
Custom vector store of conversation embeddings	✗	✓	✗	✓	✗

Matching the layers to the memory

The solution

One memory resource carries both layers, so neither the live transcript nor the cross-visit summary needs a store the team runs.

Short-term memory holds the conversation. The agent writes each turn with CreateEvent, tagged with a session identifier and an actor identifier, and reads the session back with ListEvents before composing the next prompt. Turn fifteen sees turns one through fourteen, including tool calls and their results, across reconnects and across the gaps where a customer wanders off and comes back to the same widget. The calls are yours; the table, the backups and the expiry sweep are not.

Long-term memory holds the customer. Attach one or more strategies when the memory is created and extraction runs in the background once events are written. The summary strategy condenses a session into topic-tagged records; the user-preference strategy pulls out stated preferences; the semantic strategy keeps facts. Records land in a namespace, and the summary strategy’s default is /strategy/{memoryStrategyId}/actor/{actorId}/session/{sessionId}/, so a later session reads a customer’s whole history by querying at the actor level of that path. Two weeks later the assistant has the outstanding refund in context, and nobody wrote or scheduled a summarisation job.

Two scopes, kept apart. The session scope is the conversation; the customer scope is the person. They are orthogonal on purpose, and both come from the application’s own authenticated context rather than from anything the model produced. If injected text names a different customer, retrieval still runs against the namespace the application supplied, so none of that customer’s records come back. AgentCore does not map sessions to users for you, which makes that mapping the backend’s job, and an IAM condition on bedrock-agentcore:namespace pins it. Poisoned content written into memory is a separate problem, caught by validating input before CreateEvent rather than by the scope.

Retention is one setting; erasure is a short list of scoped calls. eventExpiryDuration is required at creation and takes 3 to 365 days. It applies per event at write time, so raising it later leaves everything already written on its old expiry, and an expired event does not come back. Extracted memory records sit outside that timer. Erasing a customer means listing their sessions and events and calling DeleteEvent, then listing the records under their namespace and calling BatchDeleteMemoryRecords. Both are addressed by identifiers the application already holds, against one service.

Limits worth naming. RetrieveMemoryRecords runs a semantic search inside a namespace or a namespace path, with topK defaulting to 10 and capped at 100, so it reaches this customer’s history rather than the customer base; “find users with similar past experiences” wants a different index. Extraction is asynchronous, so something said a minute ago may not be a record yet, which is why the event log stays the source for in-session coherence. Summaries are condensed by construction, so long histories lose detail.

When build-your-own is the right call

Two situations flip the decision toward DynamoDB and a hand-rolled memory layer.

When the retention rules are yours. A regulator that dictates exactly what is kept, in what form, for how long, and in which account is easier to satisfy against a table you control than against a 3-to-365-day event expiry and records that persist until deleted. The work is real, but so is the audit.

When state is richer than turns. Conversations are not the only per-session state; a shopping cart, a configured quote, a workflow status are none of them naturally turns. DynamoDB holds that directly, and the tools read and write it.

Neither flip applies to the two-engineer support bot. The retention rule is a number of days, and the state is conversational.

The hybrid worth knowing. Teams on managed memory often add a small DynamoDB or S3 store for structured cross-session facts, ticket numbers, subscription plan, last-known issue code, that the agent needs reliably regardless of whether they survived into a generated summary. Managed memory is the prose recall; the table is the structured one. A tool the agent calls to fetch it is the clean seam.

Why Knowledge Bases is the wrong shape for conversations

Four reasons.

Chunking doesn’t match. Knowledge Bases chunk documents, and every strategy on offer assumes nearby text is topically coherent: the default splits at roughly 300 tokens on sentence boundaries, fixed-size lets you set tokens and overlap, hierarchical nests child chunks inside parents, semantic cuts where sentence embeddings diverge. A conversation transcript has rapid speaker alternation, interleaved tool outputs, and short turns; a 300-token chunk spans three sub-topics and two speakers.

Retrieval relevance is topic, not speaker. A vector search for “refund” across a knowledge base of all transcripts will return high-similarity chunks from other users’ refund conversations. Compliance problem plus correctness problem. Metadata filtering by user ID helps, but the values have to arrive as a .metadata.json file beside each source object at ingestion, which is a clumsier lever than a namespace set per write.

Summaries vs transcripts. Storing raw transcripts means retrieving fragments. The useful thing to retrieve is summaries, and generating those is the job managed long-term memory already does.

GDPR is harder. Deleting a user’s data means finding every source object holding their content, removing it, and running a sync so the incremental job drops those vectors from the index. Managed memory takes a list of record identifiers.

Knowledge Bases are correct for “what does our support policy say about refunds?”, a reference corpus shared across users. Wrong for “what did this user say yesterday?”, per-user conversational state.

Worked example

• An agent on AgentCore wrapping Claude Haiku 4.5 (anthropic.claude-haiku-4-5-20251001-v1:0, or the regional inference profile that fronts it). Latency-sensitive, cost-sensitive, and the reasoning bar for first-line support is low enough. Tools for account lookup, subscription status, and ticket create/query reach the existing internal APIs through Gateway, which fronts the Lambda and OpenAPI targets as MCP tools. One Knowledge Base attached for the product documentation corpus, the policy memory, not the user memory.
• Short-term memory: every turn written as an event. The session scope is the chat-widget session, rotated on an explicit “new conversation” or after an idle window.
• Long-term memory: a summary strategy plus a user-preference strategy, namespaced under the authenticated customer. The namespace is derived from the session the application established, never from anything the model supplied.
• Retention: eventExpiryDuration set to 90, so raw turns age out after ninety days while the extracted summaries stay until they are deleted.
• Structured cross-session state: a small DynamoDB table keyed by customer, holding open ticket IDs, subscription tier, and last-issue-code. A GetUserContext tool lets the agent fetch it at conversation start when relevant.
• GDPR delete: a Lambda triggered by account closure walks the customer’s sessions with ListSessions and ListEvents, deletes each event, batch-deletes the memory records under their namespace, deletes the DynamoDB row, and records an audit trail. A customer-managed KMS key covers the memory at rest.
• Monitoring: AgentCore observability traces each run, and a weekly anonymised sample of summaries is reviewed for quality.

No dedicated memory database, no custom summarisation cron, no per-user vector index. The memory plumbing comes with the platform; the reasoning loop stays ours.

What’s worth remembering

1. Short-term and long-term memory are different problems. Turn-level coherence within one conversation is session state; cross-visit recall is summary state. A single solution rarely does both well unless it was designed for both.
2. AgentCore’s memory capability covers both layers around a loop you still own: events for the live session, strategy-extracted records across sessions, with no store and no summarisation job to operate.
3. The session scope and the customer scope are orthogonal, and both come from the application’s authenticated context. Never let a namespace be set by something the model produced.
4. eventExpiryDuration is required and ages raw events out between 3 and 365 days; extracted records persist until DeleteMemoryRecord or BatchDeleteMemoryRecords removes them, so right-to-be-forgotten is a scripted pass over two scoped APIs.
5. Knowledge Bases are for reference corpora, not conversational state. Chunking, retrieval relevance, and per-user isolation all work against using them for past-transcript recall.

The answer: AgentCore’s managed memory, short-term events for in-conversation coherence and strategy-extracted records for cross-session recall, namespaced by session and by customer from the application’s authenticated context. Attach a Knowledge Base for product documentation, the reference corpus every customer shares. Add a small DynamoDB table of structured per-customer state (open tickets, subscription tier) behind a GetUserContext tool. Wire the scoped deletes into the account-closure path for GDPR. The two engineers ship a memory system without operating a memory system, and the reasoning loop stays theirs.