Designing Short-Term and Long-Term Memory for a Bedrock Chat Assistant

The situation

A product team is building an AI support assistant for a mid-sized SaaS company. The assistant handles first-line queries, billing, account access, feature questions, refund requests, and escalates to human agents when it can’t. Measured over six weeks of closed beta:

• Average conversation length: 15 turns, ranging from five to past thirty.
• Return rate: 40% within 30 days. Median return gap eleven days; roughly half reference something from a previous thread, “did the refund you mentioned go through?”, “I’m still seeing the login error you helped me with last week”.
• Tool useTool useLetting an LLM call structured functions you’ve defined – search, calculator, database query, API call – instead of trying to do everything in text.: three or four per conversation. Account lookups, subscription checks, ticket creation.
• Platform: Bedrock. Nothing self-hosted.
• Team: two backend engineers, one front-end, no dedicated ML-ops.
• Compliance: GDPR. Conversation content is personal data; deletion-on-request has to be clean, retention has to be bounded.

What actually matters

“Memory” is two problems, not one. The first is keeping a single conversation coherent: turn fifteen has to know what happened at turn two. The second is recognising a returning user: someone who comes back eleven days later should land on a bot that already knows about their open refund, not one that asks them to retype it. Build both with one mechanism and you usually get one that does neither well, because the two pull in different directions. In-conversation memory has to be correct on every turn and fails loudly when it isn’t, which makes it backend plumbing. Cross-visit memory can be approximate, but it has two failure modes that are worse than approximate, which makes it product policy with engineering behind it.

Those two cross-visit failures are worth naming, because they set the privacy bar. Surfacing someone else’s conversation as if it were this user’s is a wrongful-disclosure incident: a stranger’s refund thread pulled up against this user’s login question. Failing to surface this user’s own open refund when they ask about it is milder, a trust dent rather than a breach, but still a product bug. Avoiding the first means per-user isolation has to be airtight, and it can’t be talked out of place by prompt injectionPrompt injectionAn attack where untrusted text the model is processing tries to override the instructions you actually gave it.. Avoiding the second means retrieval has to work on short, fragmented conversation text, which is exactly what document-retrieval tooling is bad at.

GDPR sets the next bar. When a user asks to be forgotten, every trace of their conversations has to go, cleanly and provably. A design where deletion cascades across four stores is one that eventually fails an audit. Aim instead for one delete call per store, each scoped to an identifier the application already holds. A single opaque per-user key deletes cleanly; per-turn vectors scattered through a shared index behind metadata filters can be made to work, but they’re far harder to stand behind when someone asks you to prove the data is gone.

Then there’s the team: two backend engineers, no ML-ops. Anything that scales with conversation volume is a liability by year two. A summarisation cron firing an LLMLLMA neural network trained to predict the next token in a sequence, large enough that it generalises to tasks it wasn’t explicitly trained for. call on every session close brings its own eviction policy, retention TTL, and retry logic, all of it infrastructure to own and operate. A managed option that does the same job behind a config flag frees that attention for the product. The thing you give up is flexibility, and this product never needs it. One seam is worth leaving open, though: a billing agentAgentA system that wraps an LLM with tools, memory, and a loop, so it can take multi-step actions toward a goal rather than just answering one prompt. and a support agent may one day need to share what they each know about the same user, so memory keyed to user identity rather than to a single agent instance is the easier thing to grow into.

What we’ll filter on

Five things the design has to deliver.

1. In-session coherence. Turn fifteen must be aware of turn two. The agent needs to see the relevant history of this conversation when it generates the next response.
2. Cross-session recall. A user returning eleven days later should land on a bot that can reasonably answer “what was the last thing we talked about?” without asking them to retype context. Not perfect replay, a usable summary.
3. Orchestration included. Fifteen turns with three tool calls per conversation means the assistant is planning, calling tools, observing results, and deciding what to do next. The memory solution has to live next to the orchestration, not compete with it.
4. Retrieval quality for conversational context. Pulling the correct fact from a past conversation is a different retrieval problem from pulling the correct paragraph from a product manual. Conversation data is short, interleaved, and context-dependent.
5. Operational overhead low enough for two backend engineers. No bespoke orchestration loop, no custom summarisation pipeline, no self-hosted vector database. GDPR delete has to be a button, not a project.

The landscape

Four plausible ways to build this.

Bedrock AgentCore’s managed memory. AgentCore is the operational layer for an agent whose reasoning loop you own, and memory is one of the capabilities it supplies. It covers both halves of the problem directly: short-term memory keeps a single conversation coherent across turns and reconnects, and long-term memory carries facts, preferences, and summaries across separate sessions so a returning customer is recognised. Neither half needs a datastore, a retrieval strategy, or a retention policy designed and operated by the team. The reasoning loop stays ours, which matters here only in that the memory capability attaches to it rather than replacing it.

DynamoDB-backed session store (build-your-own). Roll the memory layer yourself. A Lambda receives the user turn, reads conversation-so-far from DynamoDB (partition key sessionId, sort key turn timestamp), builds the prompt, calls the model, writes the response back, returns it. Cross-session recall is a second table keyed by user ID holding rolled-up state. Summaries come from a model call you write and schedule.

Bedrock Knowledge Bases for long-term recall. Dump transcripts or summaries into S3 and query at runtime for “what’s this user’s history?”. Chunking strategies assume a prose document; conversations are short, fragmentary, and relevance is keyed to who spoke and when. A chunk from someone else’s refund thread retrieved as “relevant” to this user’s login question is a correctness problem with a compliance problem stapled to it.

Custom vector store with conversation embeddings. Embed each conversation (or turn, or summary) with Titan Embeddings V2, store in OpenSearch Serverless or pgvector with per-user metadata, at session start query for the current user’s top-k most relevant past interactions. Full control of chunking granularity, metadata filtering, ranking. Also a second stateful system to own alongside DynamoDB.

Evaluation

Side by side

Option	In-session coherence	Cross-session recall	Retention and delete handled	Retrieval for conversation	Low ops
AgentCore managed memory	✓	✓	✓	✓	✓
DynamoDB session store (DIY)	✓	✓	✗	✓	✗
Knowledge Bases for past transcripts	✗	✗	✗	✗	✗
Custom vector store of conversation embeddings	✗	✓	✗	✓	✗

Matching the layers to the memory

The solution

The memory capability attaches to the agent and covers both layers, so neither the live transcript nor the cross-visit summary needs a store the team runs.

Short-term memory holds the conversation. Every turn in a session sees the turns before it, including tool calls and their results, across reconnects and across the gaps where a customer wanders off and comes back to the same widget. Turn fifteen sees turns one through fourteen because the platform is holding them, not because a Lambda read them out of a table and pasted them into the prompt.

Long-term memory holds the customer. At the close of a session the platform distils what happened into a durable summary scoped to that customer, and a later session opens with it already in context. This is the half that makes the returning-user experience work: two weeks later the assistant knows a refund is outstanding without anyone writing a summarisation job or scheduling it.

Two scopes, kept apart. The session scope is the conversation; the customer scope is the person. They are orthogonal on purpose, and both come from the application’s own authenticated context rather than from anything the model produced. A model that has been talked into naming a different customer does not get that customer’s memory, because the scope was fixed by the session the application established before the model saw a token.

Retention and deletion are configuration, not a project. A retention window ages summaries out on its own, and a delete scoped to one customer removes what was kept about them. That is the right-to-be-forgotten story handled by the platform, which is the difference between a compliance control and a compliance backlog item.

Limits worth naming. Long-term recall is scoped lookup, not semantic search across the customer base, so “find users with similar past experiences” is not something this gives you. Summaries are bounded, so very long histories lose detail over time. And memory follows the agent it is attached to; moving a customer from a support agent to a billing agent means passing what matters across at the application level.

When build-your-own earns a place

Two situations flip the decision toward DynamoDB and a hand-rolled memory layer.

When the retention rules are yours. A regulator that dictates exactly what is kept, in what form, for how long, and in which account is easier to satisfy against a table you control than against a managed window. The work is real, but so is the audit.

When state is richer than turns. Conversations are not the only per-session state; a shopping cart, a configured quote, a workflow status are none of them naturally turns. DynamoDB holds that directly, and the tools read and write it.

Neither flip applies to the two-engineer support bot. The retention rule is a number of days, and the state is conversational.

The hybrid worth knowing. Teams on managed memory often add a small DynamoDB or S3 store for structured cross-session facts, ticket numbers, subscription plan, last-known issue code, that the agent needs reliably regardless of whether they survived into a generated summary. Managed memory is the prose recall; the table is the structured one. A tool the agent calls to fetch it is the clean seam.

Why Knowledge Bases is the wrong shape for conversations

Four reasons.

Chunking doesn’t match. Knowledge Bases chunk documents, fixed-size (default ~300 tokens), hierarchical, or semantic, assuming nearby text is topically coherent. A conversation transcript has rapid speaker alternation, interleaved tool outputs, and short turns; a 300-token chunk spans three sub-topics and two speakers.

Retrieval relevance is topic, not speaker. A vector search for “refund” across a knowledge base of all transcripts will return high-similarity chunks from other users’ refund conversations. Compliance problem plus correctness problem. Metadata filtering by user ID helps but has to be attached at ingestion and is less flexible than a native vector store’s.

Summaries vs transcripts. Storing raw transcripts means retrieving fragments. The correct thing to retrieve is summaries, and generating those is the job managed long-term memory already does.

GDPR is harder. Deleting a user’s data means locating every chunk that contains their content in a service-managed index, then re-ingesting. A scoped delete against managed memory is one operation.

Knowledge Bases are correct for “what does our support policy say about refunds?”, a reference corpus shared across users. Wrong for “what did this user say yesterday?”, per-user conversational state.

Worked example

• An agent on AgentCore wrapping Claude Haiku 4.5. Latency-sensitive, cost-sensitive, and the reasoning bar for first-line support is low enough. Tools for account lookup, subscription status, and ticket create/query reach the existing internal APIs through the gateway. One Knowledge Base attached for the product documentation corpus, the policy memory, not the user memory.
• Short-term memory: on for the conversation. The session scope is the chat-widget session, rotated on an explicit “new conversation” or after an idle window.
• Long-term memory: scoped to the authenticated customer, with a ninety-day retention window. The scope is derived from the session the application established, never from anything the model supplied.
• Structured cross-session state: a small DynamoDB table keyed by customer, holding open ticket IDs, subscription tier, and last-issue-code. A GetUserContext tool lets the agent fetch it at conversation start when relevant.
• GDPR delete: a Lambda triggered by account closure deletes the customer’s long-term memory, deletes the DynamoDB row, and records an audit trail.
• Retention: summaries lapse after ninety days on their own.
• Monitoring: AgentCore observability traces each run, and a weekly anonymised sample of summaries is reviewed for quality.

No dedicated memory database, no custom summarisation cron, no per-user vector index. The memory plumbing comes with the platform; the reasoning loop stays ours.

What’s worth remembering

1. Short-term and long-term memory are different problems. Turn-level coherence within one conversation is session state; cross-visit recall is summary state. A single solution rarely does both well unless it was designed for both.
2. AgentCore’s memory capability covers both layers around a loop you still own: the live transcript within a session, a durable per-customer summary across them, with no store, no summarisation job, and no TTL to operate.
3. The session scope and the customer scope are orthogonal, and both come from the application’s authenticated context. Never let a scope be set by something the model produced.
4. Retention and scoped delete are configuration. That turns right-to-be-forgotten from a project into a setting, which is the whole reason to buy this rather than build it.
5. Knowledge Bases are for reference corpora, not conversational state. Chunking, retrieval relevance, and per-user isolation all work against using them for past-transcript recall.

The answer: AgentCore’s managed memory, short-term for in-conversation coherence and long-term for cross-session recall, scoped to the session and to the customer from the application’s authenticated context. Attach a Knowledge Base for product documentation, the reference corpus every customer shares. Add a small DynamoDB table of structured per-customer state (open tickets, subscription tier) behind a GetUserContext tool. Wire the scoped delete into the account-closure path for GDPR. The two engineers ship a memory system without operating a memory system, and the reasoning loop stays theirs.