Deleting a Subscriber's Data From a RAG System

The situation

A subscription business runs a support assistant on Amazon Bedrock. Subscribers ask it about deliveries, billing, and changes to their plan, and it answers from a Bedrock Knowledge Base built over support articles, resolved ticket threads, and free-text account notes. The model serving traffic is a custom one, fine-tuned on two years of ticket transcripts so it answers in the house voice.

A subscriber has asked for their data to be deleted. Legal has accepted the request, started a thirty-day clock, and asked engineering the one question that matters operationally: where is it, and when will it be gone?

The audit comes back with nine places. The source documents in Amazon S3, ticket threads and account notes. The Bedrock Knowledge Base index and the vector store behind it, holding the embeddingsEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. of those documents. The conversation state in Amazon DynamoDB, one item per turn. The model invocation logs in Amazon CloudWatch Logs, and the second copy of them in an S3 bucket that compliance put under S3 Object Lock last year. A response cache in front of the model. The golden evaluation set, sampled from real traffic, which includes four of this subscriber’s questions. And the fine-tuning dataset that produced the custom model now serving every request.

Nobody on the team disputes that the first item has to go. The argument is about the other eight.

What actually matters

Deletion is a property of a system, not an operation on a store. Nine places came back because a record propagates two different ways, and the two need different answers. A copy is the same content sitting somewhere else: the S3 objects, the DynamoDB items, the log lines, the evaluation set, the tuning dataset. A derivative is something computed from the content that carries it forward in a different shape: the embeddings in the vector index, the cached responses, and the weights of the fine-tuned model. Copies can be found and removed. Derivatives usually cannot be edited at all, only rebuilt or destroyed wholesale, and the further a derivative is from the source the more expensive rebuilding gets. An embedding is a lossy but reversible-enough numeric representation of a passage of text, and if the passage was personal data then the vector is personal data too.

The second property worth weighing is whether the deletion is provable. Intent is not evidence. A store that supports a delete of one subject’s record gives you an API response, a CloudTrail entry, and a timestamp, and those three together are a receipt. A store that expires content on a schedule gives you a policy and a future date, which answers a retention obligation and does not answer an erasure request. Those are different instruments, and the same mechanism can serve either one, which is why they get confused. Decide up front which surfaces carry a receipt and which carry a schedule, because the scheduled ones cannot go faster than the schedule.

Third, the clock and the conflicts. The thirty days starts at the request, and every surface has its own latency floor: milliseconds for some, a full re-sync for others, a retraining cycle for the model. Worse, some surfaces exist precisely to prevent deletion. A bucket under an Object Lock retention period returns an access-denied error, and that is the control working correctly rather than failing. The conflict between a retention obligation and an erasure obligation has a legal answer as well as a technical one, and it has to be resolved when the bucket is designed. Left to request time, an engineer discovers on day nineteen that the mechanism cannot comply.

Fourth, what the deletion does to the system that remains. Removing one subscriber should not shrink the corpus that answers everybody else, invalidate the evaluation set that gates every release, or force a retrain per request. A deletion path that degrades the product gets skipped the third time it runs, which is a worse outcome than a slow one.

What we’ll filter on

1. Completeness: does the strategy reach every surface, including derivatives, or does it leave a residue somebody has to remember?
2. Latency to comply: how long from the request landing to the data being unreadable, and is that inside the regulator’s window?
3. Provability: does it produce evidence a third party can check, or only an assertion that it was done?
4. Cost: what does it cost to set up before any request arrives, and what does each request cost once it does?
5. Effect on the system that remains: does the product degrade, and does the deletion have to be repeated as new derivatives are built?

The landscape

Three strategies cover this ground. They are not mutually exclusive, and the strongest designs use two, but they behave differently enough on the filters above to be worth weighing separately.

Delete in place

Find every copy and remove it, surface by surface, driven by a subject identifier. This is the one every team reaches for first. It works well wherever the store has a single-record delete and an index that can find the record from a subject identifier: DeleteObject against S3, a DeleteItem per conversation turn in DynamoDB, an eviction from the cache, a row pulled out of the evaluation set. It degrades badly wherever the store cannot address a single subject, wherever the content is a derivative, and wherever a retention control forbids the delete.

The dependency this strategy carries is a subject index: a mapping from subscriber to every object key, item key, log stream, and dataset row that mentions them. Without it, deletion becomes a scan, and a scan over a document corpus for one person’s data is both slow and unreliable. Building that index is work you do at ingestion, not at request time.

Crypto-shred

Encrypt each subject’s data under a per-subject AWS KMS key, and delete the key when the subject asks. The ciphertext stays where it is, on every surface, in every backup, inside every replica, and stops being readable everywhere at once, including in the copies you forgot about.

One ScheduleKeyDeletion call covers surfaces you never enumerated, and the mandatory waiting period is itself a safety net against a mistaken request: seven to thirty days, thirty if you do not specify, and cancellable throughout. The evidence is strong too. Key deletion is a CloudTrail event with an actor and a timestamp, and AWS documents data encrypted under a deleted key as unrecoverable, which is a stronger claim than a delete receipt. Holding the keys yourself is what makes this available at all.

The costs are real. A key per subject is one you pay for monthly and a quota you can hit, so it usually means grouping subjects into cohorts or using a per-subject data key wrapped by a shared customer-managed key and stored in a table you can purge. Crypto-shredding is also all or nothing at the granularity of the key: anything else encrypted under it goes with it, so the encryption boundary and the deletion boundary have to be designed as the same boundary. And it does nothing for a derivative computed from plaintext and stored elsewhere, which is what an embedding and a set of model weights are.

Never store

Push the work forward to ingestion so that nothing subject-identifying reaches the durable surfaces in the first place. If the corpus, the logs, and the tuning dataset hold no identifiers, there is nothing to find later and the deletion request is answered by pointing at the pipeline.

Two families of technique sit under this. Masking replaces an identifier with a token or a placeholder while keeping the shape of the text, usually reversibly through a token vault so the application can still resolve a name when it legitimately needs one. Anonymisation removes the link to the individual irreversibly: generalising a delivery address to a suburb, dropping a customer reference entirely, replacing a name with a role. The distinction is whether a vault exists that can put the identity back. If one does, the vault is now the deletion surface, and purging the vault entry crypto-shreds by another route. If one does not, the data has left the scope of the request and there is nothing to delete.

Amazon Comprehend does the detection for both, over English and Spanish text. It locates entities in real time or as a batch job, and only the batch job redacts, so a pipeline that rewrites documents runs asynchronously. Running it over prompts and logs before they persist is the same mechanism applied at a different point in the pipeline. Bedrock Guardrails cover the runtime path, blocking or masking sensitive entities in a prompt or a model response, with one exception that matters here: masking does not reach the invocation logs, where the logged input is the original request whatever the guardrail did to what the model saw.

Bedrock’s own retention settles a related question. It is a mode set per account or per project, content is not shared with model providers, and at the strictest setting nothing from an inference request reaches durable storage. Some newer models require a more permissive mode as a condition of access, and under those the prompt and the completion are held inside the AWS boundary for up to thirty days so AWS can review them. Read the mode rather than assume no copy is kept. None of this manages the copies you made yourself, which is all nine surfaces above.

What never store gives up is utility. A support assistant that cannot see who it is talking to answers worse, and an anonymised tuning dataset teaches the model less. Make that trade deliberately, rather than discovering it after the corpus is built.

Evaluation

Side by side

Strategy	Reaches derivatives	Fast enough for the clock	Provable to a third party	Cheap per request	Leaves the product intact
Delete in place	✗	✗	✓	✗	✓
Crypto-shred	✗	✓	✓	✓	✗
Never store	✓	✓	✓	✓	✗

No strategy sweeps the table, and the two that fail on completeness fail for opposite reasons: delete in place cannot reach a derivative because there is no record to address, while crypto-shred cannot reach one because the derivative was computed from plaintext and never went through the key. Only never store reaches a derivative, by making sure it was never subject-identifying.

The strategies also apply per surface rather than per system. Nothing forces one choice for all nine places, and matching each surface to the simplest strategy that works on it beats committing to one approach and then fighting the surfaces where it does not fit.

Routing each surface

The solution

Route each of the nine surfaces through the gates above, then wire the result as one runbook triggered by a subject identifier.

The source documents and the index they feed

Deleting the S3 objects is the easy half. The half that catches teams is that removing a source object does not remove its embedding. A Bedrock Knowledge Base holds a derived copy in the vector store, and that copy survives until the data source is synchronised again. Syncing is incremental, processing only what changed since the last run, and a document that has gone is removed from the vector store. Until that job runs, a retrieval query can still surface the subscriber’s content, and a generated answer can still quote it. So the runbook has to trigger the sync rather than assume it, and the compliance clock covers both the sync duration and the few minutes it can take afterwards for most vector stores to reflect the change. If the documents were pushed in through direct ingestion instead of an S3 data source, there is no object to delete: those documents come out through the knowledge base’s DeleteKnowledgeBaseDocuments call, addressed by document identifier, ten per request. The same sync machinery that keeps a corpus current is what makes it forgettable, and the index layout decides how many stores this has to reach.

Conversation state

The DynamoDB items are addressable, so a query on the subject partition key followed by a batch delete removes them with a receipt per item. The trap is reaching for time-to-live instead. TTL is a retention control: it deletes items on a schedule, typically within a few days of expiry rather than at the instant the timestamp passes, and it is not an on-demand mechanism. Use it to cap how long conversation state lives at all, which shrinks the surface every future request has to reach, and use an explicit delete to answer this request. Where conversation state lives determines how much of this there is to reach in the first place.

The two sets of logs

Model invocation logs are the surface where retention and erasure collide hardest, because they exist to be evidence. In CloudWatch Logs the retention setting on the log group is a schedule, and the API deletes streams and groups but never a single event, so anything subject-identifying that reaches a log group stays until retention expires it, plus the up-to-72-hour lag before expired events are actually removed. A data protection policy is not a substitute: it masks matches at egress, covers only events ingested after it is set, and anyone holding logs:Unmask reads the original. That should push identifiers out of the log payload rather than push the team into deleting log groups. Watch the overflow as well, because a request or response body over 100KB, and any binary content, lands as a separate S3 object under the logging data prefix even when the destination is CloudWatch Logs.

The S3 copy is worse and better at once. S3 Lifecycle rules expire objects on a schedule, which serves the retention obligation cleanly, and a lifecycle rule is not a way to answer an erasure request because the date it fires has nothing to do with the date the subscriber asked. Where compliance has put the bucket under Object Lock in compliance mode, the object cannot be deleted before its retention period ends, by anybody, including the account root. A versioned delete against it returns a 403. A delete that omits the version identifier returns 200 and writes a delete marker over the top, so a careless runbook records a success while the locked version sits underneath. That conflict has exactly one clean technical resolution, which is to encrypt the log copies under a key you can destroy: the object stays, immutable and auditable, and becomes ciphertext nobody can read. Choosing that at bucket-design time is one line of configuration. Discovering it on day nineteen of a thirty-day clock is not. Building the audit trail and designing its deletion path are the same piece of work.

The cache

Response caches are derivatives, and discarding one outright breaks nothing. Evict by subject where the cache key carries one, and flush the segment where it does not; a cold cache adds latency for an hour and nothing else. Do it after the index re-sync rather than before, or the next miss repopulates the cache from a corpus that still contains the record.

The evaluation set and the tuning dataset

Both are copies, both are addressable, and both have a second-order problem. Pulling four questions out of the golden evaluation set changes the baseline, so every historical score was measured against a set that no longer exists. Record the removal as a versioned change to the set rather than an edit in place, so a regression comparison across the boundary is at least explicable. The tuning dataset is the same operation with much larger consequences downstream.

The model that already learned it

A fine-tuned model that memorised training records cannot be un-trained. There is no delete against a weight, and the only reliable removal is a retrain from a corrected dataset. That makes the deletion path for the model a plan rather than an operation: remove the rows from the dataset now, record that the currently-serving model was tuned on a dataset that included them, and retrain on the next planned cycle rather than per request. The way to keep that cost bounded is upstream, by scoping and anonymising the dataset before tuning so that a future request touches the dataset and never the weights. Promotion of the retrained model then follows the ordinary path.

Proving it happened

Two independent checks, plus a record. The first is a scan: run an Amazon Macie job across the S3 buckets in scope, the source corpus and the log copies, configured with a custom data identifier for the subscriber’s account reference alongside the managed identifiers for names, addresses, and payment details. Macie finding zero occurrences afterwards is machine-generated evidence, produced by something other than the process being audited. It reads S3 objects only, so the DynamoDB table and the log group need checks of their own. The second is a behavioural check. Fire the retrieval queries that used to return the subscriber’s documents straight at the knowledge base and confirm they return nothing relevant. Run one end-to-end generation to confirm the assistant no longer answers a question about that account. A store can be clean while an index is stale, and the retrieval query is what catches that.

Then write the record. The deletion event itself, with the request date, the surfaces touched, the key identifiers destroyed, the sync job identifiers, the Macie job identifier and its result, and the retrain the model is queued for, all of it in a log you keep for exactly this purpose. That record is the artefact a regulator reads, and producing it as a by-product of the runbook takes far less work than reconstructing it a year later from CloudTrail.

Worked example

Day zero, the request lands and the runbook resolves the subject identifier to a manifest: eleven S3 objects, one knowledge base document set, 340 DynamoDB items, two log groups, forty-one locked log objects, a cache segment, four evaluation rows, and 118 tuning rows.

Day zero still, the addressable surfaces go. The S3 objects are deleted, the DynamoDB items are deleted in batches, the cache segment is flushed, the evaluation set is republished as a new version with the four rows removed, and the tuning dataset is republished likewise. Each returns a per-object result, and the runbook writes all of them into the record.

Day zero plus two hours, the knowledge base data source finishes an ingestion sync and the vector store no longer holds the embeddings for the removed documents. The cache is flushed a second time, deliberately, because a request between the first flush and the sync could have repopulated it from a stale index.

Day one, the locked log objects. They cannot be deleted for another four months, so the per-subject data key that wrapped them is scheduled for deletion with a seven-day pending window. The CloudTrail entry for ScheduleKeyDeletion goes into the record; day eight is when the ciphertext becomes permanently unreadable, and the record notes the date rather than claiming it happened on day one.

Day two, verification. A Macie job runs across both buckets with the custom data identifier, and returns zero findings for the subscriber’s account reference. Six retrieval queries from the original tickets return no matching passages, and one end-to-end question about the account returns the assistant’s ordinary “I do not have information about that account” response.

Day two, the residue. The custom model was tuned on a dataset that contained those 118 rows and it is still serving traffic. The record says so plainly, names the next retrain window nineteen days out, and cites the dataset version that no longer contains them. That is the entry legal reviews, and it beats a runbook that reported success on day zero and left the model out of the manifest.

What’s worth remembering

1. A record propagates as copies and as derivatives: copies can be deleted with a receipt, derivatives such as embeddings, caches, and model weights can only be rebuilt or destroyed wholesale, and an embedding of personal data is personal data.
2. Deleting a source object does not remove its vectors until the knowledge base data source re-syncs, and documents pushed in by direct ingestion come out through DeleteKnowledgeBaseDocuments instead.
3. Retention controls and erasure requests are different instruments: S3 Lifecycle rules and DynamoDB TTL both expire content on a schedule, days late in TTL’s case, and neither answers a request made today.
4. An S3 bucket under Object Lock cannot honour a deletion inside its retention period, so encrypt what goes in it under a destroyable key and settle that conflict when the bucket is designed.
5. Crypto-shredding reaches every copy under one key in a single auditable call, which makes the encryption boundary and the deletion boundary the same design decision.
6. A model that memorised training records cannot be un-trained, so the answer is a scoped, anonymised dataset before tuning plus a scheduled retrain, and the proof of deletion is an Amazon Macie scan and a targeted retrieval query recorded as evidence.