The situation
A subscription business runs a support assistant on Amazon Bedrock. Subscribers ask it about deliveries, billing, and changes to their plan, and it answers from a Bedrock Knowledge Base built over support articles, resolved ticket threads, and free-text account notes. The model serving traffic is a custom one, fine-tuned on two years of ticket transcripts so it answers in the house voice.
A subscriber has asked for their data to be deleted. Legal has accepted the request, started a thirty-day clock, and asked engineering the one question that matters operationally: where is it, and when will it be gone?
The audit comes back with nine places. The source documents in Amazon S3, which include the ticket threads and the account notes. The Bedrock Knowledge Base index and the vector store behind it, holding the EmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. of those documents. The conversation state in Amazon DynamoDB, one item per turn. The model invocation logs in Amazon CloudWatch Logs. The second copy of those logs in an S3 bucket that compliance put under S3 Object Lock last year. A response cache in front of the model. The golden evaluation set, which was sampled from real traffic and includes four of this subscriber’s questions. And the fine-tuning dataset that produced the custom model now serving every request.
Nobody on the team disputes that the first item has to go. The argument is about the other eight.
What actually matters
Deletion is a property of a system, not an operation on a store. The reason nine places came back is that a record propagates two different ways, and the two need different answers. A copy is the same content sitting somewhere else: the S3 objects, the DynamoDB items, the log lines, the evaluation set, the tuning dataset. A derivative is something computed from the content that carries it forward in a different shape: the embeddings in the vector index, the cached responses, and the weights of the fine-tuned model. Copies can be found and removed. Derivatives usually cannot be edited at all, only rebuilt or destroyed wholesale, and the further a derivative is from the source the more expensive rebuilding gets. An embedding is a lossy but reversible-enough numeric representation of a passage of text, and if the passage was personal data then the vector is personal data too. A regulator will not accept an argument that a float array is a different thing from the sentence it encodes.
The second property worth weighing is whether the deletion is provable. Intent is not evidence. A store that supports a delete of one subject’s record gives you an API response, a CloudTrail entry, and a timestamp, and those three together are a receipt. A store that expires content on a schedule gives you a policy and a future date, which answers a retention obligation and does not answer an erasure request. Those are different regulatory instruments and they are constantly confused, because the same mechanism can serve either one. Decide up front which of your surfaces are covered by a receipt and which are covered by a schedule, because the ones covered by a schedule cannot go faster than the schedule.
Third, the clock and the conflicts. The thirty days starts at the request, and every surface has its own latency floor. Some are milliseconds, some are however long a full re-sync takes, and at least one is measured in whatever a retraining cycle costs. Worse, some surfaces exist precisely to prevent deletion. A bucket under an Object Lock retention period will refuse the delete, and that is the control working correctly rather than failing. The conflict between a retention obligation and an erasure obligation is real, it has a legal answer as well as a technical one, and it has to be resolved when the bucket is designed. Resolving it at request time means an engineer discovers on day nineteen that the mechanism physically cannot comply.
Fourth, what the deletion does to the system that remains. Removing one subscriber should not shrink the corpus that answers everybody else, invalidate the evaluation set that gates every release, or force a retrain per request. A deletion path that degrades the product will get quietly skipped the third time it runs, which is a worse outcome than a slow one. The design that survives contact with a busy team is the one where deleting a subject is boring.
What we’ll filter on
- Completeness: does the strategy reach every surface, including derivatives, or does it leave a residue somebody has to remember?
- Latency to comply: how long from the request landing to the data being unreadable, and is that inside the regulator’s window?
- Provability: does it produce evidence a third party can check, or only an assertion that it was done?
- Cost: what does it cost to set up before any request arrives, and what does each request cost once it does?
- Effect on the system that remains: does the product degrade, and does the deletion have to be repeated as new derivatives are built?
The landscape
Three strategies cover this ground. They are not mutually exclusive and the strongest designs use two of them, but they behave differently enough on the filters above that it helps to price them separately.
Delete in place
Find every copy and remove it, surface by surface, driven by a subject identifier. This is the obvious approach and the one every team reaches for first. It works well wherever the store has a single-record delete and an index that can find the record from a subject identifier: DeleteObject against S3, a DeleteItem per conversation turn in DynamoDB, an eviction from the cache, a row pulled out of the evaluation set. It degrades badly wherever the store cannot address a single subject, wherever the content is a derivative, and wherever a retention control forbids the delete.
The dependency this strategy carries is a subject index: a mapping from subscriber to every object key, item key, log stream, and dataset row that mentions them. Without it, deletion becomes a scan, and a scan over a document corpus for one person’s data is both slow and unreliable. Building that index is work you do at ingestion, not at request time.
Crypto-shred
Encrypt each subject’s data under a per-subject AWS KMS key, and delete the key when the subject asks. The ciphertext stays where it is, on every surface, in every backup, inside every replica, and stops being readable anywhere. Schedule key deletion once and every copy under that key goes dark at the same moment, including the ones you forgot about.
The appeal is completeness and speed. One ScheduleKeyDeletion call covers surfaces you never enumerated, and the pending window (seven to thirty days) is itself a safety net against a mistaken request. The evidence is strong too: key deletion is a CloudTrail event with an actor and a timestamp, and afterwards no principal on earth can decrypt the affected objects, which is a considerably stronger claim than a delete receipt. Holding the keys yourself is what makes this available at all.
The costs are real. A key per subject is a key you pay for monthly and a quota you can hit, so it usually means grouping subjects into cohorts or using a per-subject data key wrapped by a shared customer-managed key and stored in a table you can purge. Crypto-shredding is also all or nothing at the granularity of the key: anything else encrypted under it goes with it, so the encryption boundary and the deletion boundary have to be designed as the same boundary. And it does nothing at all for a derivative that was computed from plaintext and stored somewhere else, which is exactly what an embedding and a set of model weights are.
Never store
Push the work forward to ingestion so that nothing subject-identifying reaches the durable surfaces in the first place. If the corpus, the logs, and the tuning dataset hold no identifiers, there is nothing to find later and the deletion request is answered by pointing at the pipeline.
Two families of technique sit under this. The first is data masking techniques, which replace an identifier with a token or a placeholder while keeping the shape of the text, usually reversibly through a token vault so the application can still resolve a name when it legitimately needs one. The second is anonymization strategies for sensitive information, which remove the link to the individual irreversibly: generalising a delivery address to a suburb, dropping a customer reference entirely, replacing a name with a role. The distinction is whether a vault exists that can put the identity back. If one does, the vault is now the deletion surface, and purging the vault entry crypto-shreds by another route. If one does not, the data has left the scope of the request and there is nothing to delete.
Amazon Comprehend does the detection for both, and running it over prompts and logs before they persist is the same mechanism applied at a different point in the pipeline. Bedrock guardrails cover the runtime path, blocking or masking sensitive entities in a prompt or a completion before either is stored. Amazon Bedrock native data privacy features settle a related question that comes up in every one of these reviews: prompts and completions are not used to train the base foundation models and are not shared with model providers, so there is no copy of the subscriber’s question inside somebody else’s weights. What those features do not do is manage the copies you made yourself, which is all nine of the surfaces above.
The price of never store is utility. A support assistant that cannot see who it is talking to answers worse, and an anonymised tuning dataset teaches the model less. That trade is the design decision, and it is worth making deliberately rather than discovering it after the corpus is built.
Evaluation
Side by side
| Strategy | Reaches derivatives | Fast enough for the clock | Provable to a third party | Cheap per request | Leaves the product intact |
|---|---|---|---|---|---|
| Delete in place | ✗ | ✗ | ✓ | ✗ | ✓ |
| Crypto-shred | ✗ | ✓ | ✓ | ✓ | ✗ |
| Never store | ✓ | ✓ | ✓ | ✓ | ✗ |
Read the columns rather than the rows and the shape of the answer appears. No strategy sweeps the table, and the two that fail on completeness fail for opposite reasons: delete in place cannot reach a derivative because there is no record to address, while crypto-shred cannot reach one because the derivative was computed from plaintext and never went through the key. Only never store reaches a derivative, and it does so by making sure the derivative was never subject-identifying, which costs utility everywhere else.
The other thing the table hides is that the strategies apply per surface rather than per system. Nothing forces one choice for all nine places, and a design that picks the cheapest adequate strategy for each surface beats one that commits to a single approach and then fights the surfaces where it does not fit.
Routing each surface
The solution
Route each of the nine surfaces through the gates above, then wire the result as one runbook triggered by a subject identifier. The mechanics are where this gets interesting, because most of the surfaces have a gotcha that only shows up in production.
The source documents and the index they feed
Deleting the S3 objects is the easy half. The half that catches teams is that removing a source object does not remove its embedding. A Bedrock Knowledge Base holds a derived copy in the vector store, and that copy survives until the data source is synchronised again, at which point the ingestion job notices the object is gone and removes the corresponding vectors. Until that job runs, a retrieval query can still surface the subscriber’s content, and a generated answer can still quote it. So the deletion runbook has to trigger the sync rather than assume it, and the compliance clock includes the sync duration. If the documents were pushed in through direct ingestion instead of an S3 data source, there is no object to delete: those documents come out through the knowledge base’s delete-documents call, addressed by document identifier. Knowing which ingestion path a corpus uses is the difference between a deletion that works and one that silently misses the index. The same sync machinery that keeps a corpus current is what makes it forgettable, and the index layout decides how many stores this has to reach.
Conversation state
The DynamoDB items are addressable, so a query on the subject partition key followed by a batch delete removes them with a receipt per item. The trap is reaching for time-to-live instead. TTL is a retention control: it deletes items on a schedule, typically within a few days of expiry rather than at the instant the timestamp passes, and it is not an on-demand mechanism. Use it to cap how long conversation state lives at all, which shrinks the surface every future request has to reach, and use an explicit delete to answer this request. Where conversation state lives determines how much of this there is to reach in the first place.
The two sets of logs
Model invocation logs are the surface where retention and erasure collide hardest, because they exist to be evidence. In CloudWatch Logs the retention setting on the log group is a schedule, and individual log events cannot be deleted, so anything subject-identifying that reaches a log group is there until the group’s retention expires it. That fact should push identifiers out of the log payload rather than push the team into deleting log groups.
The S3 copy is worse and better at once. Amazon S3 Lifecycle configurations to implement data retention policies expire objects on a schedule, which serves the retention obligation cleanly, and a lifecycle rule is not a way to answer an erasure request because the date it fires has nothing to do with the date the subscriber asked. And where compliance has put the bucket under Object Lock in compliance mode, the object genuinely cannot be deleted before its retention period ends, by anybody, including the account root. That conflict has exactly one clean technical resolution, which is to encrypt the log copies under a key you can destroy: the object stays, immutable and auditable, and becomes ciphertext nobody can read. Choosing that at bucket-design time is cheap. Discovering it on day nineteen of a thirty-day clock is not. Building the audit trail and designing its deletion path are the same piece of work.
The cache
Response caches are derivatives and they are also cheap to discard. Evict by subject where the cache key carries one, and flush the segment where it does not; a cold cache costs latency for an hour and nothing else. What matters is that the eviction happens after the index re-sync rather than before it, or the next miss repopulates the cache from a corpus that still contains the record.
The evaluation set and the tuning dataset
Both are copies, both are addressable, and both have a second-order problem. Pulling four questions out of the golden evaluation set changes the baseline, so every historical score was measured against a set that no longer exists. Record the removal as a versioned change to the set rather than an edit in place, so a regression comparison across the boundary is at least explicable. The tuning dataset is the same operation with much larger consequences downstream.
The model that already learned it
A fine-tuned model that memorised training records cannot be un-trained. There is no delete against a weight, no scalpel that removes one subscriber’s phrasing from a checkpoint, and the honest answer is that the only reliable removal is a retrain from a corrected dataset. That makes the deletion path for the model a plan rather than an operation: remove the rows from the dataset now, record that the currently-serving model was tuned on a dataset that included them, and retrain on the next planned cycle rather than per request. The way to keep that cost bounded is upstream, by scoping and anonymising the dataset before tuning so that a future request touches the dataset and never the weights. Promotion of the retrained model then follows the ordinary path.
Proving it happened
Two independent checks, plus a record. The first is a scan: run an Amazon Macie job across the S3 buckets in scope, the source corpus and the log copies, configured with a custom data identifier for the subscriber’s account reference alongside the managed identifiers for names, addresses, and payment details. Macie finding zero occurrences after the runbook completes is machine-generated evidence, produced by something other than the process being audited. The second is a behavioural check: fire the retrieval queries that used to return the subscriber’s documents straight at the knowledge base and confirm they now return nothing relevant, and run one end-to-end generation to confirm the assistant no longer answers a question about that account. A store can be clean while an index is stale, and the retrieval query is what catches that.
Then write the record. The deletion event itself, with the request date, the surfaces touched, the key identifiers destroyed, the sync job identifiers, the Macie job identifier and its result, and the retrain the model is queued for, all of it in a log you keep for exactly this purpose. That record is the artefact a regulator reads, and producing it as a by-product of the runbook is much cheaper than reconstructing it a year later from CloudTrail.
Worked example
Day zero, the request lands and the runbook resolves the subject identifier to a manifest: eleven S3 objects, one knowledge base document set, 340 DynamoDB items, two log groups, forty-one locked log objects, a cache segment, four evaluation rows, and 118 tuning rows.
Day zero still, the addressable surfaces go. The S3 objects are deleted, the DynamoDB items are deleted in batches, the cache segment is flushed, the evaluation set is republished as a new version with the four rows removed, and the tuning dataset is republished likewise. Each returns a per-object result, and the runbook writes all of them into the record.
Day zero plus two hours, the knowledge base data source finishes an ingestion sync and the vector store no longer holds the embeddings for the removed documents. The cache is flushed a second time, deliberately, because the first flush ran before the sync completed and a request in between could have repopulated it from a stale index.
Day one, the locked log objects. They cannot be deleted for another four months, so the per-subject data key that wrapped them is scheduled for deletion with a seven-day pending window. The CloudTrail entry for ScheduleKeyDeletion goes into the record; day eight is when the ciphertext becomes permanently unreadable, and the record notes the date rather than claiming it happened on day one.
Day two, verification. A Macie job runs across both buckets with the custom data identifier, and returns zero findings for the subscriber’s account reference. Six retrieval queries drawn from the original tickets return no matching passages, and one end-to-end question about the account returns the assistant’s ordinary “I do not have information about that account” response.
Day two, the residue. The custom model was tuned on a dataset that contained those 118 rows and it is still serving traffic. The record says so plainly, names the next retrain window nineteen days out, and cites the dataset version that no longer contains them. That is the entry legal reviews, and it is a far better position than a runbook that quietly reported success on day zero and left the model out of the manifest.
What’s worth remembering
- A record propagates as copies and as derivatives: copies can be deleted with a receipt, derivatives such as embeddings, caches, and model weights can only be rebuilt or destroyed wholesale, and an embedding of personal data is personal data.
- Deleting a source object does not remove its vectors until the knowledge base data source re-syncs, and documents pushed in by direct ingestion come out through the delete-documents call instead.
- Retention controls and erasure requests are different instruments: Amazon S3 Lifecycle configurations to implement data retention policies and DynamoDB TTL both expire content on a schedule and neither answers a request made today.
- An S3 bucket under Object Lock cannot honour a deletion inside its retention period, so encrypt what goes in it under a destroyable key and settle that conflict when the bucket is designed.
- Crypto-shredding reaches every copy under one key in a single auditable call, which makes the encryption boundary and the deletion boundary the same design decision.
- A model that memorised training records cannot be un-trained, so the answer is a scoped, anonymised dataset before tuning plus a scheduled retrain, and the proof of deletion is an Amazon Macie scan and a targeted retrieval query recorded as evidence.