Keeping PII Out of LLM Prompts and Logs

The situation

The claims-processing assistant from earlier in the year now handles personally identifiable information at every step. Customer names, addresses, phone numbers, dates of birth, policy numbers, national-insurance numbers, medical diagnoses, and banking details all flow through prompts into Bedrock and back. The business requires them to flow, the assistant has to say “Hi Sarah, your claim on 15 April has been approved” to be useful. The compliance team requires them not to leak, no PII in CloudWatch logs, no PII in S3 buckets visible to the wrong principals, and no PII retained anywhere the business did not choose. Bedrock does not share prompts and completions with model providers, but retention inside AWS is a setting rather than a constant. The account and project data_retention_mode runs from none, where nothing is written to durable storage, through default, where data may be held for abuse detection, to aws_review, where inputs and outputs are kept inside the AWS boundary for up to 30 days and reviewed by AWS. Some models are only available at that last setting. The compliance team wants layered defences wherever the dial sits.

Concrete requirements:

• Ingress: PII in input must be classified, tagged, and tracked. Free-form customer messages (text, transcribed voicemail) can contain PII in unpredictable forms.
• Inference: the prompt carries what a useful answer needs and nothing more. An assistant answering “when is my next payment due?” doesn’t need the national-insurance number in the prompt even if it’s in the session context.
• Egress: model outputs must not carry fabricated PII, a policy number the model produced rather than retrieved, must not return PII that wasn’t in scope for this user, and must route any PII through the correct logging posture.
• Logs: CloudWatch logs and S3 session archives must have PII masked before they hit storage, “after-the-fact” redaction isn’t enough if the raw data sits in a log for 10 minutes first.
• Audit: for every PII touch, an audit record showing who (principal), what (PII class), when, and why (business reason tag).

What actually matters

PII handling in LLMLLMA neural network trained to predict the next token in a sequence, large enough that it generalises to tasks it wasn’t explicitly trained for. pipelines isn’t a single operation; it’s a lifecycle. Every stage has different threats and different tools.

The first decision is detection. Something has to recognise a national-insurance number, a postal code, a name, an email, a phone number in raw text. Pattern-matching works for format-constrained PII (emails, SSNs, credit-card numbers) but fails for names and addresses. ML-based classifiers cover the long tail, names, locations, organisations, free-form identifiers, across multiple languages.

The second is action on detection. Detection gives locations; action is what’s done with them. Options: redact (replace with a class marker), tokenise (replace with a reversible token that can be de-tokenised later), drop (remove the text and reject the request), or flag (annotate without changing the text). Different stages call for different actions.

The third is where in the pipeline redaction sits. Redact at ingress (before the message reaches the model at all)? At invocation time, in a guardrail layer wrapped around the call? At egress (before logging)? All of the above? The answer depends on who sees what at each stage.

The fourth is what additional safety layer the model inference itself can carry. Many inference platforms now offer an attached policy layer that filters both input and output, PII among them, along with content policies and topic restrictions. Layering one of those on top of application-level redaction catches what the application missed, and PII in a response that was never in the input.

The fifth is what reaches the model in the first place. Not all PII needs to flow to the model. If the question is “when is my next payment due?” and the session has access to a customer record, the customer’s full name does not need to be in the prompt to answer it. Trimming what goes in removes a leak path instead of patching one.

The sixth is logging and observability. Every PII touch is an audit event. Both API-call records and application logs need to be configured so PII doesn’t appear in plaintext. Encrypted storage destinations, write-time masking on log streams, and retention policies aligned to legal requirements are all part of the same posture.

One distinction runs through all of this: identifiable versus sensitive. A customer’s name is identifiable but low-sensitivity; a medical diagnosis is highly sensitive but may or may not be identifiable on its own. Good handling treats these differently, masking names for log hygiene is one thing; masking diagnoses is a different-class concern.

What we’ll filter on

1. PII coverage, which classes of PII are detected (names, addresses, emails, SSNs, medical, financial, etc.)?
2. Stage coverage, does this tool act at ingress, during invocation, at egress, on logs?
3. Language coverage, does detection work beyond English?
4. Reversibility, can redacted PII be re-hydrated for authorised callers?
5. Operational burden, what do we run and maintain?

The landscape

1. Amazon Comprehend (DetectPiiEntities and ContainsPiiEntities). A managed service that identifies PII in free-form text. PII detection takes English or Spanish, across 36 entity types: 22 universal ones including NAME, ADDRESS, EMAIL and CREDIT_DEBIT_NUMBER, and 14 country-specific ones including UK_NATIONAL_INSURANCE_NUMBER and SSN. DetectPiiEntities returns types, character offsets and confidence scores; ContainsPiiEntities returns only which types are present. A document is capped at 100 KB, and throttling on the synchronous operations is dynamic rather than a fixed published rate. Caller takes action (redact, tokenise, drop). Detect PII lists at USD$0.0001 per 100-character unit with a three-unit minimum, so USD$0.0003 is the floor for a short message. Integrates cleanly with Step Functions and Lambda pipelines.

2. Bedrock Guardrails (sensitive information filters). A guardrail configured with a PII policy acts at invocation time on both input and output. It offers 31 built-in types, grouped as general, finance, IT, and US, Canadian and UK national identifiers, each set to BLOCK, ANONYMIZE or NONE, with separate inputAction and outputAction so one class can be masked going in and blocked coming out. Custom regex patterns sit alongside the built-in types, without lookaround support. Two gaps matter here: the filter evaluates text content only, so PII the model writes into tool-call arguments, PII in tool results and PII in tool definitions is neither blocked nor masked; and the masking does not reach model invocation logging, where the input field always holds the original request.

3. Macie (for S3-stored documents). Discovers and reports sensitive data in S3 general purpose buckets, through continuous sampling or targeted discovery jobs, using managed and custom data identifiers. Not a real-time filter; a monitor. Useful for knowing what PII sits in content stored in S3 (chunks in Knowledge Bases, document archives, session transcripts), and it raises a separate policy finding when a bucket turns public or gets shared outside the account.

4. Custom regex-based redaction. A library in the application that applies patterns for known-format PII (emails, SSNs, credit cards). Fast, predictable, brittle. Misses names, addresses, and anything in unusual formats. Useful as a backstop for high-confidence formats; not sufficient on its own.

5. Prompt-level PII minimisation. Design the prompt so it doesn’t include PII the model doesn’t need. Instead of pasting the customer’s full record, pass only the fields the current question requires. Reduces PII surface at source: text that was never sent needs no redaction.

6. CloudWatch Logs data protection. Log-group data protection policies scan events on ingestion against managed and custom data identifiers, then mask the matches at every egress point, including the console, Logs Insights, metric filters and subscription filters. Configure it on one log group or account-wide; events ingested before the policy was set are not covered. Complements application-layer redaction by catching leaks that got past the application.

7. AWS Encryption SDK (client-side field encryption). A client library that encrypts a named field inside the application before it is written anywhere, using envelope encryption with a KMS data key. Detects nothing; it protects fields you have already identified. Ciphertext travels through prompts, logs, and archives without being readable there, and decryption is a KMS-authorised call.

Evaluation

Side by side

Tool	PII coverage	Stage	Language	Reversibility	Ops burden
Comprehend DetectPII	36 entity types	Ingress, egress	EN + ES	App-level tokenise	Low (API calls)
Bedrock Guardrails PII	31 types + custom regex	Invocation	Not stated in docs	Mask / block, per direction	Low (managed)
Macie	Managed + custom identifiers	Monitor (S3 only)	Many countries	No	Low (managed)
Custom regex	Format-constrained only	Anywhere	Regex-dependent	App-level	Moderate (maintenance)
Prompt minimisation	N/A	Ingress (design)	N/A	N/A	High (prompt design)
CloudWatch log protection	Managed + 10 custom	Logs	Country-scoped identifiers	Mask, logs:Unmask reads	Low (one-time config)
AWS Encryption SDK	Named fields only	Anywhere (pre-write)	N/A	Yes (KMS-gated)	Moderate (key + context design)

Every real system uses several of these together, so the work is in the composition.

The redaction lifecycle, layered

The solution

Ingress: Comprehend + tokenisation. Every user message and every retrieved context chunk flows through Comprehend’s DetectPiiEntities call. The call returns entity types (NAME, EMAIL, SSN, PHONE, ADDRESS, DATE_TIME, BANK_ACCOUNT_NUMBER, CREDIT_DEBIT_NUMBER, and 28 others) with offsets and confidence scores. For each detected entity above a confidence threshold (say 0.85), the application generates a reversible token ([NAME:tk_abc123]) and stores the mapping in a KMS-encrypted DynamoDB table scoped to the current session. The prompt that reaches Bedrock contains the tokens, not the PII.

That adds one synchronous Comprehend call per message, at a floor of USD$0.0003 a request. Set a TTL attribute on the mapping rows, but do not read it as a deletion time: DynamoDB removes expired items within a few days of expiry, not on the hour. Treat a row as live PII until it is gone, and filter expired rows out of reads rather than trusting the clock. Detokenisation for the session’s authorised user happens on the response path. Detection splits across two services here, Comprehend on text moving through the pipeline and Macie on documents that have already landed in S3.

Field-level encryption with the AWS Encryption SDK. Tokenisation covers the fields the assistant has to reason about. For the fields it never reasons about but still carries around, the bank account a claim pays into being the obvious one, encrypt the value in the application with the AWS Encryption SDK before it is written anywhere. The SDK does envelope encryption client-side: it asks KMS for a data key, encrypts the field in the process that already holds the plaintext, and emits a self-describing ciphertext with the encrypted data key attached. Server-side encryption with a KMS key on the bucket or the table protects the same value at rest, but the service receives plaintext on the way in and anyone with read access to the store gets plaintext back. Client-side, the plaintext never leaves the application, so a field that reaches a prompt template, a log line, or an S3 archive by accident is ciphertext when it lands. Bind the encryption context to the claim reference and the business reason so every decrypt call is attributable in CloudTrail. The encryption context is neither secret nor encrypted, and it appears in CloudTrail in plaintext, so it holds non-sensitive identifiers and never the values it is protecting.

Prompt minimisation. Before the ingress redaction even runs, the application trims the prompt to what’s necessary. If the question is “when is my next payment due?” and the customer record has 40 fields, the prompt carries only the two or three fields that answer the question, not the entire record. This is a prompt-design practice rather than a tool: PII that was never included needs no handling later. Structured context (JSON with named fields) makes this easy; free-text blobs make it hard.

Invocation: Bedrock Guardrails. A Guardrail attached to the model invocation runs its PII policy on both input and output. This is belt-and-braces: if the ingress tokenisation missed something (a novel PII format, a name variant Comprehend didn’t catch), Guardrails catches it at the model boundary. Set Guardrails to mask rather than block for input, we’d rather strip an unexpected PII token than fail the request, and to block for output, so PII the model fabricated never reaches the user.

Egress: detokenisation. The model’s response contains tokens ([NAME:tk_abc123]). The egress step looks up each token in the session’s mapping table and replaces it with the PII. This happens only for responses bound for the authorised user, responses stored in audit logs keep the tokens. The detokenisation call is gated by IAM; only principals with the session’s scope can de-hydrate the mapping.

Logs: CloudWatch data-protection policy + KMS-encrypted S3 archive. Application logs (request parameters, session state) go to CloudWatch Logs with a data-protection policy that masks common PII classes, a second layer behind application-level redaction. The full session archive (tokenised) goes to a KMS-encrypted S3 bucket; Macie monitors the bucket for any leaked PII that slipped through. S3 object ACLs and bucket policies prevent direct download by unauthorised principals; de-tokenisation for audit requires a privileged pipeline.

The data-protection policy carries two statements that do different jobs. The Audit statement names the data identifiers to look for and a FindingsDestination for the reports (a second log group, an S3 bucket, or an Amazon Data Firehose stream), so the privacy team sees counts of what is being detected and where without reading the log lines themselves. The Deidentify statement names exactly the same identifier list and carries an empty MaskConfig, which is what actually masks. Alongside the managed data identifiers, which cover national identifiers, bank and card numbers, health identifiers and credentials, register custom identifiers as regular expressions for the company’s own reference formats: claim references, policy numbers in the ABC123 shape, the internal customer ID. A policy takes up to ten of those, each pattern 200 characters or fewer. Attach it to the individual log group, or set it at account level, where it covers existing log groups as well as new ones; both policies apply where both exist.

Masking hides values at the points events are read rather than removing them. The raw event is still stored, and a principal holding logs:Unmask reads it in full through the console, unmask(@message) in Logs Insights, or the unmask parameter on GetLogEvents. Data protection is therefore a viewing control layered over the application’s redaction rather than a replacement for it, and logs:Unmask belongs on the same short list of privileged permissions as the detokenisation path. Note that CloudWatchLogsFullAccess already grants it.

Retention. The privacy review asks for a retention policy in writing, and two settings implement it. Use Amazon S3 Lifecycle configurations to implement data retention policies on the session archive: transition objects to S3 Glacier Instant Retrieval after 30 days, expire them at the limit legal signed off on, and give the audit prefix its own longer rule so the two clocks stay independent. Two constraints shape the numbers. Glacier Instant Retrieval has a 90-day minimum storage duration, so an expiry set inside that window still attracts the remainder as an early-deletion charge. And since September 2024 objects under 128 KB do not transition at all by default, which covers most single-session transcripts, so either add an ObjectSizeGreaterThan filter tuned to the real object sizes or aggregate transcripts before archiving them. On the log side, set a retention period on each log group, 90 days for application logs and years for the audit stream. CloudWatch Logs keeps events indefinitely by default, so a log group with no retention set is an unwritten decision to keep everything.

Audit stream. Every PII touch. Comprehend call, token creation, detokenisation, Guardrail hit, log mask, emits an audit event to a separate immutable stream (Amazon Data Firehose into an S3 bucket, lifecycled to Glacier, or a dedicated audit account). Compliance can query “every access to a name or SSN in the last 90 days by principal X” without needing the raw application logs.

Three data masking techniques get used as synonyms in privacy reviews and answer different requirements. Masking replaces the value for display while the original still exists behind it, which is what the CloudWatch policy and the Guardrail both do, and the record remains personal data. Tokenisation substitutes a reversible surrogate so an authorised path can recover the value, which is what lets the assistant greet Sarah by name; the mapping table becomes the sensitive asset, and the record is still personal data. Anonymisation strategies for sensitive information break the link for good: token-level redaction with no mapping retained, a date of birth generalised to a year, counts in place of individuals, so the original is irrecoverable and the record leaves the scope of the privacy obligation entirely. A support transcript an agent may reopen with the customer on the line needs tokenisation. An evaluation dataset needs masking, since the run scores whether a name appeared in the right slot and not which name it was. A fine-tuning corpus needs anonymisation, because a fine-tuned model can reproduce strings from its training data and there is no unmasking step to gate afterwards.

Worked example

Customer Sarah Patel, policy ABC123, asks via chat: “When is my next payment due?”

1. Ingress: session context loaded from customer record. Application trims to {plan_name, next_payment_due, next_payment_amount}. User message goes through Comprehend: name “Sarah Patel” (already in session, not in message), policy “ABC123” (also in session, not in message). Message itself contains no PII this turn. Prompt assembled with tokenised context: {customer_name: [NAME:tk_1], plan: "Basic", next_payment_due: "2026-08-21"}.

2. Invocation: Bedrock Guardrails scans input. No PII in plaintext (only tokens). Passes. Model generates response: "Hi [NAME:tk_1], your next payment is due on 21 August 2026." Output scanned by Guardrails; tokens pass; no fabricated PII in the text. Returned.

3. Egress: detokenisation on response. [NAME:tk_1] → Sarah. Response delivered: “Hi Sarah, your next payment is due on 21 August 2026.”

4. Logs: CloudWatch receives the invocation log. Application already tokenised the PII, so the log line contains tokens; CloudWatch data protection runs as a backstop and masks anything matching an SSN, email or credit-card identifier that slipped through. Log line persists for 90 days.

5. Audit: events for this turn land in the audit stream: comprehend.detect_pii_entities (session=s123, entities_found=2), token.create (types=[NAME, POLICY]), bedrock.invoke_model (guardrail_hits=0), token.resolve (types=[NAME], principal=sarah@example.com, reason=user_response).

The customer saw her first name; the model saw a token; the logs saw a token; compliance can trace every PII-touching action back to this session.

What’s worth remembering

1. PII handling is a lifecycle, not a feature. Ingress, invocation, egress, logs, audit, five stages, each with its own tooling.
2. Tokenisation preserves referential integrity across a turn. The prompt and the log line carry placeholders, the delivered response carries the real values, and the mapping is reversible for authorised callers only.
3. Bedrock Guardrails is the invocation-time safety net. Mask or block PII at model input and output; belt-and-braces behind application-level redaction.
4. Prompt minimisation removes the leak path instead of patching it. Don’t send what the model doesn’t need. Design prompts structured, not dumps.
5. CloudWatch Logs data protection catches the residuals, but it masks rather than deletes. The raw event is retained, and logs:Unmask reads it.
6. Guardrails masking stops at the text. Tool-call arguments, tool results and model invocation logs still carry the original values.