The situation
The assistant answers customer-support questions over a knowledge base. It’s a RAGA pattern where you retrieve relevant documents at query time and stuff them into the prompt so the model can ground its answer on them. app on Bedrock: a subscriber’s message goes in, relevant docs get retrieved, a PromptThe input you hand to an LLM – system instructions, user message, examples, retrieved documents, tool descriptions, the lot. gets assembled, and a completion comes back. It has been in production for four months and the support team likes it.
Now there’s a compliance review, and the reviewers arrive with a checklist. Who invoked this model, and when? What exactly did the model receive as input, and what did it return? Which model and which version produced each answer? Who approved putting this into production, and what was it approved to do? And can you prove no customer PII ended up somewhere it shouldn’t, and that the logs themselves haven’t been edited since?
The app has none of this. It logs application errors and a request count to CloudWatch, and that’s it. Nothing records the prompt-and-completion pairs, nothing records who changed the guardrail configuration last month, and there’s no document anywhere stating what the assistant is allowed to do. Getting this wrong means a failed audit and, depending on the sector, regulatory exposure. Getting it right means standing up an evidence trail that answers each question with an artifact instead of a shrug.
What actually matters
The trap is to think of this as one feature (“turn on logging”) when it’s really several concerns that live in different places and answer different questions.
The first is the record of what the model actually did: every invocation, with the full prompt and the full completion, stored durably. This is the transaction log of the AI itself. Without it, “show me what the assistant told this customer on Tuesday” has no answer.
The second is the record of who changed the system around the model. Someone enabled logging; someone configured a GuardrailA filter or rule applied to an LLM’s inputs or outputs to keep it inside safe, legal, or on-brand behaviour.; someone granted model access. Those are control-plane changes, and an auditor treats “who turned off the PII filter, and when” as seriously as any completion. Application logs won’t have it because these actions happen through the AWS API, not through the app.
The third is documentation of the solution itself. What is this assistant intended to do? What’s its risk rating? What are its known limitations, what data trained or grounds it, what did evaluation show? An auditor reviewing an AI system expects a governance document, not a code walk-through. “What data grounds it” has a tail the app can’t currently answer either: a retrieval app answers out of a knowledge base, so a reviewer will want the system each document came from, the version of it that was indexed, and the classification it carried. Where the answer’s evidence came from is evidence in its own right, and no amount of payload logging supplies it.
The fourth is the bridge from raw evidence to a compliance framework. Auditors don’t want a pile of logs; they want controls mapped to a standard (accuracy, privacy, fairness, resilience, responsible use, governance) with evidence attached to each and a report they can read. Collecting evidence is one job; organising it against a framework is another.
The fifth is the integrity of the evidence: retention long enough to satisfy the standard, encryption so the logs don’t become their own leak, tamper-evidence so nobody can claim the record was doctored, and tight access control so only the right people can read prompt-and-completion pairs that may themselves contain PII.
A sixth sits with the platform rather than with you: how Bedrock itself handles the prompts and completions. Model providers have no access to Bedrock logs or to customer prompts and completions. What AWS retains, and for how long, follows the data-retention mode set on the account or the project. You cite the setting rather than building a control around it, and the auditor will still ask.
What we’ll filter on
- Invocation payload capture, does it record the full prompt and completion of each call?
- Control-plane capture, does it record who changed configuration and access, through the AWS API?
- Framework mapping and report, does it map evidence to a compliance standard and produce an assessment?
- Provenance and documentation, does it record where the inputs came from, and produce a governance artifact describing intended use, risk, and evaluation?
- Tamper-evidence and retention, are the records immutable, encrypted, and kept long enough?
- Managed effort, how much of this is configuration versus code we own?
The landscape
Bedrock model invocation logging. Disabled by default, configured per Region, and the destination bucket or log group has to sit in the same account and Region. Once on, Bedrock delivers the request and response body of each invocation, along with the caller’s ARN, the model or inference-profile ID and the token counts, to an S3 bucket, CloudWatch Logs, or both. You select which modalities to log: text, image, EmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together., video. Bodies over 100 KB and binary data land as separate S3 objects with a reference in the log entry. This is the “what was asked and what was answered” record. It captures payloads and does not record configuration changes.
AWS CloudTrail. Records Bedrock control-plane calls as management events: PutModelInvocationLoggingConfiguration, guardrail create and update calls, model-access changes. InvokeModel and Converse are management events too, so the fact of each call (identity, timestamp, model ID) is recorded by default, without the payload. Data events are a separate opt-in selected per resource type, and AWS::Bedrock::Guardrail is the one that matters here, because it records each ApplyGuardrail evaluation and what the policy did. CloudTrail gives you tamper-evidence through log-file validation, and a multi-region trail delivered to an S3 bucket under Object Lock makes the record immutable. This is the “who did what, and when” layer.
AWS Audit Manager. Shipped a prebuilt generative-AI best-practices framework that mapped controls (accuracy, fairness, privacy, resilience, responsible use, governance) to evidence collected from CloudTrail and Config, and exported the result as an assessment report. It moved into maintenance mode on 30 April 2026 and can no longer be set up in an account or region that didn’t already have it, so an app being made audit-ready today can’t choose it. AWS points new customers at AWS Config conformance packs instead.
Amazon SageMaker Model Cards. Document the deployed model or solution: intended use, risk rating, training-data provenance, evaluation results, and caveats. This is the governance artifact that answers “what is this approved for, and what are its limits.” The card is an API object: CreateModelCard and UpdateModelCard write it, the risk rating is one of Unknown, Low, Medium or High, and the status moves from Draft to PendingReview to Approved, so the approval itself lands in the record. Any edit other than an approval-status change cuts a new version, which is what makes the history immutable. Using SageMaker AI to develop programmatic model cards is what stops the card drifting away from the model it describes, because the pipeline writes it from the evaluation job’s output instead of someone transcribing numbers into a wiki page.
AWS Glue Data Catalog and lineage tracking. Everything above describes the model; this describes what went into it. Using the AWS Glue Data Catalog to register data sources gives every ingestion job a named, versioned input rather than a path, with crawlers keeping schema, partitions, and last-updated current across the S3 prefixes the support documents live under and the ticket and product tables the pipeline joins against. Lineage is a second switch, and it doesn’t live in Glue. A Glue 5.0 or later Spark job with lineage events turned on emits OpenLineage events to a SageMaker Catalog (Amazon DataZone) domain, and the graph a reviewer walks, from knowledge-base index back through the job to the catalogued table, is held there. Spark DataFrames only; DynamicFrames emit nothing. This is the “where did this answer’s evidence come from” layer.
AWS Config. Does two jobs here. Managed and custom rules answer “is invocation logging still enabled, is the log bucket still encrypted, is the trail still running” continuously rather than on the day someone checked, which proves the posture holds rather than that it held once. It also records AWS::Bedrock::Guardrail and AWS::Bedrock::KnowledgeBase as resource types, so those configurations have a dated history. Gathered into a conformance pack, the same rules also map to a named standard: the “Security and Governance Best Practices for Amazon Bedrock” pack, deployed alongside “Security and Governance Best Practices for AI/ML Supporting Infrastructure”, is the current framework layer. The gap is the deliverable. Config has no assessment-report export, and evidence leaves it as configuration items through advanced queries, GetResourceConfigHistory, or Athena over the recorded history.
Bedrock Guardrails plus CloudWatch metrics. The enforcement half: guardrails apply the PII and topic policy, and CloudWatch surfaces the operational metrics and alarms. Worth naming here mainly to correct a tempting assumption. Masking PII at the guardrail does not clean the evidence, because the input field in the invocation log holds the original, unmodified request whatever the guardrail did, and the guardrail trace returned to your application carries the matched value in the clear as well. The enforcement side is its own subject, covered when you’re configuring guardrails for PII, topics, and grounding.
Evaluation
Side by side
| Option | Invocation payload | Control-plane | Framework mapping | Provenance & model doc | Tamper-evidence & retention | Managed effort |
|---|---|---|---|---|---|---|
| Invocation logging | ✓ request + response body | ✗ | ✗ | ✗ | via S3 (KMS, lifecycle) | Low, per-region toggle |
| CloudTrail | partial (call, not payload) | ✓ | ✗ | ✗ | ✓ log-file validation + Object Lock | Low-medium |
| Model Cards | ✗ | ✗ | ✗ | ✓ intended use, risk, eval | versioned in SageMaker AI | Low, generated in the pipeline |
| Glue catalogue + lineage | ✗ | ✗ | ✗ (feeds evidence) | ✓ source, version, lineage | partial (catalogue + job-run history) | Medium, crawlers + tagging |
| AWS Config | ✗ | ✗ (config state, not calls) | ✓ conformance pack, no report export | ✗ | ✓ config history | Medium, rules |
| Guardrails + CW | ✗ (masking never reaches the log) | ✗ | ✗ | ✗ | ✗ | Low-medium |
No row does the whole job. Invocation logging holds the payloads, CloudTrail the actions, the Glue catalogue where the inputs came from, the Model Card the solution, and Config both proves the controls stay on and carries the mapping to a named standard. Nothing on the list turns the collection into the report, which is the piece Audit Manager used to supply; that assembly is now yours, or a compliance-automation vendor’s.
The evidence stack, illustrated
The solution
The audit-ready shape is a stack, and each piece answers a distinct question. Invocation logging delivers the payload record. A multi-region CloudTrail with log-file validation, landing in an S3 bucket under Object Lock, gives the immutable who-and-when for every call and configuration change. The Glue catalogue and its lineage records answer where the retrieved content came from. A Model Card, written by the pipeline from the evaluation job’s output and moved to Approved by a named reviewer, documents the solution. Config carries the last two jobs at once: its rules prove the whole arrangement stays switched on, and the Bedrock and AI/ML conformance packs group those rules under named controls, so each piece of evidence has a control it belongs to. Assembling the controls and their evidence into the document the reviewer takes away is hand work now, because Audit Manager stopped accepting new accounts on 30 April 2026 and there is no managed replacement for its export. Keeping PII out of the payload record happens in the logging layer, since the guardrail cannot reach it: a CloudWatch Logs data protection policy audits and masks sensitive data in the log group itself.
The gotchas are where audits actually fail. Invocation logging is off by default and configured per region, so a Bedrock call in us-east-1 and another in eu-west-1 need logging enabled in each; a single toggle doesn’t cover the account. Cross-Region inference does not add a third place to enable it. CloudTrail and invocation logging both record in the source Region, the one you called, whichever destination Region actually ran the request, and the inferenceRegion field on the record names where it ran. So enable logging in every Region you call from, and read inferenceRegion when a reviewer asks where the processing happened.
The logs themselves are a liability. Prompt-and-completion pairs can contain exactly the PII the audit is worried about, so encrypt them with a customer-managed KMS key and lock the S3 bucket and CloudWatch log group down to a named set of principals. Mask in the log group with a data protection policy rather than at the guardrail. An audit trail that leaks is worse than none.
Where the logs land is a real trade-off. CloudWatch Logs gives you search and alarms and fast lookup, which suits “find every prompt from last Tuesday” and “alarm if a completion trips a filter.” S3 gives you cheap long-horizon retention with lifecycle policies, and Athena over the bucket answers analytical questions across months. Most audit-ready setups send to both: CloudWatch for the operational window, S3 for the retention horizon. Those invocation records are what a reviewer means by decision logs, and collecting them in CloudWatch Logs turns “show me what the assistant returned for this customer” into a query with a time range on it; CloudTrail sits alongside, answering who touched the system rather than what it returned. Residency needs stating in two halves, because the records and the processing can sit in different places. The logs stay in the Region you configured them in. Where the inference ran depends on the inference profile: a geographic profile keeps processing inside its geography, a global profile can route to any commercial Region, and for a model that requires retention, the retained inputs and outputs are stored in the destination Region.
The input side needs work of its own, because none of that record says where the retrieved content came from. Register the knowledge-base sources in the Glue Data Catalog and let crawlers keep schema, partitions, and last-updated current across the S3 prefixes and the ticket and product tables. Turn on lineage events on the Glue 5.0 Spark jobs so each index has a traceable ancestry in the catalogue domain: this job, that catalogued table, that version of it. Then carry the same tagging through the chain, so every S3 object and every .metadata.json beside a knowledge-base document holds source system, document version, classification, and ingestion date, and those fields come back on the retrieved reference the citation is built from. A reader of the answer can then see which document version produced it without anyone going near a log.
Worked example
The reviewer sits down and asks six concrete things. Each one lands on a different part of the stack.
“Show me every prompt this assistant received last Tuesday.” Model invocation logging. The full request-and-response records for that day are in CloudWatch Logs (queried by time range) and in S3 (queried with Athena for anything older than the CloudWatch retention window).
“Which version of which document produced this sentence?” The citation on the answer, resolved against the catalogue. The invocation log shows the retrieved text that went into the prompt and stops there; the citation carries the source system, document version, classification, and ingestion date from the metadata tags applied at ingestion, and the lineage record ties that version back to the catalogued table and the job that indexed it.
“Who turned off the PII filter, and when?” CloudTrail. The guardrail-update call is a management event carrying the caller identity, the timestamp and the request parameters, sitting in the validated multi-region trail. For the before-and-after, go to Config: it records AWS::Bedrock::Guardrail, so the configuration-item history shows the old state, the new one, and when it changed.
“What is this model approved for?” The SageMaker Model Card. Intended use, risk rating, known limitations, and the evaluation results that supported sign-off, in one versioned document with the approver on it.
“Prove these logs weren’t altered.” CloudTrail log-file validation confirms the trail’s own integrity, and S3 Object Lock on the invocation-log and trail buckets means the objects couldn’t have been overwritten or deleted inside the retention period, whatever anyone’s credentials allowed.
“Who can see our prompts, and how long are they kept?” A platform property and a setting rather than a log. Model providers have no access to Bedrock logs or to customer prompts and completions. How long AWS keeps them follows the data-retention mode configured on the account or the project, and reading that mode back is the evidence.
Then the reviewer wants the whole thing as one document, and there is no longer a service that presses that button. The conformance-pack dashboard gives the control-by-control state, Config advanced queries and GetResourceConfigHistory export the configuration items behind each rule, and the invocation logs, the trail, the lineage records, and the Model Card attach as the evidence under the controls they satisfy. Someone writes the covering narrative. The earlier work on configuring guardrails was the enforcement side; this is the evidence side, and the two meet in that document.
What’s worth remembering
- Audit-readiness is several concerns, not one toggle: payload record, control-plane record, source provenance, documentation, framework mapping, and evidence integrity each live in a different service, and since Audit Manager closed to new accounts, none of them writes the report.
- Bedrock model invocation logging is off by default and configured per Region, so enable it in every Region you call from; under cross-Region inference the record still lands in the source Region, and
inferenceRegionsays where the request ran. - CloudTrail records who changed the configuration; invocation logging records what the model returned. The auditors want both, and application logs give you neither.
- Log-file validation plus an Object-Lock S3 bucket is how you answer “prove the record wasn’t altered” instead of asking to be believed.
- The logs can contain the very PII under review, so encrypt with a customer-managed key and restrict access tightly; masking at the guardrail will not help, because the invocation log keeps the original request.
- Invocation logs prove what the model returned; a registered, tagged, lineage-tracked catalogue proves where the input came from, and “which document version produced this sentence” needs the second one.