The situation
A public-facing assistant runs an Amazon Bedrock model behind Amazon API Gateway, with Amazon Cognito issuing the tokens that identify each user. It is free to sign up for, which is a product decision and also the reason this scenario exists. Anyone with an email address can get an identity and start asking questions.
Three weeks after launch, the pattern in the logs is ugly in three separate ways. One identity trips the guardrail’s prompt-attack filter about forty times an hour, every hour, in what looks like a scripted sweep. A second burns roughly ten times the median tokens per session, pasting enormous documents in and asking for full rewrites. A third asks the same off-policy question in twenty different phrasings until one of them slips past the denied-topic filter, then goes quiet for a day and starts again.
None of this was found by a control. It was found by an engineer scrolling through model invocation logs on a Friday afternoon. The team has dashboards and traces for the healthy path and guardrails in the invocation path, so the raw material exists. What is missing is anything that watches the material without a human in front of it, and any agreement on what the system is allowed to do by itself when it finds something.
What actually matters
Detection and response are two decisions, and running them together is how a team ends up with a dashboard nobody looks at. A detection decision asks what evidence exists that a pattern is abuse. A response decision asks what the system may do about it without asking permission. They fail differently, they need different evidence, and the second one is where the argument actually is. Almost any team will agree to more logging. Far fewer will agree, in advance, that a rule may cut off a paying customer at three in the morning.
The signals themselves are not interchangeable, because they see different things. A volume signal counts requests, bytes, and tokens. It is cheap, it works in milliseconds, and it has no idea what was asked. A semantic signal knows the content was a prompt attack or an off-policy topic, because something read it, and it costs a model invocation to produce. A sequence signal is the hardest of the three. Twenty rephrasings of the same off-policy question only show up when you look at a session as a whole, and neither a per-request volume counter nor a per-request content filter can see a session. The third user in this scenario is invisible to the two cheapest controls, which is roughly why they picked that approach.
Attribution is the hinge that decides how useful any of this is. A signal you cannot tie to an identity can only produce a global response, and a global response punishes everyone to reach one person. Rate limiting by source address throttles a shared corporate NAT gateway along with the attacker sitting behind it. Counting guardrail interventions across the whole application tells you that abuse is happening and nothing about who is doing it. Every signal here is worth as much as its ability to name a Cognito subject. Getting that name into the signal is usually something the application has to do deliberately, not something a service does for free.
Then there is the asymmetry in what being wrong costs. A false positive that raises an alert costs an engineer five minutes. A false positive that throttles an API key costs a customer a slow afternoon. A false positive that disables a Cognito user costs that customer the product entirely, at the moment they were using it hardest. The people most likely to look statistically abnormal are power users doing exactly what you built the thing for. So the response gets graduated by how confident the signal is. Every automated action leaves an audit record naming the rule that fired and the evidence behind it, and a suspension needs a route back for someone who did nothing wrong. Continuous monitoring that can only shout or only shoot is not much use in either mode.
What we’ll filter on
- What the signal can see: request volume, prompt or response semantics, or a pattern across a session.
- Attribution: does the signal arrive already carrying an identity, or does the application have to stamp one on?
- Timing: does it act before the response returns, within a minute or two, or only when somebody runs a query afterwards?
- Baseline: does it need a fixed threshold somebody has to guess, or a learned band that needs history before it means anything?
- Response tier the signal can safely trigger on its own, given how often it will be wrong.
The landscape
Guardrail metrics
Amazon Bedrock Guardrails emit CloudWatch metrics broken out by policy type and by intervention category. You can see how many requests tripped the prompt-attack filter as distinct from the denied-topic filter, the word filter, or the contextual-grounding check. This is the only signal in the set that natively understands what was asked, because the guardrail read it. The catch is attribution: a guardrail metric tells you the application intervened forty times, not that one subject caused all forty. Bedrock lets a request carry request-level metadata, and stamping the Cognito subject onto every invocation is what turns an aggregate counter into something you can act on per user.
Model invocation logs
Bedrock model invocation logging is the record of what was actually asked and answered, delivered to CloudWatch Logs, to Amazon S3, or to both. Sent to CloudWatch Logs, it is queryable in Logs Insights for the recent window, which suits an on-call engineer chasing a pattern from the last few hours. Sent to S3, it becomes a partitioned dataset that Amazon Athena can query over months. That suits the sequence signal: finding one subject asking twenty near-identical questions needs a group-by across a session, not a per-request rule. Response logging matters as much as prompt logging here, because a jailbreak is only confirmed by what came back.
Anomaly detection on the metrics
CloudWatch anomaly detection fits a model to a metric’s own history and draws an expected band around it. That is how you catch token burst patterns without guessing a number. Tokens per session and invocations per minute are both good candidates, because both have a strong daily and weekly shape that a fixed threshold either sits above uselessly or below noisily. The band is only as good as the history behind it. A fortnight of baseline is the realistic minimum before an alarm on the band means anything, and a launch, a marketing push, or a pricing change resets what normal looks like.
AWS WAF in front of the API
AWS WAF sits on the API Gateway stage. Its rate-based rules count requests per source over a rolling window and block above a limit, and its managed rule groups cover the general web nastiness that has nothing to do with this application. It acts before the request reaches your code, which makes it the cheapest possible stop for volume abuse. It also cannot read prompt semantics in any useful way, so it counts and nothing more. Rate-based rules key on the source address by default, which is why they are blunt against anyone behind a shared egress.
API Gateway usage plans and per-key throttling
Usage plans attach a request-rate limit, a burst limit, and a daily quota to an API key. Where WAF counts by source address, a usage plan counts by key, so the throttle lands on an identity rather than a network location. This is the right shape for a signed-up user of a public assistant, and it is also the control that a gateway in front of Bedrock usually already has wired. It is still a volume control: it knows how many calls, never what was in them.
CloudTrail data events
AWS CloudTrail records the management plane by default and Bedrock invocations only when data events are switched on for the model resources. The record it produces is a caller-and-action one: which principal invoked which model, from where, at what time. For a public assistant, most invocations run under one application role. It will not spot a chatty end user, but it will notice a role nobody expected calling models, which is the access-governance question rather than the abuse one.
The response side
The second axis has four rungs, and they are ordered by how much they cost when they are wrong. Alert only, which notifies a human and changes nothing. Throttle, which drops the offending key’s usage plan to a low rate for a set period and is reversible on a timer. Suspend, which disables the Cognito user so their tokens stop working. And route to human review, which parks the evidence in a queue for a person to decide. That is the right destination for anything expensive and ambiguous, and Amazon Augmented AI exists for this kind of human-in-the-loop step.
Wiring is where these stop being policy statements. A CloudWatch alarm or a metric-filter match publishes to Amazon EventBridge, and EventBridge routes to an AWS Step Functions state machine. The state machine calls Lambda functions to do the work: read the recent invocation logs for that subject, decide the tier, apply the throttle or the disable, write the audit record, and notify both the user and the on-call channel. Automated alerting and remediation workflows built this way leave an execution history for every decision, which is what makes the remediation itself auditable rather than a Lambda that quietly disabled somebody’s account.
The pre- and post-processing filters run alongside all of it. Amazon Comprehend can screen an input before it reaches the model, and guardrails apply the model-based checks in the invocation path. On the way back, a Lambda function validates the completion, and API Gateway filters the response, catching anything that survived the previous three. AI output policy filters at that last layer are cheap insurance for the case where a new jailbreak beats the guardrail, and the exfiltration controls live in the same place.
Evaluation
Side by side
| Signal | Reads content | Names an identity | Acts before the response returns | Needs a baseline period | Highest tier it should trigger alone |
|---|---|---|---|---|---|
| WAF rate-based rules and managed rule groups | ✗ | ✗ (source address) | ✓ | ✗ | Throttle |
| API Gateway usage plans, per-key throttling | ✗ | ✓ (per key) | ✓ | ✗ | Throttle |
| Guardrail CloudWatch metrics by policy type | ✓ | ✗ unless stamped | ✓ | ✗ | Alert |
| CloudWatch anomaly detection bands | ✗ | ✗ | ✗ | ✓ | Alert |
| Invocation logs in Logs Insights and Athena | ✓ | ✓ | ✗ | ✗ | Human review |
| CloudTrail data events | ✗ | ✓ (principal) | ✗ | ✗ | Alert |
Read the last two columns together. The controls that act fastest are the ones that understand the least, and the control that understands the most is the one that only answers questions after the fact. No row does both jobs. Pick a single row and you either catch volume abuse and miss the semantic kind, or reconstruct the semantic kind hours later while the volume abuse ran unchecked. The tier column is the operational consequence: a signal that cannot name a subject has no business suspending one.
Signal to control to response tier
The solution
Layer the detection so each of the three patterns has a control that can actually see it, and graduate the response so the severity matches how confident the signal is.
Detection, layered
WAF rate-based rules go on the API Gateway stage as the outer layer, with the managed rule groups switched on for the generic web traffic. That handles crude flooding before it costs a token. API Gateway usage plans sit behind it with a per-key rate, burst, and daily quota, so an individual signed-up identity has a ceiling that a shared source address cannot smear across everyone.
Guardrail CloudWatch metrics carry the semantic layer, split by policy type and intervention category so a spike in prompt-attack interventions is distinguishable from a spike in the denied-topic filter. Make the application stamp the Cognito subject onto every Bedrock request as metadata. That single change turns a global counter into per-subject evidence; without it the semantic signal can only raise a general alarm. Turn on model invocation logging to both CloudWatch Logs and S3: Logs Insights for the last few hours, and Athena over the S3 copy for the session-shaped queries and the retention window compliance wants.
Anomaly detection bands go on tokens per invocation, tokens per session, and invocations per minute. Token burst patterns show up there without anybody guessing a threshold that will be wrong by next month. Give them a fortnight before you alarm on them, and expect to reset the baseline after any launch that changes traffic shape. CloudTrail data events on the Bedrock model resources cover the caller side, feeding the same alerting path.
That set gives you automated detection for misuse, drift, and policy violations across all three shapes: WAF and usage plans on volume, guardrail metrics on semantics, and scheduled Athena queries on sequence. The Athena query is the one people leave out, and it is the only thing in the design that catches the third user.
Response, graduated
Every detection publishes to EventBridge. A Step Functions state machine reads the event, gathers context from the invocation logs for that subject, and picks a tier.
Tier one is alert only, and it is where anything unattributed lands. An anomaly band firing on aggregate tokens gets a notification and a dashboard link, because there is no subject to act against.
Tier two is a throttle, applied by dropping that key’s usage plan to a low rate for a fixed window. It is reversible by expiry rather than by someone remembering, and the user gets told what happened and why.
Tier three is human review. The state machine writes the evidence, the matching prompts and responses, and the rule that fired into a review queue, and stops. Anything that would end an account goes through here, because the cost of being wrong is the whole relationship.
Tier four is suspension of the Cognito user. It fires automatically only for the narrow, high-confidence cases agreed in advance, such as a subject over a hard interventions-per-hour threshold with a matching prompt-attack signature. It always writes an audit record, always notifies the user, and always creates a review case so a human confirms it afterwards.
The gotchas
Guardrail metrics do not name a user unless you stamp a request attribute. This is the most common gap: the team builds a lovely per-policy dashboard, sees the interventions climb, and cannot answer which of forty thousand accounts is responsible.
WAF cannot read prompt semantics, so it counts volume and nothing else. A patient attacker sending one carefully crafted prompt every ten minutes is entirely invisible to it, and the adversarial testing exercise that found the jailbreak will tell you exactly how patient they need to be.
Anomaly bands need history. Alarming on a band with three days behind it produces noise, teaches the on-call to ignore the alarm, and leaves the detection technically present and practically dead.
Automatic suspension needs an appeal path. The legitimate power user who pastes a hundred-page contract in every morning looks like the token-burn attacker until a human reads the prompts. Without a way back, the automation converts an unusual customer into a churned one overnight.
And the response workflow itself has to be logged as carefully as the assistant is. A Lambda function that disables accounts and writes nothing down is a worse governance problem than the misuse it was built to stop. Tagging the whole detection and response stack alongside the rest of the workload keeps its own cost visible too, which matters once Athena is scanning months of logs on a schedule.
Worked example
Take the three offenders in order.
The scripted sweep tripping the prompt-attack filter forty times an hour is caught by guardrail metrics, per policy type, with the subject stamped on each request. A metric filter counting interventions per subject crosses its threshold in the first hour and EventBridge fires. Step Functions confirms the pattern against the last hour of invocation logs, then applies a tier-two throttle plus a tier-three review case. The user gets a message saying their requests are being rate limited; a human confirms within the day and moves it to suspension.
The token burner shows up on the anomaly band for tokens per session, which alarms at tier one because the band is an aggregate. That alert is enough for the state machine to run a targeted Athena query grouping the last day by subject, which names them immediately. The evidence goes to review rather than to an automatic action, and the reviewer finds a translation agency running exactly the workload the product is for. The outcome is a sales conversation and a higher usage-plan tier, not a suspension. That is the design working.
The patient rephraser trips nothing in real time, because each individual prompt is unremarkable and the volume is low. The nightly Athena query over the S3 invocation logs groups by subject and by embedding-neighbourhood of the prompt, finds twenty near-identical asks against one denied topic, and finds the one response that got through. That last part rests on response logging. The query proves a policy violation happened, not only that one was attempted. The case goes straight to human review, and the surviving completion goes to whoever owns the denied-topic wording, because the fix is a better guardrail, not a banned account.
What’s worth remembering
- Detection and response are two separate decisions, and continuous monitoring is only useful when the second one has been agreed in advance rather than improvised during an incident.
- Volume signals like WAF rate-based rules and API Gateway usage plans act fastest and understand least, guardrail metrics understand content, and only a session-level query over model invocation logs sees a pattern spread across requests.
- A signal that cannot name a Cognito subject can only trigger a global response, so stamping the subject onto every Bedrock request is what makes guardrail metrics actionable.
- CloudWatch anomaly detection catches token burst patterns without a fixed threshold, and needs about a fortnight of baseline before an alarm on the band means anything.
- Route automated alerting and remediation workflows through EventBridge to Step Functions to Lambda so every automated action has an execution history, an audit record, and a reversal path.
- Graduate the tiers by the cost of being wrong: alert for unattributed signals, a reversible throttle for attributed volume, human review for anything that would end an account.