The situation
A community app has bolted a generative feature onto an existing user-generated-content platform, and the surface area for unsafe content has quietly exploded. Members upload profile photos and short clips. They record voice notes that the app plays back to other members. They write posts and comments in free text. On top of that, a new assistant powered by a model on Amazon Bedrock takes a member’s prompt and generates replies, captions, and summaries that get shown to everyone else.
Every one of those paths can carry something the platform does not want to publish: explicit or violent imagery, a slur buried in a comment, a phone number or credit-card detail pasted into a post, a voice note that is abusive, or a model completion that drifts into a topic the brand has said it will never discuss. The team’s first instinct was to reach for one moderation API and run everything through it. That does not exist. Images are not text, audio is not an image, and moderating what a member typed is a different problem from moderating what the model said back.
The real question is a routing question. For each kind of content, and at each stage of the pipeline, which AWS service is built to screen it, and how do those services combine into one pipeline that covers uploads, prompts, and outputs without gaps.
What actually matters
The first thing that decides everything is modality. A moderation service is trained on one kind of media, and it cannot see the others. Amazon Rekognition analyses pixels; it has no idea what a sentence means. Amazon Comprehend analyses text; it cannot look at an image. Audio is a third case, because none of the text or vision services can hear it, so it has to be turned into text first before a text tool can read it. Getting the modality match right is most of the decision, and forcing one service onto the wrong media is the classic mistake.
The second axis is the stage in the pipeline, which matters most once a generative model is in the loop. There are three distinct places content needs screening, and they are not interchangeable. User uploads arrive before the model and are screened at ingestion. The prompt going into the model is an input that can carry abuse or an attempt to steer the model somewhere unsafe. The completion coming out of the model is fresh content the model just produced, and it needs screening before it reaches another member even if the prompt was clean. A tool aimed at uploads does nothing for what the model generates, and vice versa.
Third is what “unsafe” even means here, because it is several different concerns, not one. Explicit imagery is one thing; personally identifiable information like a phone number or a card is a completely different detection problem; toxicity and harassment is a third; and staying off brand-forbidden topics is a fourth. Some services cover one of these, some cover several, and the ones that overlap do so at different stages, so knowing which concern you are solving narrows the field fast.
Fourth is confidence and the grey zone. None of these services returns a clean yes or no; they return labels with confidence scores, and the platform sets the threshold. That immediately creates a band of borderline cases that sit below the auto-block line but above the auto-approve line, and those are exactly the ones a human should look at. A moderation design that has no path for the uncertain middle either over-blocks safe content or ships unsafe content, so a human-review step for the borderline band is part of the architecture, not an afterthought.
Fifth is that these are building blocks, not a product, and the pipeline is where the value is. A single upload might need Rekognition on the image and Comprehend on the caption; a voice note needs Transcribe then Comprehend; a model turn needs Guardrails on both ends. The services are designed to be composed, and the moderation posture comes from wiring the right ones into each path rather than from any one call.
What we’ll filter on
- Modality, is the content an image, video, audio, or text?
- Pipeline stage, is this a user upload, a model input, or a model output?
- Concern, explicit and unsafe visuals, PII, toxicity, or a brand-forbidden topic?
- Confidence handling, does the path route the borderline band to human review?
- Composability, does the content need two services chained rather than one?
The moderation landscape
Amazon Rekognition content moderation. The vision service for still images and stored or streaming video. Its moderation feature returns a hierarchy of moderation labels, such as explicit nudity, violence, weapons, drugs, hate symbols, and self-harm imagery, each with a confidence score and a top-level and second-level category. For images you call DetectModerationLabels; for stored video you run an asynchronous job with StartContentModeration and collect the results with GetContentModeration, which timestamps where in the clip each label appears. This is the tool for screening uploaded photos, profile pictures, and clips. It reads pixels only, so a caption attached to the image is out of its scope.
Amazon Comprehend for text. The natural-language service, and it covers more than one moderation concern. Its PII detection finds and can redact entities like names, phone numbers, email addresses, and payment-card numbers in free text. Its toxicity detection classifies text across categories such as hate speech, harassment, threats, and profanity with confidence scores. When the platform’s notion of unwanted content is specific to it and not a general category, a custom classifier can be trained on labelled examples to flag it. Comprehend is the reader for posts, comments, and any text extracted from another modality. It cannot see an image, so a screenshot of abusive text sails past it; that would be a Rekognition text-in-image job feeding its output onward.
Amazon Transcribe as the audio bridge. Nothing in the vision or text services can hear a voice note, so audio is converted to text first, and Transcribe is that step. It also carries its own moderation options: a vocabulary filter that masks or removes a supplied list of words, and toxicity detection that scores segments of speech for harassment and abuse using both the words and acoustic cues. In practice you transcribe the audio, optionally act on Transcribe’s own toxicity signal, and pass the resulting transcript into Comprehend for the fuller PII and toxicity read. Audio is never moderated directly; it is transcribed, then the text is moderated.
Amazon Bedrock Guardrails. The moderation layer for the generative model itself, and the one that sits on the prompt and the completion rather than on stored uploads. A guardrail bundles several policies: content filters that block categories like hate, insults, sexual content, violence, and prompt-attack attempts at configurable strengths; denied topics defined in natural language so the model refuses to engage with subjects the brand has ruled out; word and profanity filters; and sensitive-information policies that block or mask PII in the prompt or the response. A guardrail is applied to the input going into the model and to the output coming back, either through the ApplyGuardrail API directly or by attaching it to a Bedrock model invocation. This is what screens what a member asked and what the model generated, and it is model-facing, so it does nothing for a photo sitting in a bucket.
Amazon Augmented AI (A2I) for the grey zone. The human-review layer. A2I lets you define a human-review workflow with a worker task template and route items to reviewers when a moderation call lands in the borderline confidence band, or on a sample of traffic for quality auditing. It integrates with Rekognition content moderation directly and can be wired in behind any of the others through your own logic. This is the home for everything that is neither a confident block nor a confident approve.
Side by side
| Service | Modality | Stage it screens | PII | Toxicity | Visual unsafe content | Denied topics |
|---|---|---|---|---|---|---|
| Rekognition content moderation | Image, video | User uploads | ✗ | ✗ | ✓ | ✗ |
| Comprehend | Text | Uploads and extracted text | ✓ | ✓ | ✗ | ✗ (custom classifier approximates) |
| Transcribe | Audio to text | User uploads (audio) | ✗ (masks via vocab filter) | ✓ (own signal) | ✗ | ✗ |
| Bedrock Guardrails | Text (model I/O) | Model input and output | ✓ | ✓ | ✗ | ✓ |
| A2I | Any (review) | Borderline band | n/a | n/a | n/a | n/a |
Reading the table against the app: uploaded photos and clips go to Rekognition; posts and comments go to Comprehend; voice notes go through Transcribe then Comprehend; the assistant’s prompts and completions go through Guardrails on both ends; and anything in the uncertain middle of any of those calls goes to A2I.
The moderation map
The picks in depth
Rekognition for the uploaded pixels. Point it at the image or the stored video and it returns moderation labels with a two-level taxonomy and a confidence score on each. The platform sets a MinConfidence on the request and a block threshold on the results, so you decide how aggressive to be; a dating app and a children’s app draw the line in different places. For video the job is asynchronous and the results are timestamped, which lets you flag the exact second an issue appears rather than rejecting the whole clip blind. Rekognition also has text-in-image detection, which matters when abuse arrives as a screenshot; you pull the text out and hand it to Comprehend, because the moderation labels themselves are about visual content, not the words printed on it.
Comprehend for every path that ends in text. Its two moderation-relevant features solve different concerns. PII detection is an entity problem, finding and optionally redacting names, numbers, and card details, and it is what stops a member publishing someone’s phone number. Toxicity detection is a classification problem, scoring text for harassment, hate, threats, and profanity. When the unwanted content is specific to this community and not a generic category, a custom classifier trained on the platform’s own labelled examples fills the gap. Comprehend is also the second half of the audio and screenshot paths, reading text that a different service extracted.
Transcribe as the only way audio gets moderated. There is no direct audio-moderation service in this stack, so the pattern is fixed: transcribe first, then read the transcript. Transcribe earns its place with two built-in helpers, a vocabulary filter that masks a supplied word list at transcription time and a toxicity-detection option that scores speech segments using acoustic cues as well as the words, which catches tone a plain transcript loses. Even with those, the fuller PII and toxicity read still comes from passing the transcript to Comprehend, so audio is a two-service chain by design.
Guardrails for both ends of the model. This is the pick people miss, because uploads and generation feel like the same “moderation” job but they are not. A guardrail is attached to the model invocation or called through ApplyGuardrail, and it screens the prompt going in and the completion coming out. Content filters catch hate, insults, sexual content, violence, and prompt-attack attempts at strengths you set. Denied topics let the brand describe, in plain language, subjects the assistant must refuse, which no upload-facing service does. The sensitive-information policy blocks or masks PII on either side. Screening the output matters even when the input was clean, because the completion is new content the model just produced, and it is the thing that actually gets shown to another member.
A2I for the band nobody can auto-decide. Every service here returns confidence, not certainty, so set two thresholds rather than one: above the upper line, auto-block; below the lower line, auto-approve; in between, route to a human-review workflow. A2I provides that workflow with a worker task template, and it plugs into Rekognition content moderation directly. Sampling a slice of the confident decisions through the same review loop is how you catch threshold drift before members do.
A worked example: one post with a photo, a caption, and a voice note
A member submits a single post that carries three things at once: a photo, a typed caption, and a voice note, and then asks the assistant to write a summary of it for the feed.
The photo goes to Rekognition. DetectModerationLabels comes back with “Violence” at 91% and the platform’s block threshold is 80%, so the image is held. Because it is over the block line, it does not need a human; if it had come back at, say, 74%, it would fall in the review band and go to A2I instead of being published on a guess.
The caption goes to Comprehend. Toxicity detection scores it low, but PII detection finds a phone number, so the pipeline redacts that entity before the caption is stored rather than rejecting the whole post. One modality, two different concerns, one service handling both.
The voice note cannot be read by anything yet, so Transcribe converts it to text, with its vocabulary filter masking a couple of slurs inline and its toxicity signal flagging one segment as harassment. The transcript then goes to Comprehend for the same PII and toxicity read as the caption got, because the audio path always ends in a text service.
Finally the member asks the assistant to summarise the post, and that model turn is wrapped in a Bedrock guardrail. The prompt is screened on the way in, and the generated summary is screened on the way out against the content filters and denied topics, so even a clean prompt cannot produce a completion that reopens the violent content or drifts onto a forbidden subject. Four services, one post, each piece routed to the tool built for its media and its stage, with A2I standing by for whatever lands in the middle.
What’s worth remembering
- Modality decides the service first: Rekognition for images and video, Comprehend for text, and audio has to be transcribed before any text tool can touch it.
- Rekognition content moderation returns a two-level label taxonomy with confidence scores, synchronous for images and asynchronous with timestamps for stored video.
- Comprehend covers three text concerns that are genuinely different problems: PII detection and redaction, toxicity classification, and custom classifiers for platform-specific content.
- Audio is never moderated directly; Transcribe converts it to text, offers its own vocabulary filter and toxicity signal, and the transcript then goes to Comprehend.
- Bedrock Guardrails is the model-facing layer and screens both the prompt and the completion, with content filters, denied topics, word filters, and PII policies.
- Screen model output as well as input, because the completion is fresh content the model produced and can be unsafe even when the prompt was clean.
- Uploads, model inputs, and model outputs are three separate stages; a tool built for one does nothing for the others.
- Every service returns confidence, not a verdict, so set an auto-block and an auto-approve threshold and route the band between them to human review.
- Amazon A2I provides the human-review workflow for the borderline band and plugs into Rekognition content moderation directly.
- Real moderation is a pipeline, not a call: a single post can need Rekognition, Comprehend, Transcribe, and Guardrails together, each on the piece it was built for.