Exam-style
Before a generative feature launches, the legal team wants a measured report on toxicity and bias across a 2,000-prompt test set, using a foundation model the company did not train. What produces that report?
Reveal the answer
A. The open-source fmeval library or a Bedrock model-evaluation job, scoring toxicity, prompt stereotyping, accuracy, and semantic robustness over the prompt set
The fmeval library, born as Clarify’s foundation-model evaluation and now open source, is built to score generated text on toxicity, prompt stereotyping, accuracy, and robustness, which is the shape bias takes for a generative model; a Bedrock model-evaluation job covers the same ground as the managed path. Guardrails enforce at runtime and produce blocks rather than measurements. The classical pre-training bias metrics need labelled training data the team does not have for someone else’s foundation model, and Comprehend sentiment measures tone rather than stereotyping or quality disparity.
Q. You must measure a GenAI feature for bias and toxicity before launch. Which tool?
A. The open-source fmeval library (SageMaker Clarify’s foundation-model evaluation, released as code) scores accuracy, toxicity, semantic robustness, and prompt stereotyping (bias), anywhere Python runs. A Bedrock model-evaluation job is the managed path, with toxicity and stereotyping metrics built in. The Clarify service itself moved to maintenance in June 2026 and closes to new customers from the end of July; existing deployments keep running.
Why? Bias for a generative model shows up as stereotyping and quality disparity, measured offline, not as label parity.