The situation
A retrieval assistant is fed by a nightly refresh. Raw data lands in S3 from three places: an export of resolved support tickets, a sync of product documentation, and a dump from an operational database. Some tickets are blank, some documents are duplicated across sources, some rows carry email addresses and account numbers that must never surface in an answer or a log. Once the data is clean it goes two places, into a Bedrock Knowledge Base for retrieval, and occasionally into a fine-tuning set.
The team keeps reaching for whatever tool is nearest, and each person is reaching for a different problem without saying which. One wants every file checked as it lands, because the outage they remember was a single truncated export that poisoned a whole night’s index before anyone noticed. One wants the assembled corpus profiled before it moves anywhere, because the failure they remember was slower and worse: the meaning of “resolved ticket” drifted over a quarter and the answers quietly got less true. One wants the email addresses and account numbers gone at the boundary, because that failure has a regulator attached to it and does not care how good the rest of the batch was.
They are all partly right, and they are not really arguing about tools. They are arguing about which of three jobs comes first, and until that is settled the tool comparison cannot start, because the tools that do these three jobs are not competitors and do not substitute for one another.
Keeping PII out of prompts and logs is the downstream concern; this is the upstream one, catching the data before it is ever embedded.
What actually matters
The first thing to separate is the three jobs hiding inside “data quality”. They are not one job, and they need pulling apart before anything can be chosen.
Moving and reshaping. Reading a few million rows and a pile of documents out of one place, changing their form, writing them somewhere else. This job is defined by volume and by the cost of a pass over the data.
Judging. Deciding whether what arrived is fit to use. This job needs two things the first one doesn’t: a written definition of “fit”, and a verdict something downstream can act on, so a bad batch stops rather than proceeds.
Restricting. Deciding who may see which parts. This is policy, and it stays true on a night when nothing is running, which is the clue that it isn’t a step in the pipeline at all.
They come apart cleanly. Moving happens whether or not anyone is judging. Judging needs something to have been moved first. Restricting applies to the data at rest, in flight, and to people who will never run the pipeline. So a tool built for one of them does the other two badly or not at all, and this pipeline needs all three, in that order, from more than one tool.
The useful consequence: most of the candidates below are not alternatives to each other. Ruling one out because another is cheaper is a category error, and half the apparent disagreement in the room dissolves once each person says which of the three they were solving for.
The second is where in the lifecycle the check runs. There is a difference between validating a dataset at rest, before it feeds a model, and monitoring live data as it flows through a deployed endpoint. Both get called “data quality”, and both even use the same open-source engine underneath, but they sit at opposite ends of the pipeline. A corpus assembled tonight for tomorrow’s retrieval is an at-rest problem. Drift in the requests hitting a production model is a live problem. Reaching for the live-monitoring tool to gate a batch corpus is the classic mismatch.
The third is declarative rules versus custom code. Most quality checks are expressible as rules: this column is never null, this value is unique, this string matches a pattern, this count stays within a range. A declarative engine lets you write those as rules and get a score and a pass or fail, with the results catalogued. But some checks are genuinely bespoke, a cross-field business invariant, a call to an external service, a format no rule language covers. That is code, and code needs an event-driven runtime rather than a rules engine.
The fourth is the shape of the data. Rule-based quality engines are built for tabular data with columns and types. A pile of PDFs and HTML documents is not that, and validating unstructured content (is this document in the right language, is it long enough to chunk, is it a near-duplicate of one already indexed) leans more on custom code and lighter profiling than on a columnar rules engine. The corpus for a knowledge base is usually a mix, and the mix determines the tool.
The fifth is that governance stands on its own. Deciding that the PII columns are visible to the ingestion role but masked from analysts is not a quality check and not a transform. It is access policy, enforced centrally, ideally by tag rather than by hand-maintained grants. That is a distinct tool with a distinct model, and it runs alongside the quality gate rather than inside it.
What we’ll filter on
- Which job, ingest and transform, check against rules, or govern access?
- Lifecycle stage, validating a dataset at rest, or monitoring live inference data?
- Rules or code, declarative constraints, or bespoke custom logic?
- Data shape, tabular columns, or unstructured documents?
- Automation, an unattended pipeline gate, or interactive exploration?
The landscape
AWS Glue ETL, with Glue Data Quality. Glue is the batch workhorse: Spark jobs that read from S3 or a database, transform at scale, and write back. Its quality layer, Glue Data Quality, lets you define rules in DQDL (Data Quality Definition Language), things like Completeness "ticket_body" > 0.95, Uniqueness "doc_id" = 1.0, ColumnValues "language" in ["en","fr"]. It can recommend a starting ruleset from a table, run inside an ETL job or against a Data Catalog table, produce a quality score, and emit results to CloudWatch and EventBridge so a failing batch can be quarantined automatically. Under the hood it is the open-source Deequ engine. This is the default automated gate for a tabular corpus at rest.
AWS Lambda. The escape hatch. When a check is event-driven (an S3 upload triggers validation of one file) or too bespoke for a rule language (a cross-field invariant, a language-detection call, a near-duplicate check against an existing index), a Lambda is the right size. It is code, it runs per event in milliseconds to seconds, and it handles the unstructured cases a columnar rules engine cannot. It is the wrong tool for validating a multi-million-row dataset in one pass; that is Glue’s job.
SageMaker Data Wrangler and Glue DataBrew. Both are interactive preparation surfaces. Data Wrangler lives in SageMaker Studio with several hundred built-in transforms and a data-quality-and-insights report; DataBrew is a no-code visual profiler with its own transform library and quality statistics. They shine while a human is exploring and shaping a dataset, and they can export a repeatable recipe or job. They are not the unattended gate in a nightly pipeline; they are how you design what that gate should check.
SageMaker Model Monitor (Data Quality). This is the tool most often misapplied here. Model Monitor computes a baseline (statistics and constraints, again via Deequ) from a training dataset, then watches the live data hitting a deployed endpoint and alerts when it drifts from that baseline. It is production monitoring of inference traffic, not a gate for a batch corpus. If the scenario is “the data feeding tonight’s knowledge base refresh”, Model Monitor is the wrong stage of the lifecycle. It is also no longer a tool to adopt: it moved to maintenance in June 2026 and closes to new customers from the end of July, so existing schedules keep running while a new build assembles the drift job from CloudWatch metrics and alarms, model invocation logging, and scheduled Bedrock evaluation jobs.
SageMaker Clarify. Also frequently confused with quality. Clarify measures bias (class imbalance, difference in positive proportions, and related metrics) and produces feature-importance explanations. A dataset can pass every completeness and uniqueness rule and still be badly skewed, and that is a bias-measurement job, not Glue Data Quality’s. Clarify moved to maintenance in June 2026 and closes to new customers from the end of July; existing deployments keep running, and its foundation-model evaluation code lives on as the open-source fmeval library, with Bedrock evaluation jobs as the managed path. Reach for those when the concern is fairness of the data or the model, not when the concern is malformed or missing records.
AWS Lake Formation. The governance layer. Lake Formation centralises permissions over Data Catalog resources down to the column, row, and cell, and its tag-based access control (LF-Tags) lets you label the PII columns once and grant against the label rather than maintaining per-table grants. It shares governed data across accounts. It does not transform data and does not check quality; it decides who sees what. In this pipeline it is what keeps the account-number column visible to the ingestion role and masked from everyone else.
Glue Data Catalog and crawlers. The substrate the rest sits on. Crawlers infer schema and partitions and register tables; the Catalog holds that metadata and is the thing Glue Data Quality scores and Lake Formation governs. It is not a quality or governance tool by itself, but nothing else works cleanly without it.
Evaluation
Side by side
| Tool | Job it does | Lifecycle stage | Rules or code | Data shape |
|---|---|---|---|---|
| Glue ETL + Data Quality | transform + check | dataset at rest | declarative (DQDL) | tabular |
| Lambda | check (bespoke) | event / at rest | code | any, incl. unstructured |
| Data Wrangler / DataBrew | prepare + profile | interactive design | visual / recipe | tabular |
| Model Monitor (Data Quality) | check for drift | live inference | baseline (Deequ) | tabular features |
| Clarify | bias + explainability | dataset or model | metrics | tabular features |
| Lake Formation | govern access | at rest | policy (LF-Tags) | catalogued tables |
| Glue Data Catalog | metadata substrate | all stages | n/a | catalogued tables |
The pipeline, stage by stage
The solution
Glue Data Quality is the gate. For the tabular parts of the corpus, the resolved-ticket export and the database dump, write a DQDL ruleset and run it as a step in the Glue job that lands the data. Completeness "ticket_body" > 0.95 catches the blank tickets; Uniqueness "doc_id" = 1.0 catches the cross-source duplicates; a ColumnValues rule pins the language and the allowed sources. The job publishes a quality score, and a rule failure raises an EventBridge event that routes the bad batch to a quarantine prefix instead of into the Knowledge Base. Because the ruleset lives with the Data Catalog table, the checks are visible and auditable rather than buried in code.
Lambda handles what rules cannot. The documentation sync is not tabular, and some checks do not fit DQDL. A Lambda triggered on each uploaded document can detect the language, reject anything too short to chunk usefully, and compare a hash or a cheap embedding against what is already indexed to drop near-duplicates. This is the code path, and keeping it as small event-driven functions rather than folding it into the Spark job keeps each check independently testable. The line to hold is scale: one file per invocation is Lambda’s shape; validating the whole ten-million-row dump in one pass is Glue’s.
Lake Formation governs, in parallel. Governance is not a stage the data flows through so much as a policy laid over the catalogued tables. Label the account-number and email columns with an LF-Tag once, grant the ingestion role access to the tag, and mask it from the analyst roles. Now the same clean dataset presents differently depending on who reads it, and the PII never depends on a hand-maintained grant that someone forgets to update. This is the part an audit will ask about, and it is enforced centrally rather than re-implemented in every job.
Drift and bias come later, or elsewhere. Neither belongs in tonight’s refresh. The drift question matters once the assistant is in production and you want to know when the questions users ask start diverging from what the corpus was built for; for a Bedrock workload that signal is assembled from CloudWatch metrics and alarms over model invocation logging, plus scheduled Bedrock evaluation jobs, now that Model Monitor is maintenance-only. The bias question matters when the concern shifts from “is this record malformed” to “is this dataset skewed”, for a fine-tuning set where balance across classes actually matters, and it is answered with a Bedrock evaluation job or the open-source fmeval library. Both are real questions; answering them in the ingestion gate is answering a question nobody asked yet.
Worked example
The three sources land in a raw S3 prefix at 01:00. A Glue crawler updates the Data Catalog with any new partitions. A Glue job reads the two tabular sources, applies its transforms, and runs a DQDL ruleset: completeness on the body fields, uniqueness on the identifiers, allowed-value checks on language and source, a row-count range so a truncated export cannot pass as complete. The job writes a quality score; a score below threshold fires an EventBridge rule that moves the batch to quarantine/ and pages nobody until morning.
In parallel, each document from the docs sync triggers a Lambda that checks language, minimum length, and near-duplication, dropping or flagging the failures. Lake Formation policies, keyed on LF-Tags applied to the PII columns, mean the ingestion role sees the account numbers it needs to redact while the analytics team querying the same catalogued tables sees them masked. Only the batches that clear both the DQDL gate and the Lambda checks reach the RAGA pattern where you retrieve relevant documents at query time and stuff them into the prompt so the model can ground its answer on them. ingestion job, which chunks, embeds, and upserts. Nothing in this flow measures live drift or dataset bias, because those questions belong to other stages.
What’s worth remembering
- Separate the three jobs first: transform, check, govern. Most wrong answers come from treating them as one, and naming the job halves the option list.
- Glue Data Quality is the automated gate for a tabular corpus at rest. DQDL rules, a quality score, EventBridge on failure, catalogued and auditable.
- Glue Data Quality checks a dataset at rest; drift monitoring watches live inference traffic. SageMaker Model Monitor packaged the drift side with the same Deequ engine, but it is maintenance-only now; a new build assembles drift from CloudWatch, invocation logging, and scheduled evaluation jobs. Match the job to the stage.
- Lambda is the escape hatch for bespoke and unstructured checks. Event-driven, per file, arbitrary code, and the right home for language, length, and near-duplicate checks on documents.
- Lake Formation governs access, in parallel with the pipeline, not inside it. Tag PII columns once with LF-Tags and grant against the tag rather than per-table.
The nightly refresh lands on Glue Data Quality for the tabular gate, Lambda for the document and bespoke checks, and Lake Formation for the PII boundary, with the Catalog underneath all three. Drift monitoring and bias measurement stay out of it, not because they are weak ideas, but because they answer questions this stage of the pipeline is not asking. Getting the pipeline right is mostly getting those boundaries right.