The situation
A retrieval assistant is fed by a nightly refresh. Raw data lands in S3 from three places: an export of resolved support tickets, a sync of product documentation, and a dump from an operational database. Some tickets are blank, some documents are duplicated across sources, some rows carry email addresses and account numbers that must never surface in an answer or a log. Once the data is clean it goes two places, into a Bedrock Knowledge Base for retrieval, and occasionally into a fine-tuning set.
The team keeps reaching for whatever tool is nearest, and each person is reaching for a different problem without saying which. One wants every file checked as it lands, because the outage they remember was a single truncated export that poisoned a whole night’s index before anyone noticed. One wants the assembled corpus profiled before it moves anywhere, because the failure they remember was slower and worse: the meaning of “resolved ticket” drifted over a quarter and the answers got less true with nothing in the pipeline flagging it. One wants the email addresses and account numbers gone at the boundary, because that failure has a regulator attached to it, and a clean batch everywhere else is no defence.
They are all partly right, and they are not really arguing about tools. They are arguing about which of three jobs comes first, and until that is settled the tool comparison cannot start, because the tools that do these three jobs are not competitors and do not substitute for one another.
Keeping PII out of prompts and logs is the downstream concern; this is the upstream one, catching the data before it is ever embedded.
What actually matters
The first thing to separate is the three jobs hiding inside “data quality”. They are not one job, and they need pulling apart before anything can be chosen.
Moving and reshaping. Reading a few million rows and a pile of documents out of one place, changing their form, writing them somewhere else. This job is defined by volume and by the cost of a pass over the data.
Judging. Deciding whether what arrived is fit to use. This job needs two things the first one doesn’t: a written definition of “fit”, and a verdict something downstream can act on, so a bad batch stops rather than proceeds.
Restricting. Deciding who may see which parts. This is policy, and it stays true on a night when nothing is running, which is the clue that it isn’t a step in the pipeline at all.
They come apart cleanly. Moving happens whether or not anyone is judging. Judging needs something to have been moved first. Restricting applies to the data at rest, in flight, and to people who will never run the pipeline. So a tool built for one of them does the other two badly or not at all, and this pipeline needs all three, in that order, from more than one tool.
The useful consequence: most of the candidates below are not alternatives to each other. Ruling one out because another is cheaper is a category error, and half the apparent disagreement in the room dissolves once each person says which of the three they were solving for.
The second is where in the lifecycle the check runs. There is a difference between validating a dataset at rest, before it feeds a model, and monitoring live data as it flows through a deployed endpoint. Both get called “data quality”, and both even use the same open-source engine underneath, but they sit at opposite ends of the pipeline. A corpus assembled tonight for tomorrow’s retrieval is an at-rest problem. Drift in the requests hitting a production model is a live problem. Reaching for the live-monitoring tool to gate a batch corpus is the classic mismatch.
The third is declarative rules versus custom code. Most quality checks are expressible as rules: this column is never null, this value is unique, this string matches a pattern, this count stays within a range. A declarative engine lets you write those as rules and get a score and a pass or fail, with the results catalogued. But some checks are genuinely bespoke, a cross-field business invariant, a call to an external service, a format no rule language covers. That is code, and code needs an event-driven runtime rather than a rules engine.
The fourth is the shape of the data. Rule-based quality engines are built for tabular data with columns and types. A pile of PDFs and HTML documents is not that, and validating unstructured content (is this document in the right language, is it long enough to chunk, is it a near-duplicate of one already indexed) leans more on custom code and lighter profiling than on a columnar rules engine. The corpus for a knowledge base is usually a mix, and the mix determines the tool.
The fifth is that governance stands on its own. Deciding that the PII columns are visible to the ingestion role but masked from analysts is not a quality check and not a transform. It is access policy, enforced centrally, ideally by tag rather than by hand-maintained grants. That is a distinct tool with a distinct model, and it runs alongside the quality gate rather than inside it.
What we’ll filter on
- Which job, ingest and transform, check against rules, or govern access?
- Lifecycle stage, validating a dataset at rest, or monitoring live inference data?
- Rules or code, declarative constraints, or bespoke custom logic?
- Data shape, tabular columns, or unstructured documents?
- Automation, an unattended pipeline gate, or interactive exploration?
The landscape
AWS Glue ETL, with Glue Data Quality. Glue is the batch workhorse: Spark jobs that read from S3 or a database, transform at scale, and write back. Its quality layer, Glue Data Quality, lets you define rules in DQDL (Data Quality Definition Language), things like Completeness "ticket_body" > 0.95, Uniqueness "doc_id" = 1.0, ColumnValues "language" in ["en","fr"]. It runs from two entry points, inside an ETL job or against a Data Catalog table, and the Catalog entry point will also recommend a starting ruleset from the table. Either way it produces a quality score (the percentage of rules that pass), publishes Amazon CloudWatch metrics so the score can be trended and alarmed on, and emits results to EventBridge so a failing batch can be quarantined automatically. Under the hood it is the open-source Deequ engine. This is the default automated gate for a tabular corpus at rest.
AWS Lambda. The escape hatch. When a check is event-driven (an S3 upload triggers validation of one file) or too bespoke for a rule language (a cross-field invariant, a language-detection call, a near-duplicate check against an existing index), a Lambda is the right size. It is code, it runs per event in milliseconds to seconds, and it handles the unstructured cases a columnar rules engine cannot. It is the wrong tool for validating a multi-million-row dataset in one pass; that is Glue’s job.
Amazon SageMaker Processing. A managed job that runs a container of your choosing over data in S3, on instances you size, and shuts them down when the job finishes. It fills the gap between the other two: a transformation too heavy for a standard Lambda’s fifteen-minute ceiling and too bespoke for Glue’s Spark idiom. Resizing and re-encoding images before a multimodal embed, segmenting audio into chunks ahead of Transcribe, a tokenisation or deduplication pass over a whole corpus. SageMaker Processing is a processing job rather than a quality gate, so it complements Glue Data Quality instead of replacing it: Processing reshapes the data, and the ruleset is still what says whether what came out is fit to index.
SageMaker Data Wrangler and Glue DataBrew. Both are interactive preparation surfaces. Data Wrangler began as a Studio Classic feature and now runs inside SageMaker Canvas, with several hundred built-in transforms, a natural-language interface alongside the visual one, and a data insights and quality report; DataBrew is a no-code visual profiler with over 250 ready-made transformations and its own quality statistics. They are at their best while a human is exploring and shaping a dataset, and they can export a repeatable recipe or job. They are not the unattended gate in a nightly pipeline; they are how you design what that gate should check.
SageMaker Model Monitor (Data Quality). This is the tool most often misapplied here. Model Monitor computes a baseline (statistics and constraints, again via Deequ) from a training dataset, then compares the live data hitting a deployed endpoint against that baseline and reports violations when it drifts. It is production monitoring of inference traffic on tabular features, not a gate for a batch corpus, so for “the data feeding tonight’s knowledge base refresh” it is the wrong stage of the lifecycle. It is also no longer a tool to adopt: it moved to maintenance in June 2026 and closed to new customers on 30 July 2026. Existing schedules keep running, and AWS points new builds at the open-source SageMaker monitoring solutions with CloudWatch, or at Bedrock evaluation jobs for a generative workload.
SageMaker Clarify. Also frequently confused with quality. Clarify measures bias (class imbalance, difference in proportions of labels, and related pre- and post-training metrics) and produces feature-importance explanations built on SHAP. A dataset can pass every completeness and uniqueness rule and still be badly skewed, and that is a bias-measurement job, not Glue Data Quality’s. Clarify moved to maintenance in June 2026 and closed to new customers on 30 July 2026; existing deployments keep running, its bias metrics are published formulas a team can compute itself, and its foundation-model evaluation code lives on as the open-source fmeval library, with Bedrock evaluation jobs as the managed path. Reach for those when the concern is fairness of the data or the model, not when the concern is malformed or missing records.
AWS Lake Formation. The governance layer. Lake Formation centralises permissions over Data Catalog resources down to the column, row, and cell, and its tag-based access control (LF-Tags) lets you label the PII columns once and grant against the label rather than maintaining per-table grants. It shares governed data across accounts. It does not transform data and does not check quality; it controls who sees what. In this pipeline it is what keeps the account-number column visible to the ingestion role and masked from everyone else.
Glue Data Catalog and crawlers. The substrate the rest sits on. Crawlers infer schema and partitions and register tables; the Catalog holds that metadata and is the thing Glue Data Quality scores and Lake Formation governs. It is not a quality or governance tool by itself, but nothing else works cleanly without it.
Evaluation
Side by side
| Tool | Job it does | Lifecycle stage | Rules or code | Data shape |
|---|---|---|---|---|
| Glue ETL + Data Quality | transform + check | dataset at rest | declarative (DQDL) | tabular |
| Lambda | check (bespoke) | event / at rest | code | any, incl. unstructured |
| SageMaker Processing | transform (heavy, bespoke) | dataset at rest | code, in a container | any, incl. images and audio |
| Data Wrangler / DataBrew | prepare + profile | interactive design | visual / recipe | tabular |
| Lake Formation | govern access | at rest | policy (LF-Tags) | catalogued tables |
| Glue Data Catalog | metadata substrate | all stages | n/a | catalogued tables |
The pipeline, stage by stage
The solution
Glue Data Quality is the gate. For the tabular parts of the corpus, the resolved-ticket export and the database dump, write a DQDL ruleset and run it as a step in the Glue job that lands the data. Completeness "ticket_body" > 0.95 catches the blank tickets; Uniqueness "doc_id" = 1.0 catches the cross-source duplicates; a ColumnValues rule pins the language and the allowed sources. The job publishes a quality score, and a rule failure raises an EventBridge event that routes the bad batch to a quarantine prefix instead of into the Knowledge Base. Because the ruleset lives with the Data Catalog table, the checks are visible and auditable rather than buried in code.
Lambda handles what rules cannot. The documentation sync is not tabular, and some checks do not fit DQDL. A Lambda triggered on each uploaded document can detect the language, reject anything too short to chunk usefully, and compare a hash or a cheap embedding against what is already indexed to drop near-duplicates. This is the code path, and keeping it as small event-driven functions rather than folding it into the Spark job keeps each check independently testable. The line to hold is scale: one file per invocation is Lambda’s shape; validating the whole ten-million-row dump in one pass is Glue’s.
Lake Formation governs, in parallel. Governance is not a stage the data flows through so much as a policy laid over the catalogued tables. Label the account-number and email columns with an LF-Tag once, grant the ingestion role access to the tag, and mask it from the analyst roles. Now the same clean dataset presents differently depending on who reads it, and the PII never depends on a hand-maintained grant that someone forgets to update. This is the part an audit will ask about, and it is enforced centrally rather than re-implemented in every job.
Drift and bias come later, or elsewhere. Neither belongs in tonight’s refresh. The drift question matters once the assistant is in production and you want to know when the questions users ask start diverging from what the corpus was built for; for a Bedrock workload that signal is assembled from CloudWatch metrics and alarms over model invocation logging, plus scheduled Bedrock evaluation jobs, now that Model Monitor is closed to new customers. The bias question matters when the concern shifts from “is this record malformed” to “is this dataset skewed”, for a fine-tuning set where balance across classes actually matters, and it is answered with a Bedrock evaluation job or the open-source fmeval library. Both are real questions; answering them in the ingestion gate is answering a question nobody asked yet.
Lifting the quality of what gets through
A gate that only rejects is half a pipeline. Data validation workflows that cover the whole ingest also improve the records that pass, and doing that work once at ingest avoids repeating it on every retrieval afterwards. Three moves, in rising order of cost and risk.
The first is Amazon Comprehend to extract entities, detect the dominant language, and pull key phrases, with all of it written back as metadata on the chunk so the retriever can filter on it later. The second is Lambda functions to normalise data before it is embedded: dates into one format, units into one system, casing and whitespace made consistent, boilerplate headers and footers stripped so the same page does not embed as three near-identical chunks. The third is Amazon Bedrock to reformat text, for input that is genuinely unstructured, a scanned transcript turned into clean prose or a table of readings turned into consistent rows.
The costs differ sharply. An FM reformatting pass spends tokens on every document it processes, and it can rewrite a fact while tidying a sentence with nothing in the output to mark the change, so keep it behind a validation check and aim it at the documents that need it rather than running it over the whole corpus by default. Comprehend and a Lambda normaliser are deterministic and cheap: the same input produces the same output every night, and a diff against yesterday’s run says exactly what changed.
Of the three, entity extraction is the one to do first. A Bedrock Knowledge Base filters retrieval on document metadata attributes, and authorship, domain, and date are what those filters lean on hardest. Extract them once at ingest rather than recomputing them on every query for as long as the document stays in the index.
Worked example
The three sources land in a raw S3 prefix at 01:00. A Glue crawler updates the Data Catalog with any new partitions. A Glue job reads the two tabular sources, applies its transforms, and runs a DQDL ruleset: completeness on the body fields, uniqueness on the identifiers, allowed-value checks on language and source, a row-count range so a truncated export cannot pass as complete. The job writes a quality score; a score below threshold fires an EventBridge rule that moves the batch to quarantine/ and pages nobody until morning.
In parallel, each document from the docs sync triggers a Lambda that checks language, minimum length, and near-duplication, dropping or flagging the failures. Lake Formation policies, keyed on LF-Tags applied to the PII columns, mean the ingestion role sees the account numbers it needs to redact while the analytics team querying the same catalogued tables sees them masked. Only the batches that clear both the DQDL gate and the Lambda checks reach the RAGA pattern where you retrieve relevant documents at query time and stuff them into the prompt so the model can ground its answer on them. ingestion job, which chunks, embeds, and upserts. Nothing in this flow measures live drift or dataset bias, because those questions belong to other stages.
What’s worth remembering
- Separate the three jobs first: transform, check, govern. Most of the confusion comes from treating them as one, and naming the job halves the option list.
- Glue Data Quality is the automated gate for a tabular corpus at rest. DQDL rules, a quality score, EventBridge on failure, catalogued and auditable.
- Glue Data Quality checks a dataset at rest; drift monitoring compares live inference traffic against a training baseline. SageMaker Model Monitor packaged the drift side with the same Deequ engine, but it closed to new customers on 30 July 2026, so a fresh build assembles drift from CloudWatch, invocation logging, and scheduled evaluation jobs. Match the job to the stage.
- Lambda is the escape hatch for bespoke and unstructured checks. Event-driven, per file, arbitrary code, and the right home for language, length, and near-duplicate checks on documents.
- Lake Formation governs access, in parallel with the pipeline, not inside it. Tag PII columns once with LF-Tags and grant against the tag rather than per-table.
- Improve the input as well as rejecting it. Comprehend for entities and language, a Lambda normaliser for formats, Bedrock reformatting kept narrow because it costs tokens and can rewrite facts, and SageMaker Processing for the heavy image and audio work Lambda cannot hold.
The nightly refresh lands on Glue Data Quality for the tabular gate, Lambda for the document and bespoke checks, and Lake Formation for the PII boundary, with the Catalog underneath all three. Drift monitoring and bias measurement answer questions that belong to later stages, and they stay there.