Lab: Build a Data-Quality Gate

This is one of the hands-on labs that run alongside these posts. The scaffolding is low now: you get the plumbing and write the decision. The full lab is in lab-07-quality-gate.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours.

The scenario

Before documents reach a knowledge base, something has to stop the bad ones. Raw support-ticket records land in s3://bucket/incoming/. Most are fine; some have a body too short to answer from, a nonsense priority, a missing email, or are not even valid JSON. Ingest those and retrieval degrades with no error to show for it. An answer grounded in a half-empty record is worse than no answer. A gate reads each record, decides, and routes it: good records to clean/, bad ones to quarantine/ with the reasons attached. Only clean/ goes on to be embedded.

What you’re given

An S3 bucket, a Lambda with least-privilege access to it, six seeded records (three good, three broken in different ways), and the handler’s reading, routing, and reporting. The gap is validate().

Your task

Write the rules that decide fitness. validate(record) returns a pair: True and an empty list when the record is fit, False and a reason string for every rule it breaks. Match the seeded samples with four checks: an id that is present and non-empty, a body of at least ten characters, a priority drawn from VALID_PRIORITIES, and an email with an @ in it. Collect the reasons as you go rather than bailing at the first failure, so a record that breaks two rules is quarantined with both.

A record that breaks a rule is quarantined with its reasons; a clean one passes. The malformed-JSON record is caught before validate even runs, because a parse failure is a different kind of bad.

Deploy and prove it

cd lab-07-quality-gate
./scripts/deploy.sh
./scripts/test.sh
./scripts/teardown.sh

You get {"clean": 3, "quarantined": 3}, three objects under each prefix, and T-5’s quarantine note naming its two failures. Nothing is deleted; the bad records are held with an explanation.

When you want the reference answer, deploy it with SRC=solution ./scripts/deploy.sh, or unfold it here:

def validate(record):
    reasons = []
    if not record.get("id"):
        reasons.append("missing id")
    body = record.get("body") or ""
    if len(body) < 10:
        reasons.append("body missing or too short")
    if record.get("priority") not in VALID_PRIORITIES:
        reasons.append("priority not one of high/medium/low")
    if "@" not in (record.get("email") or ""):
        reasons.append("email missing or malformed")
    return (len(reasons) == 0, reasons)

The ideas underneath

• A gate routes; it does not delete. Quarantine with reasons keeps the data and makes the failure auditable, which is what an ingestion pipeline for a regulated system needs.
• The rules are the declarative part. Here they are a few lines of Python. AWS Glue Data Quality expresses the same intent as DQDL rules in a ruleset attached to a Data Catalog table, and reports a data quality score: the percentage of rules that pass. The mechanism you built by hand is what it manages. The drift version of the idea checks live traffic against a baseline instead of a batch against rules. For a Bedrock workload you assemble that yourself from CloudWatch metrics, model invocation logging, and Bedrock evaluation jobs re-run against a fixed prompt set. SageMaker Model Monitor packaged it for SageMaker endpoints, and is now closed to new customers, with existing customers able to carry on as before.
• Malformed input is a distinct failure from a rule failure. Catch the parse error first; both belong in quarantine, but for different reasons, and conflating them hides real problems.
• The gate belongs before embedding. One check there stops a bad record. Re-embedding a corpus you later discover was poisoned means reprocessing all of it, so the quality check is an ingestion-time concern rather than a clean-up job.

What’s worth remembering

1. Validate before you embed. One check at the gate stops a bad record; finding a poisoned corpus later means reprocessing all of it.
2. Route, do not drop. Quarantine failures with their reasons, so nothing is lost and the pipeline stays auditable.
3. Parse failures differ from rule failures. Malformed input is caught before validate runs; valid but unfit records break a rule; both quarantine, for different reasons.
4. Glue Data Quality manages the gate. DQDL rulesets on a Data Catalog table; the score is the percentage of rules that pass.
5. Bedrock drift checks are self-assembled. Combine CloudWatch metrics, invocation logging and re-run evaluation jobs; SageMaker Model Monitor is closed to new customers.