Lab: Build a Data-Quality Gate

This is one of the hands-on labs that run alongside these posts. The scaffolding is low now: you get the plumbing and write the decision. The full lab is in lab-07-quality-gate.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours.

The scenario

Before documents reach a knowledge base, something has to stop the bad ones. Raw support-ticket records land in s3://bucket/incoming/. Most are fine; some have an empty body, a nonsense priority, a missing email, or are not even valid JSON. Ingest those and retrieval degrades with no error to show for it. An answer grounded in a half-empty record is worse than no answer. A gate reads each record, decides, and routes it: good records to clean/, bad ones to quarantine/ with the reasons attached. Only clean/ goes on to be embedded.

What you’re given

An S3 bucket, a Lambda with least-privilege access to it, six seeded records (three good, three broken in different ways), and the handler’s reading, routing, and reporting. The gap is validate().

Your task

Write the rules that decide fitness. validate(record) returns a pair: True and an empty list when the record is fit, False and a reason string for every rule it breaks. Match the seeded samples with four checks: an id that is present and non-empty, a body of at least ten characters, a priority drawn from VALID_PRIORITIES, and an email with an @ in it. Collect the reasons as you go rather than bailing at the first failure, so a record that breaks two rules is quarantined with both.

A record that breaks a rule is quarantined with its reasons; a clean one passes. The malformed-JSON record is caught before validate even runs, because a parse failure is a different kind of bad.

Deploy and prove it

cd lab-07-quality-gate
./scripts/deploy.sh
./scripts/test.sh
./scripts/teardown.sh

You get {"clean": 3, "quarantined": 3}, three objects under each prefix, and T-5’s quarantine note naming its two failures. Nothing is deleted; the bad records are held with an explanation.

When you want the reference answer, deploy it with SRC=solution ./scripts/deploy.sh, or unfold it here:

def validate(record):
    reasons = []
    if not record.get("id"):
        reasons.append("missing id")
    body = record.get("body") or ""
    if len(body) < 10:
        reasons.append("body missing or too short")
    if record.get("priority") not in VALID_PRIORITIES:
        reasons.append("priority not one of high/medium/low")
    if "@" not in (record.get("email") or ""):
        reasons.append("email missing or malformed")
    return (len(reasons) == 0, reasons)

The ideas underneath

• A gate routes; it does not delete. Quarantine with reasons keeps the data and makes the failure auditable, which is what an ingestion pipeline for a regulated system needs.
• The rules are the declarative part. Here they are a few lines of Python. AWS Glue Data Quality expresses the same intent as DQDL rules in a ruleset attached to a Data Catalog table, and reports a data quality score: the percentage of rules that pass. The mechanism you built by hand is what it manages. The drift version of the idea checks live traffic against a baseline instead of a batch against rules. For a Bedrock workload you assemble that yourself from CloudWatch metrics, model invocation logging, and Bedrock evaluation jobs re-run against a fixed prompt set. SageMaker Model Monitor packaged it for SageMaker endpoints, and is now closed to new customers, with existing customers able to carry on as before.
• Malformed input is a distinct failure from a rule failure. Catch the parse error first; both belong in quarantine, but for different reasons, and conflating them hides real problems.
• The gate belongs before embedding. One check there stops a bad record. Re-embedding a corpus you later discover was poisoned means reprocessing all of it, so the quality check is an ingestion-time concern rather than a clean-up job.

What’s worth remembering

1. Validate data before it is embedded; one check at the gate stops a bad record, and the alternative is reprocessing the whole corpus later.
2. Route, do not drop: quarantine failures with their reasons so nothing is lost and the pipeline is auditable.
3. Separate a parse failure (malformed input) from a rule failure (valid but unfit); both quarantine, for different reasons.
4. The rules are declarative intent; AWS Glue Data Quality is the managed version, with DQDL rulesets on a Data Catalog table and a score for the share of rules that pass.
5. Drift detection applies the same constraint idea to live inference data, a later stage of the same lifecycle; for a Bedrock workload you assemble it from CloudWatch, invocation logging, and evaluation jobs you re-run yourself, since Model Monitor is closed to new customers.
6. A knowledge base is only as trustworthy as the gate in front of it; quality is an ingestion property, not an afterthought.