Lab 07 — A data-quality gate

Scaffold: 2/5. The pipeline reads, routes, and reports. You write the rules that decide what is fit to feed a model.

The scenario

Before documents reach a knowledge base, something has to stop the bad ones. Raw support-ticket records land in s3://BUCKET/incoming/: most are fine, some have an empty body, a nonsense priority, a missing email, or are not even valid JSON. Ingesting those poisons retrieval. A gate reads each record, decides, and routes it: good records to clean/, bad ones to quarantine/ with the reasons attached. This is the hand-built version of AWS Glue Data Quality, so the idea of a declarative gate is concrete.

The requirement

Run the gate and every record ends up in the right place: the three good tickets in clean/, and the three broken ones in quarantine/, each with a note of what failed. Only clean/ would go on to ingestion.

What’s provided

template.yaml — an S3 bucket and a Lambda with least-privilege access to it.
samples/ — six records: three valid, one with a too-short body, one with a bad priority and missing email, and one that is malformed JSON.
src/handler.py — the S3 read, the routing to clean/ or quarantine/, and the count report. The gap is validate().
solution/handler.py — the reference answer.
scripts/ — deploy (seeds incoming/), test (runs the gate and lists the routing), teardown (empties the bucket, then deletes it).

Your task

Implement validate(record) in src/handler.py, returning (ok, reasons):

id present and non-empty,
body present and at least 10 characters,
priority one of high, medium, low,
email contains an @.

A record that breaks any rule goes to quarantine with the reasons; a clean one passes. (The malformed-JSON record is caught for you before validate runs.)

Run it

./scripts/deploy.sh
./scripts/test.sh
./scripts/teardown.sh

What success looks like

test.sh prints {"clean": 3, "quarantined": 3}, lists three objects under clean/ and three under quarantine/, and shows T-5’s reasons (bad priority, missing email). Before you fill validate(), the function raises NotImplementedError.

If it fails

Everything quarantines — check validate returns (True, []) for a good record; an empty reasons list means clean.
AccessDenied on S3 — redeploy; the role grants read and write on this bucket only.
Teardown fails with BucketNotEmpty — teardown.sh empties it first; if you deleted the stack by hand, empty the bucket in the console then retry.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

A quality gate is a routing decision, not a delete: bad records are quarantined with reasons, so nothing is lost and failures are auditable.
The rules are the declarative part. Here they are Python; AWS Glue Data Quality lets you write the same intent as DQDL rules and get a score. Same idea, managed. The drift cousin on live data gets assembled from CloudWatch metrics, invocation logging, and scheduled evaluation jobs now that SageMaker Model Monitor is closed to new customers.
Catching malformed input before the rules matters; a parser error is a different failure from a rule failure, and both belong in quarantine.
The gate sits before embedding, the cheapest place to stop bad data, because re-embedding a poisoned corpus later is the expensive fix.

Lab 08 — Answer a metric question with text-to-SQL. You move from documents to structured data: a question becomes a SQL query the model writes, run against a real table, with the guardrails that keep generated SQL safe.