The situation
A recruitment company holds 1.4 million application records going back eight years. They sit in two places: the applicant tracking system it has always run, and a stack of CSV exports inherited from a smaller agency it bought three years ago. Between them they carry names, addresses, dates of birth, right-to-work identifiers, uploaded CVs as free text, interview notes typed by consultants, and the outcome of each application.
The company wants to fine-tune a model to draft shortlisting summaries: given a role and a batch of applications, produce a paragraph on each candidate that a human recruiter then edits. Fine-tuning means the data becomes part of the model, so this dataset is not a lookup the application reads at run time. It is going into the weights.
The planning meeting produces four requirements from four people, and nobody agrees on which of them are the same problem. The data engineer has noticed that the outcome field is empty on about a third of the agency records and that “Senior Developer”, “Sr. Developer” and “Developer (Senior)” are three job titles as far as any query is concerned. The privacy officer wants to know why a training set needs anybody’s date of birth. The head of recruitment says the agency-side records were collected under a different privacy notice and the internal consultants must not be able to read the candidate names in them. The auditor asks a quieter question: in a year, when somebody asks which exact extract this model was trained on, what will you show them, and how will you know it has not been edited since?
Four requirements, four different things being asked for. The dataset has to be right, it has to be safe to hold, it has to be readable only by the people entitled to read it, and it has to stay what it was. Best practice for secure data engineering is exactly this list, and the first job is telling the four apart.
What actually matters
The four properties are independent. A dataset can be immaculate, with no missing values, no duplicates and consistent job titles, and still contain every candidate’s date of birth in the clear. It can be scrubbed of personal data and sitting in a bucket that half the company can read. It can be locked down tightly and quietly corrupted by a rerun of a broken job three weeks ago, with nobody the wiser. Passing one property tells you nothing about the other three, which is why a single “is the data ready?” tick box always misses something. Each one has its own question, its own AWS service and its own place in the pipeline.
They also apply at different moments, and getting the moment wrong is more expensive than getting the service wrong. Quality has to be established before ingestion, because a defect that survives into the training set becomes a defect in the model, and the way you find out is a fine-tuning run that costs real money and produces confident nonsense. Privacy treatment has to happen before the set is written down, because once personal data is in the weights there is no extraction step that gets it back out. Access control is enforced at read time, every read, forever. Integrity has to hold continuously from the moment the data lands, since the interesting failure is a change nobody noticed.
The privacy property carries a cost the other three do not, and it is worth naming before anyone volunteers to over-apply it. Treating personal data makes the dataset carry less signal. If you replace every candidate name with a stable identifier, the records still join and the model has lost nothing it should have been using anyway. Strip the free-text CVs of every place name, employer and university, and you have removed much of what the model was going to learn from. If you aggregate to counts by role and region, there is nothing left to fine-tune on at all. Heavier treatment is safer and weaker, and the recruitment company should choose where on that line it sits deliberately, with the privacy officer and the person who owns the model quality target in the same room. A dataset nobody may use is not a win.
One more thing separates a control that works from one that files a report. A profiling job that draws a chart of null rates tells you the outcome field is a third empty; a rule that fails the batch stops that third from reaching the training set. Both are useful, and only one of them is a gate. When a requirement is written as “we should monitor data quality”, ask what happens on the day the number goes bad, and if the answer is “somebody sees it on a dashboard”, the requirement has not been met.
What we’ll filter on
- Which of the four questions the requirement answers: is the data right, is it safe to hold, who may read it, or is it still what it was.
- Where in the pipeline the control applies: before ingestion, during preparation, at read time, or continuously in storage.
- Whether the control blocks bad data or only reports on it.
- Whether the control acts on the data itself or on the person reaching for it.
- What it costs the model: how much signal the treatment removes.
The landscape
Assessing data quality
Assessing data quality means checking five properties, and it helps to have the names because they turn a vague worry into a rule you can write. Completeness is whether the values that should be there are there, which is the empty outcome field on a third of the agency records. Accuracy is whether a value is true, which is the hardest to check automatically and usually needs a reference to compare against. Consistency is whether the same thing is represented the same way everywhere, which is the three spellings of Senior Developer, and also the two systems disagreeing about whether a date is day-first or month-first. Timeliness is whether the data is current enough for what it is being used for, which for eight-year-old salary expectations it emphatically is not. Uniqueness is whether one real thing appears once, which is the candidate who applied to nine roles and shows up as nine people.
On AWS, two services split this work. AWS Glue DataBrew is the exploratory half: point it at the data and it profiles the columns, giving you distributions, null counts, cardinality, outliers and inferred types, in a visual interface that does not require writing code. That is how you find out what is wrong with an inherited CSV pile you have never opened. AWS Glue Data Quality is the enforcement half: you define rules about the data, attach them to the pipeline, and evaluate them on every batch. The rules can be written by hand or recommended from a sample of the data, and a run that fails them can stop the job rather than pass the batch along. Profile once to learn what the rules should be, then run the rules forever.
Privacy-enhancing technologies
Privacy-enhancing technologies are the family of methods for reducing what personal data a dataset exposes, and they are not interchangeable. Six are worth being able to tell apart.
- Redaction removes a value outright. The name comes out and nothing goes back in its place.
- Masking obscures part of a value while leaving a usable remainder, which is how a card number becomes four visible digits and how a date of birth becomes a birth year.
- Pseudonymisation swaps a real identifier for a consistent fake one, so the same candidate is the same identifier everywhere and records still join across tables. The link back to the real person exists, held separately, which is why pseudonymised data is still personal data.
- Anonymisation breaks that link entirely. There is no mapping table, and no way back. It is a stronger promise and much harder to make honestly, because combinations of ordinary fields can re-identify someone even after the obvious identifiers are gone.
- Aggregation reports groups instead of individuals: 340 applications for warehouse roles in Perth last quarter, rather than 340 rows.
- Differential privacy adds carefully calibrated noise to results so that whether any one person is in the dataset cannot be determined from what comes out, at a measured cost to accuracy.
Two AWS services do most of the practical work here. Amazon Macie discovers sensitive data already sitting in Amazon S3 and reports what it found and where, which is how you learn that the agency exports contain identifiers nobody catalogued. It finds; it does not treat. Amazon Comprehend detects personally identifiable information inside free text and can redact it, which matters here because the CVs and the interview notes are prose, and no column-level rule will ever find a national insurance number that a consultant typed into a comment box. Macie reads objects and Comprehend reads text, and a dataset with both structured fields and free text needs both.
Data access control
Data access control decides who may read what, and it is enforced when somebody reaches for the data rather than when the data is prepared. Three layers stack.
IAM policies say which identity may perform which action on which resource, and they are attached to the role a person or a job assumes. S3 bucket policies work from the other end, attached to the bucket, saying which principals the bucket itself will serve; that is where you deny anything arriving without TLS, or restrict a bucket to one account. Both are all-or-nothing about an object: an identity that can read the file can read every row and column in it.
AWS Lake Formation is what you use when that is too coarse. It sits over data catalogued in the AWS Glue Data Catalog and grants permissions at the level of the table, the column and the row. The head of recruitment’s requirement is exactly a Lake Formation shape: internal consultants may query the applications table, and the candidate name and contact columns are not returned to them, and the rows sourced from the acquired agency are filtered out of their results entirely. One physical dataset, different views by role, enforced by the catalogue rather than by every query remembering to add a WHERE clause.
Data integrity
Data integrity is whether the data is still what it was: unchanged since it was written, or changed only in ways you can see and account for. It is the property the auditor is asking about, and it is the one teams most often assume they have.
Amazon S3 versioning keeps every version of an object rather than overwriting, so an accidental rerun that writes a bad extract over a good one leaves the good one recoverable and both visible. S3 Object Lock goes further and holds a version under a retention period, and the retention mode decides how hard that promise is. Governance mode blocks the ordinary delete, and a principal holding s3:BypassGovernanceRetention can still shorten the retention or remove the version, which is enough to stop an accidental overwrite and not enough to answer an auditor asking whether an administrator could have edited the extract. Compliance mode blocks it for everybody, including an administrator and the account root user, until the retention expires. A training extract signed off in March survives either way; only compliance mode makes that a claim the company can defend. Checksums are how you prove the bytes are unchanged: S3 computes and stores them, and a read that does not match its checksum fails rather than silently returning corrupt data. Encryption with AWS KMS protects the data at rest and has a second effect. Decryption only succeeds if the ciphertext is intact, so tampering shows up as a failure rather than as plausible-looking altered data. It also gives you a key policy of your own and a CloudTrail record of who used the key.
Evaluation
Side by side
| Property | The question it answers | What carries it on AWS | When it applies | What goes wrong without it |
|---|---|---|---|---|
| Data quality | Is this data right? | AWS Glue DataBrew to profile, AWS Glue Data Quality to enforce rules | Before ingestion, on every batch | The model learns the errors and repeats them fluently |
| Privacy-enhancing technologies | Is it safe for us to hold and train on? | Amazon Macie to find it, Amazon Comprehend to detect and redact PII in text | During preparation, before the set is written | Personal data goes into the weights and cannot be pulled back out |
| Data access control | Who may read which rows and columns? | IAM policies, S3 bucket policies, AWS Lake Formation | At read time, on every read | The dataset is fine and the wrong people are reading it |
| Data integrity | Is it still what it was? | S3 versioning, S3 Object Lock, checksums, AWS KMS | Continuously, from the moment it lands | Nobody can say which extract trained the model |
Read the last column downwards and the four failures are unrelated to each other. That is the argument for treating them as four properties with four owners rather than one readiness review, and it is also why a requirement that sounds like one of them is often another. “Nobody outside the team should see candidate names” sounds like a privacy problem and is a data access control problem, because the names are staying in the dataset and the restriction is on the reader. “Half the outcome fields are empty” sounds like a completeness problem to be fixed later and is a gate that should reject the batch now.
Which requirement is which property
The solution
The recruitment company builds one preparation pipeline with four gates in it, in the order the properties apply.
Quality comes first, on the raw landing zone. AWS Glue DataBrew profiles both sources, which is a morning’s work and produces the list of what is actually wrong: the null rate on the outcome field, the title cardinality, the duplicate candidates, the date-format disagreement between the two systems. That profile becomes an AWS Glue Data Quality ruleset with rules the team can defend. Outcome must be non-null on every row that reaches the training set. Job title must be one of the values in the canonical list. Candidate identifier must be unique. Application date must parse, and must fall in the last four years. That settles timeliness by dropping the 2018 records rather than arguing about them. The ruleset runs on every batch and fails the job when it does not pass, so the agency records with no outcome never make it into the set. Assessing data quality once, at the start, would have caught the same defects; running the rules on every batch is what stops the next import reintroducing them.
Privacy comes second, on the data that survived. Amazon Macie runs over the landing bucket and reports what sensitive data is in there and where, which produces a shorter and more alarming list than anybody expected, including identifiers in a folder of scanned attachments nobody knew was in scope. Then comes treatment, chosen field by field rather than applied uniformly. Names and contact details are pseudonymised. A candidate becomes a stable identifier, and their records still join across tables. Date of birth is masked down to a birth year, and then dropped entirely once the team admits nothing in a shortlisting summary should depend on it. Right-to-work identifiers are redacted. The CVs and interview notes go through Amazon Comprehend to detect and redact personally identifiable information in the prose. That is the only one of these steps that finds a phone number a consultant typed into a free-text box.
That last decision is where the trade-off gets made in public. The privacy officer’s opening position was to strip employers and universities from the CV text as well. The team pushed back with a number: those are the fields a shortlisting summary is mostly made of, and removing them makes the fine-tune close to pointless. What they agreed instead is that employers and institutions stay, personal identifiers go, the resulting dataset is still treated as personal data and stored accordingly, and the decision is written down with both names on it. That is the shape this argument should have. Not a default in either direction, but a decision recorded with the reasoning and the cost visible.
Access control comes third, and it is the layer that keeps working after the pipeline has finished running. The prepared dataset is catalogued in the AWS Glue Data Catalog and governed by AWS Lake Formation. The machine learning team’s role gets the whole table. The internal consultants’ role gets the table with the pseudonym column instead of the name column, plus a row filter that excludes the acquired agency’s records. The different privacy notice is then enforced by the catalogue rather than by everyone remembering. Underneath, the bucket policy denies non-TLS access and any principal outside the account, and no human role carries write access to the prepared data at all.
Integrity comes fourth and runs continuously. Versioning is on for the bucket. The signed-off training extract is written once, given an S3 Object Lock retention in compliance mode that outlasts the model, and encrypted under a customer-managed AWS KMS key so the key policy and the CloudTrail record of key use belong to the company. Compliance mode rather than governance mode, because the auditor is asking whether anybody could have altered the extract, and governance mode leaves a principal with the bypass permission who could. The auditor’s question now has an answer that takes thirty seconds: this object, this version, this checksum, locked on this date, and here is who has used the key since.
Two confusions are worth naming because they show up in every version of this conversation. Encryption is not data access control: an encrypted bucket that a hundred roles can read is readable by a hundred roles, since they all decrypt transparently on the way through. And a data quality dashboard is not a data quality gate. If the number going bad does not stop something from happening, the dataset is not being protected, it is being observed. A dataset can also be clean, private, governed and still unfit to train on for reasons none of these four properties touch, which is a separate review with a separate owner.
Worked example
Two requirements from the same meeting that sound like each other and are not.
“Nobody outside the team should see candidate names”
The tempting answer is a privacy-enhancing technology: redact the names, and nobody can see what is not there. Look at where the restriction actually falls. The names are needed by the recruiters, who are inside the team, so the values have to stay in the dataset. The restriction is on the reader, not on the data. That makes it data access control, and the mechanism is a Lake Formation column grant. If the requirement had been “the model must never learn a candidate’s name”, the values would not need to survive at all, and it would be a privacy treatment on the training extract instead. Same words, opposite mechanism, and the difference is whether anybody still needs the value.
“We found 40,000 duplicate applications”
This sounds like a storage problem and is uniqueness, one of the five properties in assessing data quality. It is more than tidiness. A candidate who applied nine times appears nine times in the training data, weighted nine times as heavily as one who applied once, so the model learns their pattern disproportionately. The rule that deduplicates on candidate identifier and application reference is a quality gate that also protects the fine-tune from a skew nobody would have spotted in the output. Deduplicate before the split, not after, and log how many rows the rule removed on every run, since a sudden jump in that number means an upstream system started behaving differently.
What’s worth remembering
- Secure data engineering is four independent properties, and a dataset passing one of them says nothing about the other three: assessing data quality, privacy-enhancing technologies, data access control, and data integrity.
- Assessing data quality means completeness, accuracy, consistency, timeliness and uniqueness, profiled with AWS Glue DataBrew and then enforced as AWS Glue Data Quality rules that fail the batch rather than draw a chart.
- Privacy-enhancing technologies are not interchangeable: redaction removes a value, masking obscures part of it, pseudonymisation swaps in a stable fake identifier so records still join, anonymisation breaks the link entirely, aggregation reports only groups, and differential privacy adds calibrated noise; Amazon Macie finds what needs treating and Amazon Comprehend redacts PII from free text.
- Data access control is about the reader, not the data, and when IAM and S3 bucket policies are too coarse, AWS Lake Formation grants access by table, column and row over the catalogued dataset.
- Data integrity is answered by S3 versioning, S3 Object Lock, checksums and KMS encryption together, which is how you name the exact extract a model was trained on and show it has not changed since; choose compliance mode when the retention has to hold against an administrator, since governance mode can be bypassed by a principal with
s3:BypassGovernanceRetention. - Heavier privacy treatment costs model quality, because a pseudonymised or aggregated dataset carries less signal, so decide how far to go deliberately and write down what was traded away.