The situation
A pharmaceutical company runs a research assistant for its medical writers. Ask it a question about a compound and it retrieves the relevant passages from an internal corpus (study reports, statistical analysis plans, published literature, and years of correspondence with regulators), then an Amazon Bedrock model answers from those passages. The corpus lives in Amazon S3 and is indexed by an Amazon Bedrock Knowledge Base. Separately, a model the company trained itself on Amazon SageMaker AI ranks incoming adverse-event reports so the safety team reviews the serious ones first.
A submission goes out. Four months later a regulator’s assessor writes back disputing one sentence in it: the assistant said a dosing interval had been supported by a phase II study, and the assessor cannot find that support in the study as filed. The sentence went into the submission because a medical writer read it, believed it, and kept it.
The company has a fortnight to respond, and four people want four different things. Nadia, the medical writer, wants the passage that sentence came from so she can read it herself. Wes, the data engineer, wants to know which ingestion run loaded that passage and what it did to it on the way in, because he suspects the corpus holds a superseded draft. Bronwyn, the compliance lead, wants the list of datasets the corpus is built from, who owns each one, and whether any of them are licensed material the company was never entitled to feed a model. And the assessor wants something else again: how the adverse-event ranking model was built, what it was evaluated against, and who approved it for use. Every one of them says “where did this come from”, and no single artefact answers all four.
What actually matters
The four questions are about four different objects. Nadia is asking about one answer. Wes is asking about one record inside the corpus. Bronwyn is asking about the corpus as a whole. The assessor is asking about a model. Sorting the question by its object is what picks the artefact, and it is faster than arguing about which team owns provenance.
They also differ in when the record has to have been written. A citation is produced at answer time, by the system, every time, and it exists because the application asked for it. The other three have to have been created before the question was asked. Nobody can reconstruct which ingestion run touched a document six months ago if nothing recorded ingestion runs. Nobody can produce an inventory of datasets by remembering. A model card written during the response to a regulator is a document about a model rather than a record of how it was built, and an assessor can usually tell. This is the same trap as any question about the past that only a recorder already running can answer.
Two of these artefacts get treated as substitutes for each other and they are not. A citation proves that a sentence came from a particular document; it proves nothing about whether that document is right, current, or one the company should have been holding. The disputed sentence may well carry a perfect citation to a real passage in a real report that was withdrawn a year ago, which is a citation working correctly and an answer that is still wrong. Going the other way, a model card documents how a model was built and evaluated, and says nothing at all about any single answer that model produced. The two sit at opposite ends of the same system and neither covers the other’s ground.
One boundary is worth drawing before any of the work starts, because it costs teams weeks. Provenance is not explainability. Provenance answers where the input came from: which document, which pipeline run, which dataset, which training set. Explainability answers why the output came out the way it did: which features drove the score, why this application was declined. They are separate obligations in separate parts of the material, they need different artefacts, and satisfying one does not satisfy the other. A perfectly cited answer from a model nobody can interpret is still a decision the affected person cannot be given a reason for.
What we’ll filter on
- What the record is about: one answer, one record inside the data, the whole collection of datasets, or the model itself.
- Who consumes it: the reader of an answer, an engineer debugging the data, a compliance or legal owner, or an external assessor.
- When it is produced: at answer time, at ingestion time, when a dataset is registered, or when a model is built and approved.
- Whether it has to have existed before the question was asked, or can be generated on demand.
- What it cannot tell you, which is where the substitution mistakes happen.
The landscape
The AI Practitioner material files all of this under one objective, source citation and documenting data origins, and names three examples: data lineage, data cataloging, and Amazon SageMaker Model Cards. Those three plus the citation itself are the four artefacts, and each one belongs to one of the four people.
Source citation
A source citation is a link, returned alongside an answer, back to the retrieved passage the answer was written from. The retrieval step already knows which chunks it fetched and which document each chunk came from. A citation is that knowledge carried through to the interface instead of discarded on the way.
An Amazon Bedrock Knowledge Base returns citations as part of the response when the application asks for a generated answer rather than raw chunks. Each citation names the source location (for an S3-backed knowledge base, the object the chunk came from) and the span of text that supported the sentence. What the application does with that is a design decision, and it is where most of the value is won or lost. A citation rendered as a document title tells Nadia which of four hundred PDFs to go and read. A citation rendered as a link that opens the document at the passage lets her check the claim in ten seconds. The mechanics of a system where no sentence is allowed out without a passage behind it are a build decision taken at the start.
Two limits are worth stating plainly. A citation does not make an answer true, and treating it as a truth signal is how a stale corpus produces confident, well-sourced, wrong answers; that failure and its remedies belong to the separate job of keeping an assistant from making things up. And a citation only reaches back as far as the document. It says the sentence came from report 0412; it cannot say which version of report 0412, when that version was loaded, or whether a newer one exists. That question is the next artefact along.
Data lineage
Data lineage is the record of where a piece of data came from and every transformation it passed through on the way to where it now sits. Read forwards it answers “what did this source affect”; read backwards it answers “what produced this row, and through which steps”. For Wes, it is what turns “the corpus contains a superseded draft” into “the corpus contains a superseded draft because the 3 February run picked up the archive folder, and here is the run”.
There is no single AWS service called lineage, which trips people up. Lineage is produced by the machinery that moves the data, and the practical form is a small number of habits. Run the ingestion as a defined pipeline rather than a script somebody executes, so each run has an identifier, a start time, and a record of its inputs and outputs. Keep the source object’s identity all the way through, so a chunk in the index still knows the S3 key and version it came from. Turn on Amazon S3 versioning so overwriting a document leaves the previous version in place and visible rather than gone. And keep the API record: AWS CloudTrail records who called what and when, which is how an unexplained deletion gets a name attached.
On the model side of the house the equivalent already exists as a feature. Amazon SageMaker Pipelines records which data and which code produced which model artefact on every run. SageMaker ML Lineage Tracking stores and queries that graph, so “which exact extract trained version 7” has an answer nobody had to write down by hand. That is a large part of what running a pipeline instead of a script actually buys.
Lineage tells you the journey of one piece of data. It does not tell you what datasets exist in the first place, or who is allowed to say yes to using one.
Data cataloguing
A catalogue is the inventory: which datasets exist, what is in each one, where it lives, who owns it, how sensitive it is, and on what terms it may be used. It is the artefact Bronwyn is asking for, and it is organisational as much as technical, because the field that settles her licensing worry is a named owner rather than a schema.
On AWS the technical carrier is the AWS Glue Data Catalog, a central metadata store holding table definitions, schemas, locations and partitions for data sitting in S3 and elsewhere. Crawlers can populate it by inspecting the data, which gets the mechanical half done quickly. The half that matters here is the half people have to supply: table and column comments saying what a field means, and tags recording the owning team, the sensitivity classification, the licence, and the retention rule. A catalogue with schemas and no ownership answers a query planner’s questions and none of Bronwyn’s.
AWS Lake Formation sits over the catalogue and turns it into a governance surface. Permissions are granted centrally at table, column and row level rather than by handing out bucket access, so the record of who may read which dataset lives next to the record of what the dataset is. That access-control side is worked through in the four properties a dataset needs before a model sees it; here it matters because a catalogue entry with an owner and a permission attached is one a person actually maintains, and an unowned entry rots within a year.
A catalogue tells you what the corpus is made of. It cannot tell you what any single answer was based on, and it says nothing about a model.
Amazon SageMaker Model Cards
Amazon SageMaker Model Cards are a versioned record, held with the model, documenting its intended use, the data it was trained on, how it was evaluated and against what, its known limitations, and its approval status. One document, written by the people who built the model, kept in step with it as it changes.
This is the artefact an external assessor is usually asking for, and it is the one that most often does not exist. The assessor’s question about the adverse-event ranking model is not about a row or a passage. It is a set of build questions. What was this made to do, what was it not made to do, what was it measured on, how did it do on the groups that matter, what did the builders know was weak about it, and who signed it off. A model card holds those in one place, so the answer is a document rather than four people reconstructing a year of decisions from memory.
Two things keep a card honest. Version it alongside the model, so a card describes one model version and a retrain produces a new card rather than an edit to the old one. And record the approval on it, so the card carries the decision to run this model in production and the name of whoever made it. A card written after the fact is a story about a model. A card written at build time and approved before deployment is a record of one.
What a model card cannot do is explain a particular output. It documents the model, not the answer.
Evaluation
Side by side
| Artefact | The question it answers | Who consumes it | When it is produced | What it cannot tell you |
|---|---|---|---|---|
| Source citation | Which passage did this sentence come from? | The reader of the answer | At answer time, on every response | Whether that passage is correct or current |
| Data lineage | Where did this record come from and what happened to it? | Data engineers, incident responders | At ingestion and processing time, on every run | What else is in the corpus, or who owns it |
| Data cataloguing | What datasets do we have, who owns them, on what terms? | Compliance, legal, data owners | When a dataset is registered, and maintained after | Anything about one record or one answer |
| Amazon SageMaker Model Cards | How was this model built, evaluated and approved? | Assessors, auditors, model risk reviewers | At build time, versioned per model version | Why the model produced one particular output |
Read the last column down and the four artefacts stop looking interchangeable. Each one is blind exactly where the next one sees. That is why the pharmaceutical company cannot answer the regulator with any single one of them. It is also why a team that has three of the four usually has the wrong three. Citations and lineage tend to exist because engineers needed them. The catalogue and the model card tend to be missing, because nobody inside the company needed them until somebody outside it asked.
The timing column is the one that decides what is possible in a fortnight. Only the citation can be produced now. The other three are records that either were kept or were not, and no amount of effort during the response creates a history that was never recorded.
The solution
Start with the citation, because it is the only artefact available today and it settles the immediate dispute. Pull the assistant’s stored response for that conversation, read the citation attached to the disputed sentence, and open the passage. Two outcomes are possible and both are useful. If the passage says what the sentence said, the dispute is between the company and the assessor about a document, which is a normal regulatory conversation. If the passage says something else, the assistant paraphrased badly and the response has to cover how that is being caught in future. If there is no stored response at all, that is the first finding: an assistant that produces citations for the reader but keeps no record of what it said is unauditable the moment anybody asks about last quarter.
Wes’s half is next, and it is the one with a deadline behind it. Take the S3 key from the citation and work backwards. Object versioning shows whether that document was replaced and when. The ingestion pipeline’s run records show which run indexed it and what the run’s inputs were. If those records exist, the answer is a date and a run identifier. If they do not, the honest response is that the corpus contains that document and the company cannot say how it arrived. Say so, then say what changes. Ingestion becomes a defined pipeline with run identifiers, source keys and versions are carried through to the index, and the corpus is reconciled on a schedule against the systems of record that own the documents.
Bronwyn’s catalogue is the slowest and the most valuable. Register every data source that feeds the corpus in the AWS Glue Data Catalog. A crawler populates the schema. People fill in the rest by hand: owning team, sensitivity, licence terms, retention, and the system of record it was copied from. Then put AWS Lake Formation over it and grant read access through the catalogue rather than through bucket policies, so the inventory and the permissions stay in one place and drift together instead of apart. The licensed-literature worry is answered by a licence field with an owner’s name against it, and by nothing else.
The assessor’s question needs a model card for the adverse-event ranking model, and there is no shortcut. Write it from what does exist (the training extract, the evaluation results, the pipeline runs that produced the current version) and be explicit about which parts were reconstructed. Record the intended use, what the model is not for, the evaluation set and results including per-group figures, the known limitations, and the approval. Then make the next one automatic: the model card is produced as part of the build, approved before deployment, and versioned with the model, so it is a record and not a reconstruction.
Two gotchas are worth naming. First, a citation nobody sees is a citation nobody uses. Returning citations from the API and printing only the answer text is a common build, and it carries the cost of provenance without the benefit. Second, provenance for a generative answer includes more than the documents. Which model version answered, which prompt version was in force, and which knowledge base version was queried all belong in the stored record, because a change to any of them changes the answer. Keeping the prompt under version control is the same requirement as managing prompts as versioned artefacts across many services, and it is the third leg of an answer you can reconstruct a year later.
Worked example
Tracing the disputed sentence
Nadia opens the stored conversation and finds the disputed sentence carries one citation: an S3 object, study-reports/CMP-114/phase2-csr.pdf, and a passage on page 96. She reads the passage. It supports the sentence exactly.
Wes takes the same key and lists its versions. There are two. The current version was written in February; the one before it dates from the previous September. He opens the September version and finds the same page 96 with different text, because the February write was a superseded draft that a folder reorganisation swept back into the ingestion path. The ingestion run record names the job, the date, and the prefix it scanned, which is enough to say what went wrong in one sentence.
So the assistant retrieved honestly from the corpus it was given, cited correctly, and answered from a document that should not have been there. The citation proved the sentence’s origin and could never have flagged the problem; the lineage record is what identified it. The remedy sits in ingestion (scan only the approved prefix, carry the document’s effective date into the index, and filter retrieval on it) rather than anywhere near the model, and the retrieval design behind that is the ordinary shape of answering questions from your own documents.
What the assessor gets
The assessor’s question about the ranking model gets a model card and its supporting records: intended use (ordering incoming adverse-event reports for human review), explicitly not intended use (deciding that a report needs no review), the training extract identified by dataset and version, the evaluation set and results, per-category performance including the rare serious categories where the numbers are weakest, the stated limitation that performance on those categories is measured on small samples, and the approval with a name and a date. Behind it, the pipeline lineage links the card’s version to the exact extract and code that produced it.
What the assessor does not get, and does not ask for, is why the model scored one specific report the way it did. That is a different obligation with a different answer, and offering a model card in response to it is how a fortnight gets wasted.
What’s worth remembering
- Source citation is produced at answer time and links one sentence back to the retrieved passage it came from, which makes a claim checkable without making it correct.
- Data lineage records where a piece of data came from and every transformation it passed through, so a bad record can be traced to the run that produced it.
- Data cataloguing is the inventory of what datasets exist, what is in them, who owns them and how sensitive they are, carried on AWS by the AWS Glue Data Catalog and governed with AWS Lake Formation.
- Amazon SageMaker Model Cards document a model’s intended use, training data, evaluation results, limitations and approval status in one versioned record, and are what an external assessor is usually asking for.
- Provenance answers where the input came from and explainability answers why the output happened, so a fully cited answer from an uninterpretable model still leaves the second obligation open.
- Only the citation can be generated on demand; lineage, catalogue and model card are records that were either kept before the question was asked or cannot be produced at all.