The situation
A produce-box delivery business has two pieces of work that both currently live on somebody’s laptop. The first is a churn model: a notebook that reads six years of subscription history, does a lot of cleaning in the first forty cells, and predicts which subscribers are likely to cancel in the next month. It scores well. It has never run anywhere except that notebook.
The second is a summariser. Support gets around ninety thousand messages a year, and one of the developers wired a prompt to an Amazon Bedrock model that turns a support thread into three lines an agent can read at a glance. It also works. It is also a script on a laptop, with the prompt pasted into a string literal.
The ask from the operations director is one page. Draw the pipeline that puts both of these into production, name the stage each box is, and name the AWS service that would do the work. Nobody has asked for a build yet. What they want first is the map, and the arguments in the room are all about which boxes exist and which words belong on them.
What actually matters
The first four boxes get collapsed into “data prep” by almost everybody, and they are four different jobs with four different outputs. Data collection gets raw records into one place you are allowed to read: pulling the subscriptions table, the deliveries table and the support inbox out of the systems that own them, plus any data bought from outside. Exploratory data analysis (EDA) is looking without changing: distributions, missing values, how many subscribers have never paused, whether the support inbox is half automated receipts. EDA produces understanding and a list of problems, not a file. Data pre-processing is fixing those problems: deduplicating, filling or dropping missing values, correcting types, joining the three tables into one, encoding categories. It produces a clean dataset. Feature engineering is inventing the input variables the model will actually learn from, which is a different act again: days since the last pause, boxes per month over the trailing quarter, substitution rate. The clean dataset is what happened; the features are what you think matters about it.
The middle three get conflated in a different way. Model training fits weights to the training split with the hyperparameters held still. Hyperparameter tuning runs training many times over with different settings: learning rate, tree depth, number of epochs. It picks the combination that scores best on the validation split. It is a search across training runs rather than a stage after one. Evaluation then scores the chosen model against data it has never seen, on metrics like accuracy, precision, recall and F1, and adds the fairness and explainability checks. Tuning optimises. Evaluation judges, and it is allowed to say no.
Deployment and monitoring look adjacent and behave nothing like each other. Deployment is an event: the model becomes something that answers requests, at an address, with a version number. Monitoring is a standing process that starts the moment deployment finishes and never stops, watching for input data that no longer looks like the training data, for accuracy decay, for latency and for cost. Monitoring is also the stage that closes the loop: what it finds becomes the reason for the next round of data collection and retraining.
Now the structural claim that makes one page possible for both projects. The classic ML pipeline and the foundation-model pipeline share their beginning and their end, and differ in the middle. Both start with the same four data stages and finish with the same three. In the middle, a classic pipeline produces a model: you engineer features, you train, you tune, and the artefact you end up owning is a set of weights. A foundation-model pipeline chooses one. The weights already exist, so the work is selecting a model and adapting it with prompts, retrieval and sometimes fine-tuning. The artefact you end up owning is a prompt and a configuration. That is why the churn model and the summariser can share a diagram, and why the two boxes in the middle should not be labelled the same.
Two smaller decisions sit inside that middle. Where the model comes from is one of three answers. Open source pre-trained models, whose weights you can download and deploy yourself. Proprietary models, which you reach through a vendor’s API. Or training custom models from scratch on your own data, the most expensive answer and the rarest. How it gets served is one of two. A managed API service leaves the servers, the scaling and the patching with AWS, and you make an API call. A self-hosted API runs the model on infrastructure in your account, where the instance type, the auto scaling and the security patches are yours. The economics of that hosting choice get worked in detail elsewhere; at this level, name the two shapes and know who owns the servers in each.
What we’ll filter on
- Which stage of the AI/ML pipeline the service actually serves, and whether it serves only that one.
- Who owns the infrastructure the stage runs on: AWS, or you.
- What artefact comes out: raw data, a clean dataset, features, model weights, or a prompt and a chosen model.
- Whether the stage runs once per cycle or runs continuously once it has started.
- Which pipeline it belongs to: the classic one, the foundation-model one, or both.
The landscape
Nothing below is a full tour of what each service is; these are the stages each one turns up at.
Getting the data together
Amazon S3 is where the collected data lands and stays, and every other stage reads from it. AWS Glue is the extract, transform and load service that moves records out of the source systems into it, and catalogues what arrived. AWS Data Exchange is for data collection from outside the organisation: subscribing to a third-party dataset, such as postcode-level demographics, and having it delivered into your account. AWS Lake Formation sits over the top and grants governed access, so the analyst who needs deliveries data does not get the support inbox with it. Collection is a stage that runs on a schedule forever, not a one-off import.
Looking at it, cleaning it, and shaping features
Amazon Quick Sight, the business-intelligence part of Amazon Quick, is where a non-notebook look at the data happens: charts, distributions and dashboards over what has been collected, so the people who know the business can do exploratory data analysis (EDA) without writing code. It is the same surface a production dashboard lands on later. The other part of Amazon Quick, the finished assistant that answers questions over company documents with citations, has no stage on this map. SageMaker Data Wrangler does the same looking from inside SageMaker AI and then carries straight on into data pre-processing, because the transformations you select there become a repeatable flow. AWS Glue DataBrew is the visual preparation tool for the same job on data-engineering terms, with a library of built-in transformations and no code. SageMaker Feature Store is the only one of the four that belongs to feature engineering rather than preparation. Features are ingested once and read twice: at low latency for live predictions, and as history for training. That is how a feature means the same thing in both places.
Training, tuning and evaluating
SageMaker AI training jobs are where model training happens: you point a job at the training data and an algorithm or container, it provisions instances, runs, writes the model artefact to S3 and shuts the instances down. Automatic model tuning is the same service running that job repeatedly for hyperparameter tuning and reporting which combination won. SageMaker JumpStart is the model hub: open source pre-trained models you can deploy or fine-tune onto your own endpoints, which is how it appears in a training box even though nothing is trained from scratch.
For evaluation, SageMaker Clarify measures bias in the data before training and in the model after it, and attributes a prediction to the features that drove it. Amazon Bedrock Model Evaluation is the foundation-model equivalent. It scores candidate models against a task, either automatically or with human reviewers. That gives the summariser a number to compare on, rather than an impression formed from reading ten outputs.
Serving and watching
Amazon Bedrock is the managed API service: a single API over proprietary and open-weight foundation models, with no instances in your account. A SageMaker AI endpoint is the self-hosted API for a model you own, whether that is the churn model or a JumpStart deployment. Amazon EC2, Amazon ECS and Amazon EKS are the same idea further down the stack, with more of the running yours. The same model can often be reached either way, and the difference is ownership rather than capability.
After deployment, SageMaker Model Monitor watches a live endpoint for data quality problems, drift and bias appearing over time. Amazon CloudWatch carries everything else: invocation counts, latency, errors, and the alarms that wake somebody.
Where the model comes from
Three sources, and this is a stage in the foundation-model pipeline rather than a procurement question. Open source pre-trained models come through SageMaker JumpStart, where you get the weights and the responsibility for hosting them. Proprietary models come through the Amazon Bedrock API, where you get access and the provider keeps the weights. Training custom models from scratch means SageMaker AI training jobs, your own corpus, and a bill that only a very specific requirement justifies. Most teams choosing a foundation model are choosing between the first two, and the data you feed it needs its own checks whichever way that goes.
The build side
Kiro is the agentic development environment the pipeline code gets written in: the notebook’s forty cells of cleaning become a Glue job and a Data Wrangler flow, and that is code somebody has to write. Strands Agents is the SDK for the case where the summariser grows into something that calls tools rather than just returning text. Neither is a pipeline stage. They are how the stages get built, and they turn up on the in-scope service list for that reason. Kiro, Amazon Quick and Bedrock are easy to blur together because all three answer questions; they sit at different places on this map.
Evaluation
Side by side
| Stage | AWS services | Artefact produced | Feeds | Classic ML | Foundation model |
|---|---|---|---|---|---|
| Data collection | Amazon S3, AWS Glue, AWS Data Exchange, AWS Lake Formation | Raw records in one governed place | EDA | ✓ | ✓ |
| Exploratory data analysis (EDA) | Amazon Quick Sight, SageMaker Data Wrangler | Understanding and a list of problems | Data pre-processing | ✓ | ✓ |
| Data pre-processing | AWS Glue DataBrew, SageMaker Data Wrangler | A clean, joined dataset | Feature engineering or model selection | ✓ | ✓ |
| Feature engineering | SageMaker Feature Store | Named features, online and offline | Model training | ✓ | ✗ |
| Model training | SageMaker AI training jobs, SageMaker JumpStart | Model weights in S3 | Hyperparameter tuning | ✓ | ✗ |
| Hyperparameter tuning | SageMaker AI automatic model tuning | The winning hyperparameter set | Evaluation | ✓ | ✗ |
| Model selection | SageMaker JumpStart, Amazon Bedrock | A chosen model and its source | Adaptation | ✗ | ✓ |
| Adaptation | Amazon Bedrock prompts, knowledge bases, fine-tuning | A prompt and a configured model | Evaluation | ✗ | ✓ |
| Evaluation | SageMaker Clarify, Amazon Bedrock Model Evaluation | A ship or no-ship verdict with numbers | Deployment | ✓ | ✓ |
| Deployment | Amazon Bedrock, SageMaker endpoints, Amazon EC2, ECS, EKS | A callable, versioned endpoint | Monitoring | ✓ | ✓ |
| Monitoring | SageMaker Model Monitor, Amazon CloudWatch | Drift, quality, latency and cost signals | Data collection | ✓ | ✓ |
| Build tooling | Kiro, Strands Agents | The code the stages run as | Every stage | ✓ | ✓ |
Read the last two columns together and the shape falls out. Seven of the twelve rows are ticked twice, which is the shared spine. Three rows are classic-only and two are foundation-model-only, and they all sit between preparation and evaluation. The Feeds column is worth reading down as well: it ends where it started, because monitoring is what tells you the data has moved on.
The two pipelines on one page
The dashed box on the right is the one people argue about. Leaving the foundation-model lane empty there invites somebody to write “training” in it, and a foundation-model pipeline has no training pass unless you have chosen to fine-tune. Saying so on the page settles the argument once.
The solution
The one page is three bands. The top band is shared, and runs left to right through data collection, exploratory data analysis (EDA) and data pre-processing. AWS Glue moves data into Amazon S3 and AWS Lake Formation grants access to it. Amazon Quick Sight, SageMaker Data Wrangler and AWS Glue DataBrew do the looking and the cleaning. Both projects sit in this band together. The support inbox and the subscriptions table are collected the same way and cleaned the same way. One ends up as features and the other as prompt context, and neither of those changes how it gets into S3.
The middle band splits. On the churn side, features go into SageMaker Feature Store, a SageMaker AI training job fits the model, automatic model tuning searches the hyperparameters, and the artefact is a set of weights in S3 with a version. On the summariser side, model selection picks between an open source pre-trained model through SageMaker JumpStart and a proprietary model behind the Amazon Bedrock API. Adaptation is prompt work, retrieval over the support archive, and fine-tuning only if the prompt work runs out of road. The artefact is a prompt, a model identifier and a set of inference parameters, all of which belong in version control exactly as the training code does.
The bottom band rejoins. Evaluation runs SageMaker Clarify over the churn model for accuracy, precision, recall, F1 and bias across cities. Amazon Bedrock Model Evaluation scores two or three candidate summarisers against a set of real threads with known good summaries. Deployment splits by ownership rather than by pipeline. The churn model goes to a SageMaker AI endpoint, a self-hosted API where you pick the instance type and own the scaling and the patching. The summariser calls Amazon Bedrock, a managed API service where none of that is yours. Monitoring covers both, with SageMaker Model Monitor watching the endpoint for drift against the training baseline and Amazon CloudWatch carrying invocation counts, latency, error rates, token spend and the alarms.
Three things are worth writing on the page while it is being drawn. The first is that the forty cells of cleaning have to exist in exactly one place, used by both training and serving. A feature computed one way at training time and another way at request time gives a model that is wrong from its first day live. Feature Store exists for that reason. The second is that the arrow from monitoring back to data collection is a real arrow and not decoration; drift is discovered there and fixed at the top of the map. The third is that Kiro and Strands Agents are not stages. They sit alongside the diagram as how the boxes get built, because somebody still has to write the Glue job, the endpoint configuration and the code that calls Bedrock.
Worked example
Follow one subscriber through the left lane. Their row is collected nightly by an AWS Glue job into S3 alongside every delivery and every pause. Nobody changes it during EDA in Amazon Quick Sight, but that is where somebody notices eleven per cent of postcodes are blank. Data pre-processing in SageMaker Data Wrangler drops those rows, joins deliveries to subscriptions and casts the pause dates. Feature engineering turns the row into four numbers (weeks subscribed, days since last pause, substitution rate, boxes skipped in the trailing quarter) and writes them to SageMaker Feature Store. Model training reads a year of those feature values. Hyperparameter tuning runs it thirty times to settle the tree depth, and evaluation scores the winner on a held-out month. Deployment puts it behind a SageMaker endpoint. Four months later, monitoring notices that substitution rate has drifted because the summer range changed.
Now one support thread through the right lane. It is collected into the same bucket by the same Glue job. EDA finds that a third of threads are automated delivery receipts with no human text, and data pre-processing strips them. There is no feature engineering; the thread stays as text. Model selection compares a JumpStart-hosted open-weight model with two models on Amazon Bedrock. Adaptation writes the prompt and adds retrieval over past resolved threads. Amazon Bedrock Model Evaluation then scores all three against forty threads a support lead has already summarised by hand. Deployment is a Bedrock API call from the support tool. Monitoring is CloudWatch on latency, token cost per thread and the rate at which agents rewrite the summary before sending it.
What’s worth remembering
- Data collection gathers, exploratory data analysis (EDA) looks without changing, data pre-processing fixes what EDA found, and feature engineering invents the variables the model learns from; four stages, four different outputs.
- Model training fits weights with settings held still, hyperparameter tuning searches across many training runs for the best settings, and evaluation scores the winner on unseen data and can refuse to ship it.
- Deployment happens once per version and monitoring runs forever afterwards, feeding what it finds back into the next round of data collection.
- The classic and foundation-model pipelines share the four data stages and the three production stages, and differ only in the middle: one produces weights, the other selects a model and adapts it.
- A model comes from one of three places: open source pre-trained models through SageMaker JumpStart, proprietary models through the Amazon Bedrock API, or training custom models from scratch on your own data.
- A managed API service like Amazon Bedrock leaves the servers, scaling and patching with AWS, while a self-hosted API on a SageMaker AI endpoint, Amazon EC2, ECS or EKS leaves all three with you.