The situation
A subscription media company is two weeks from launching a subscriber-facing assistant. It answers billing and account questions on the help pages: what a plan includes, why an invoice changed, how to pause. It runs an Amazon Bedrock model over a knowledge base built from the help centre and the plan terms, with an agent that calls two tools, one to read account state and one to raise a ticket when it cannot answer. Traffic modelling says around forty thousand conversations a week in the first month.
The platform group will not sign the launch off without a written architecture review. Two reviews have already happened. Both were meetings, both produced a page of notes in a document, and nobody can say which of the things raised in the first one were ever done. The engineering manager wants the third attempt to leave behind something that still exists in six months, and the compliance lead wants something she can hand to an auditor without writing a covering essay first.
Nobody disagrees that a review should happen. The argument is about what kind, how long it takes, and what it produces.
What actually matters
A review earns its cost by producing risks the team had not already written down. If it surfaces only what the engineers already worry about in stand-up, it consumed a week and moved nothing. That means the questions have to come from outside the team, because a team cannot interrogate its own blind spots by talking harder about them. Structure is what makes questions arrive from outside. A fixed set someone else wrote, asked in the same order every time, keeps the review from drifting towards whatever the loudest person in the room finds interesting.
The questions also have to match the technology. General architecture questions are good questions. Blast radius, failure modes, recovery time, cost shape, who is on call: all of those will find real problems in this workload, because it is a distributed system like any other. What they will not ask is whether the model was picked against an evaluation set or against a demo. Or whether a document retrieved from the knowledge base can carry an instruction the model then follows. Or whether an answer is grounded in retrieved text or invented, what the token spend looks like when a subscriber pastes an entire invoice history into the box, what happens on the day the model version is retired, and where a human sits between a wrong answer and a subscriber acting on it. A general review passes a workload with none of those handled, because it never asked. Producing comprehensive architectural designs that align with specific business needs and technical constraints is only half the job here. One of the constraints is a component whose behaviour is probabilistic, and the pillars have no vocabulary for that.
Then there is what the review leaves behind. A document records what people thought on a Tuesday. Six months later nobody knows which findings were fixed, which were accepted, and which were forgotten, because a document has no state. What survives is a record: the question, the answer given, whether it was judged high risk, who owns it, and a dated point to measure from. The dated point matters more than it sounds, because the second review is only useful if it can say what moved. Without a baseline, every review starts from zero and produces the same list.
Finally, the review has to fit the two weeks it has. A review that lands three weeks after launch is a post-mortem. Scoping is how that gets solved, and scoping honestly is harder than it looks, because a long questionnaire has a specific failure mode: one person sits down and answers all of it alone in an afternoon to unblock the release. What comes out is a completed form and no risk list. Answering the sections that apply, with the people who actually know, produces fewer answers and more findings.
What we’ll filter on
- Coverage: does it ask generative-AI-specific questions about model selection, grounding and hallucination, prompt injection, token cost, model deprecation, and human oversight?
- Structure: are the questions fixed and written by someone outside the team, so the review is repeatable rather than shaped by whoever is in the room?
- Output: does it leave a tracked record with high-risk items, owners, and dated milestones, or a document?
- Effort: can it be scoped to the parts of the workload that exist and finished before launch?
- Evidence: is the output something a compliance lead can hand over as-is?
- Reuse: do the findings feed anything the next team inherits, or does the next workload start from a blank page?
The landscape
The first option is the one the company has already tried twice: an internal design review. Senior engineers read the diagrams, ask what they think to ask, and write notes. It is cheap, it can happen this week, and the quality of it is exactly the quality of the people in the room on the day. It is not repeatable, it has no fixed question set, and the notes have no status field.
The AWS Well-Architected Framework is the structured alternative. It is a body of prose organised into six pillars: Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, and Sustainability. Each pillar carries design principles, best practices, and a set of questions. The questions do the work: a review is conducted by answering them about a specific workload, not by reading the pillars. The Framework itself is guidance. It has no memory of your workload and no notion of whether you did anything about what it told you.
The AWS Well-Architected Tool is where the workload lives. You create a workload record, describe it, choose which lenses apply, and answer the questions. The Tool holds the answers, marks the ones flagged as high risk, lets you write an improvement plan against them, and lets you save a milestone, which freezes the current state of every answer as a named point in time. It produces a report you can export. The Framework tells you what to ask. The Tool remembers what you said, what was risky about it, and what you did next.
The AWS WA Tool Generative AI Lens is a lens applied to a workload in that Tool, and it adds questions the six pillars do not ask. It is organised around the lifecycle of a generative-AI workload rather than around the pillars, in six phases. Scoping asks whether generative AI is the right approach for this use at all, what the business outcome is, and how success will be measured, which is the question that should have been asked before any of this was built. Model selection asks how the model was chosen: benchmarks, evaluation against your own data, cost per token at your volumes, latency, regional availability, and what the fallback is. Model customisation asks about fine-tuning and retrieval: data provenance, whether the customisation was evaluated against the base model, and what governs the resulting artefacts. Development and integration asks about prompt management, retrieval design, guardrails, injection defence, and the tools an agent is permitted to call. Deployment covers the path to production, testing, rollout, and rollback, the same ground as promoting a proof of concept. Continuous improvement asks what happens after launch: what is monitored, how drift and quality regressions are detected, how feedback reaches the prompts, and how model deprecation is handled.
A lens is not limited to the ones AWS publishes. A custom lens is how an organisation adds its own questions to the Tool, so that the things this company always gets wrong get asked every time alongside the AWS ones. That matters later rather than now, because a custom lens is worth writing once you have run enough reviews to know what your recurring findings are.
The fourth option is a scoping decision rather than a different tool: apply the lens, and answer only the phases the workload has actually reached. This assistant has been scoped, has a model selected, uses retrieval but no fine-tuning, is fully built, and is about to deploy. Continuous improvement exists only as a plan. Answering model customisation as a page of “not applicable” adds rows and no risk.
Evaluation
Side by side
| Review | GenAI-specific questions | Fixed, repeatable question set | Tracked plan, owners, milestones | Fits two weeks | Exportable evidence | Feeds reusable components |
|---|---|---|---|---|---|---|
| Internal design review | ✗ (whatever the room asks) | ✗ | ✗ (notes in a doc) | ✓ | ✗ | ✗ |
| Six pillars in the WA Tool | ✗ | ✓ | ✓ | ✗ (all six pillars) | ✓ | ✓ (slowly) |
| Lens applied, every phase answered | ✓ | ✓ | ✓ | ✗ | ✓ (padded) | ✓ |
| Lens applied, phases in play only | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Read the first column and the second row is the trap. A full six-pillar review is a serious piece of work that produces a real improvement plan, and it will still hand this team a clean bill of health on a workload where nothing has been tested against prompt injection, because injection is not a Reliability question, a Cost Optimization question, or any of the other four. The pillars are about the system around the model. The lens is about the model and everything the model makes uncertain.
Read the fourth column and the third row is the other trap. Answering every phase looks thorough and produces the tick-box outcome, because the sections that do not apply are answered fastest and set the tempo for the ones that do.
Which review, and why
The gates are in that order because each one is cheaper to answer than the next. Whether the review produces a record is a decision about tooling and takes a minute. Whether the questions cover generative AI is a decision about lenses and takes an hour of reading. Whether the scope is honest is a judgement about your own workload, and that is the one worth spending the afternoon on.
The solution
Create the workload in the AWS Well-Architected Tool and apply the AWS WA Tool Generative AI Lens to it. Answer the phases the assistant has reached, which here means scoping, model selection, development and integration, deployment, and the parts of continuous improvement that describe what will happen after launch. Mark model customisation as out of scope with a sentence saying why, so the next reviewer knows it was considered rather than skipped. Run the sessions with the people who hold the answers: the engineers for integration and deployment, the product owner for scoping, the compliance lead for oversight and evidence. Three sessions of ninety minutes each will do more than one person working through the whole list.
Save a milestone the day the assistant goes live and name it for the launch. That milestone is the baseline the next review moves from. Six months in, a second pass reopens the same questions, and the Tool shows which answers changed and which high-risk items are still open, so the conversation is about movement rather than about whether anyone remembers what was said in August.
Then route each high-risk item to the decision that closes it, rather than to a summary. A finding about unbounded token spend belongs against a spend guardrail with an actual limit on it. A finding that only this team can reach the model, with no story for the next three teams, belongs against how model access is governed across teams. A finding that there is no way to reconstruct what the assistant told a subscriber belongs against the work that makes the application audit-ready. A high-risk item with an owner, a date, and a named piece of engineering against it is a plan. One with a paragraph of intent against it is the same document the last two reviews produced.
The lens has no teeth, and that is fine if you know it
The lens is guidance. Answering its questions changes nothing in the account. There is no policy that blocks a deployment because a question was answered badly, no alarm when an answer goes stale, and no relationship at all between a green-looking workload record and what the running application does. The review tells you a guardrail should exist; only the guardrail stops anything. So the improvement plan has to terminate in enforcement: a guardrail configured, an evaluation gate in the pipeline, a budget action, an IAM boundary. A review that closes its own findings by recording that they were discussed has closed nothing.
A milestone freezes a moment, not the workload
A milestone is a snapshot of the answers on the day you saved it. The workload keeps moving. Swap the model version, add a second data source, give the agent a third tool that can write rather than read. A good number of the answers are now wrong, and the record still shows them as accepted. Treat the milestone as evidence of what was true at launch, and re-answer the affected phases when the workload changes shape. A model swap invalidates model selection and most of development and integration. A new data source invalidates the retrieval answers and probably the injection ones. Trusting last quarter’s review through a change like that is how a workload becomes compliant on paper and unreviewed in fact.
Do not answer it all in one sitting
The strongest predictor of a worthless review is one person completing the whole question set alone to unblock a release. The questions are written to be argued about, and the value is in the argument, not the answer field. Split by phase, put the right people in each session, and let a question that nobody can answer stay open and become a finding, because “we do not know” is the most useful thing a review can produce two weeks before launch.
Turning findings into components the next team inherits
The reviews start repeating themselves after three or four workloads. Every team gets asked whether a guardrail is configured, whether model invocation logging is on and going somewhere retained, and whether the invoking role is scoped to specific model ARNs rather than to Bedrock as a service. Every team answers from scratch, slightly differently. That repetition is the signal to stop reviewing the same decision and start shipping it.
The move is to write the recurring answers once as standardized technical components so that you get consistent implementation across multiple deployment scenarios instead of five teams converging on five variants. In practice that is an AWS CDK construct, or an AWS CloudFormation template. It provisions the guardrail, the invocation logging configuration and its destination, the KMS key, and the scoped invocation role as one unit, with the review’s conclusions already in its defaults. Publish it through AWS Service Catalog as a product the platform group owns and versions, and the next team launching an assistant inherits the review instead of repeating it. Their lens session then spends its time on what is genuinely new about their workload, because the questions the component already answers are answered by pointing at the component.
The organisation’s own recurring questions go into a custom lens beside the Generative AI Lens, so the review asks them every time rather than depending on whether the reviewer remembers. Between the custom lens and the Service Catalog product, the second workload starts where the first one finished.
Worked example
The question nobody could answer
The model selection session took twenty minutes to fall over. The lens asks how the model was chosen and against what evidence, and the honest answer was that an engineer had tried three models on about a dozen sample questions in March, one had felt better, and that model had been in the code ever since. There was no evaluation set, no record of the dozen questions, and no way to tell whether a newer model would be better, worse, or cheaper.
That became a high-risk item, and the thing that made it useful was that it was not fixable in two weeks. The plan against it was two dated items. Before launch, capture two hundred real questions from the help centre logs and record the current model’s answers as a baseline. That is a day’s work, and it produces the evaluation set that did not exist. After launch, run that set against two alternative models and write down the result. The launch went ahead. The finding stayed open with a date on it, which is a different state from being forgotten.
The finding that became a Service Catalog product
The development and integration session found that the assistant had no guardrail configured. The reasoning had been that the knowledge base only contains help-centre content, so there is nothing harmful to retrieve. That argument survives right up until someone points out that the subscriber types the input, and that the agent can be talked into using its ticket-raising tool as an output channel.
The fix was a guardrail, an afternoon’s work. What made it worth more than an afternoon was the platform group noticing something. The same finding had come up in a review of the internal search assistant six weeks earlier, and had been fixed there separately, with different denied topics and a different logging destination. So the guardrail, the logging configuration, and the scoped invocation role went into a CDK construct with sensible defaults, and the construct went into Service Catalog. The third assistant to launch got all three by declaring one resource, and its review spent its time on the tool permissions instead, which was the part that was actually new.
What’s worth remembering
- The AWS Well-Architected Framework is a question set organised into six pillars (Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, Sustainability); the AWS Well-Architected Tool is the workload record holding the answers, high-risk items, improvement plan, and milestones.
- The AWS WA Tool Generative AI Lens adds the questions the six pillars never ask, organised by lifecycle phase: scoping, model selection, model customisation, development and integration, deployment, and continuous improvement.
- Answer only the phases the workload has reached and mark the rest out of scope with a reason; answering everything in one sitting produces a completed form rather than a risk list.
- Save a milestone at go-live so the next review measures movement, and re-answer the affected phases whenever the model, the data sources, or the agent’s tools change.
- The lens produces guidance and no enforcement, so every high-risk item has to terminate in something that actually blocks the failure: a guardrail, an evaluation gate, a budget action, a scoped role.
- When the same finding appears in three reviews, ship it as standardized technical components, a CDK construct or CloudFormation template published through AWS Service Catalog, so the next team inherits the answer.