Lab 11 — Managed RAG with a Bedrock Knowledge Base

Scaffold: 3/5. The first of the managed-track labs. Lab 05 made you write retrieval by hand; here the same documents go into a Knowledge Base and you write the two calls that query it.

The scenario

The Greenbox support assistant works, and the retrieval loop behind it is five documents held in memory, embedded on cold start, scored with a cosine function you wrote. That is fine for five documents and hopeless for five thousand: nothing re-embeds when a document changes, nothing chunks a long document into pieces small enough to retrieve usefully, and every cold start pays to embed the whole corpus again.

A Knowledge Base takes that job. It watches a bucket of documents, chunks them, embeds the chunks, keeps them in a vector index, and re-embeds only what changed when you sync. The same five topics are here, expanded into documents long enough that chunking matters, plus one that support staff can read and subscribers must not.

The requirement

• “When will my box arrive?” is answered from the documents and comes back with the S3 URI of the document it used.
• The retrieved chunks come back too, with their similarity scores, so you can see what the answer was built from rather than trusting it.
• “How much goodwill credit can I get?” answers from the internal runbook when nothing is filtered, and stops answering when the query is filtered to audience = subscriber.
• A question no document covers gets an honest “I do not know”.

What’s provided

• template.yaml — an S3 bucket for the documents, an S3 Vectors vector bucket and index, the Knowledge Base and its service role, an S3 data source with fixed-size chunking configured, and a query Lambda with its role. All of it is CloudFormation-native, including the vector store.
• docs/ — five Greenbox documents, each with a .metadata.json sidecar carrying topic and audience attributes.
• src/handler.py — request parsing, the response helper, and _vector_search_config() which builds the retrieval configuration including the metadata filter. The gaps are retrieve() and answer().
• solution/handler.py — the reference answer.
• scripts/ — deploy (which also uploads the documents and runs the ingestion job), test, teardown.

Your task

In src/handler.py, fill the two gaps:

1. retrieve(question, k, audience) — call Retrieve on bedrock-agent-runtime and reshape retrievalResults into {"text", "score", "source", "audience"} dicts. This is the raw search: no model, no prose, just chunks and scores.
2. answer(question, k, audience) — call RetrieveAndGenerate, which does the same search and then writes an answer over the results. Return the text from output.text and the de-duplicated S3 URIs from citations[].retrievedReferences[].location.s3Location.uri.

Both pass their retrieval configuration through _vector_search_config(), so numberOfResults and the metadata filter are wired for you. The docstring has the exact request and response shapes.

Run it

# Prerequisite: Model access enabled for BOTH the generation model and the
# Titan embedding model, in your region.
./scripts/deploy.sh          # ~5 minutes the first time: the KB and index build
./scripts/test.sh
./scripts/teardown.sh

Defaults are stack genai-lab-11, region us-east-1, amazon.nova-lite-v1:0 for generation and amazon.titan-embed-text-v2:0 at 1024 dimensions for embedding. Override any of them with environment variables (STACK, AWS_REGION, MODEL_ID, EMBED_MODEL_ID, EMBED_DIMENSIONS, CHUNK_MAX_TOKENS, CHUNK_OVERLAP_PERCENTAGE).

What it costs. Small, and mostly one-off. Embedding five short documents is a fraction of a cent. S3 Vectors bills about USD$0.06 per GB-month of vectors and a per-query fee measured in dollars per million queries, so a few hundred vectors and a handful of test queries round to nothing. Nova Lite generation for the six test questions is well under a cent. The reason this lab uses S3 Vectors rather than OpenSearch Serverless is the shape of the bill: an OpenSearch Serverless collection charges for capacity units whether or not you query it, and forgetting to delete one is the expensive mistake in this track. Tear down when you finish anyway.

What success looks like

Before you fill the gaps, the function raises NotImplementedError. After:

• “When will my box arrive?” comes back with Tuesdays and Fridays, a citation pointing at s3://.../delivery-days.txt, and three chunks with scores attached.
• Dropping k to 1 on the card-declined question narrows what the answer can draw on; watch the citation list shrink.
• The goodwill-credit question answers from ops-runbook.txt unfiltered, then refuses once the filter pins audience to subscriber, because the runbook chunks are no longer eligible to be retrieved.
• The carbon-footprint question comes back with an honest refusal.

Then change something. Edit docs/delivery-days.txt to add a Wednesday run, re-run ./scripts/deploy.sh, and ask again. The upload and the ingestion job are both in the deploy script, and Bedrock only re-embeds what changed.

If it fails

• The stack fails to create the Knowledge Base with an access error on the role — IAM propagation. A create that fails this way rolls back and leaves the stack in ROLLBACK_COMPLETE, which CloudFormation cannot update, so the stack has to be deleted before you retry. Run ./scripts/deploy.sh again: it spots that state, deletes the stack, and deploys from scratch. The second attempt usually succeeds.
• AccessDeniedException naming a model — enable Model access for both the embedding model and the generation model. They are separate grants, and the embedding one is the easy one to forget.
• ValidationException about on-demand throughput — your generation model is only served through a cross-region inference profile. Redeploy with MODEL_ARN=arn:aws:bedrock:us-east-1:<account>:inference-profile/us.amazon.nova-lite-v1:0.
• The ingestion job reports documents failed — deploy.sh prints failureReasons. A malformed .metadata.json is the usual cause; the sidecar must be valid JSON and named exactly <document>.metadata.json.
• Retrieval returns nothing — the ingestion job ran but found no documents, or the filter excludes everything. Check the ingestion statistics printed by the deploy script, then try the same question with no audience.
• The filter has no effect — filter keys come from the sidecar files, so audience must exist in docs/*.metadata.json and the documents must have been re-ingested since you added it.
• Teardown fails on the vector bucket — a vector bucket has to be empty to be deleted. Check the console for an index CloudFormation could not remove, delete it, and re-run teardown.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• A Knowledge Base is the Lab 05 loop, operated for you. Chunk, embed, index, search. What you gain is sync, scale, and citations; what you give up is control over exactly how it chunks and scores.
• Chunking is configured on the data source, not the knowledge base. ChunkingStrategy: FIXED_SIZE with MaxTokens and OverlapPercentage is the setting, and it cannot be changed after the data source is created. Changing it means replacing the data source and re-ingesting.
• Ingestion is an operation, not a resource. Nothing happens until StartIngestionJob runs, and re-running it after a document changes is what keeps answers current. Bedrock crawls incrementally: it re-embeds the changed documents and leaves the rest alone.
• Retrieve and RetrieveAndGenerate are two different jobs. Retrieve gives you chunks and scores and no model call, which is what you want for debugging retrieval, for reranking yourself, or for feeding your own prompt. RetrieveAndGenerate does the whole thing and hands back citations. Use the first when you want to see, the second when you want an answer.
• numberOfResults is top-k, and it is a real dial. Too few and the answer misses context that exists; too many and the model is drowning in near-misses, at a token cost you pay every query.
• Metadata filters are access control you can afford. The filter keys come from the .metadata.json sidecar next to each document, and filtering at retrieval time keeps the internal runbook out of a subscriber’s answer far more reliably than asking the model nicely.
• S3 Vectors puts the vector store on the same bill as the documents. It gives up hybrid search and ultra-low latency; for a corpus this size, and for most internal RAG, that is a trade worth making.

Next

Lab 12 — Fine-tune a model and read the loss curves. Retrieval fixes what the model does not know. Fine-tuning fixes how it answers: you prepare a training set, run a customisation job, and measure whether the tuned model is actually better than the prompt you already had.