Lab: Stand Up a Bedrock Knowledge Base

This starts the managed track of the hands-on labs. The first ten build everything by hand against the model API. These two use a managed service instead, and the point of going second is that you already know what the service is doing for you. The from-scratch lab made you write embed-compare-rank yourself; this one hands the same Greenbox documents to a Knowledge Base and asks you to write the two calls that query it. The full lab is in lab-11-knowledge-base.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. It reaches this lab’s Knowledge Base and its S3 Vectors store too.

The scenario

The support assistant works. Behind it are five documents held in memory, embedded on cold start, scored with a cosine function you wrote. That is fine for five documents. It falls over at five thousand: nothing re-embeds when a document changes, nothing splits a long document into pieces small enough to match precisely, and every cold start re-embeds the whole corpus and is billed for every token of it.

A Knowledge Base takes that job. It crawls a bucket when you sync it, chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. what it finds, embeds the chunks, keeps them in a vector index, and on later syncs re-embeds only what changed. The documents in this lab are the same five Greenbox topics, expanded until they are long enough that chunking matters, plus one internal support runbook that staff can read and subscribers must not.

What you’re given

CloudFormation builds the lot: a bucket for the documents, an S3 Vectors vector bucket and index, the Knowledge Base with its service role, an S3 data source with fixed-size chunking configured, and a query Lambda. Every piece is a native CloudFormation resource, so nothing is created out of band, and AWS::S3Vectors::VectorBucket plus AWS::S3Vectors::Index mean the vector store is torn down with the stack.

S3 Vectors rather than OpenSearch Serverless is a cost decision. A collection is billed for its capacity units whether or not you query it, and an orphaned one is the expensive mistake in this track. A vector bucket is billed for the vectors it stores and the queries you run against it, which for a few hundred vectors rounds to nothing.

The docs/ directory carries a .metadata.json sidecar next to each document, tagging it with a topic and an audience. Those become filterable attributes on every chunk, which is what makes the last part of the lab work.

src/handler.py has the request parsing, the response helper, and a small function that builds the retrieval configuration. Two gaps are left.

Your task

First, the raw search. No model, no prose, just chunks and scores. retrieve() is one call to the client’s Retrieve operation: the Knowledge Base id, a retrieval query carrying the question text, and a retrieval configuration whose vectorSearchConfiguration comes from the helper below. The reply is a list of retrievalResults, each holding the chunk text under content, a score, the source document under location.s3Location.uri, and the sidecar attributes under metadata. Reshape each result into a dict of text, score, source and audience, in the order the service returned them.

Then the whole thing, search and generation together, with citations attached. answer() calls RetrieveAndGenerate: the question goes in as the input text, and the configuration is type KNOWLEDGE_BASE, naming the Knowledge Base id, the generation model ARN, and the same vector search configuration from the helper. The generated answer comes back under output.text, and each entry in citations links a span of that text to the retrievedReferences behind it. Walk those references, collect the S3 URIs de-duplicated in first-seen order, and return the answer text alongside that list. The module docstring in src/handler.py has the exact request and response shapes for both calls.

Both go through the same helper, which is where top-kTop-kHow many chunks a retrieval step returns per query – the dial that trades answer coverage against token cost. and the metadata filter live:

def _vector_search_config(k, audience):
    config = {"numberOfResults": k}
    if audience:
        config["filter"] = {"equals": {"key": "audience", "value": audience}}
    return config

Note the client. Retrieve and RetrieveAndGenerate are on bedrock-agent-runtime, not the bedrock-runtime every earlier lab used. Same account, same credentials, different service endpoint, and reaching for the wrong one is the first thing that goes wrong.

Deploy and prove it

cd lab-11-knowledge-base
./scripts/deploy.sh
./scripts/test.sh
./scripts/teardown.sh

The first deploy takes about five minutes, most of it the Knowledge Base and index coming up. After the stack, the script uploads docs/, uploads your handler, then starts an ingestion job and polls until it completes, printing how many documents were scanned and indexed.

The test script asks six questions. “When will my box arrive?” comes back with Thursdays and Fridays, a citation pointing at delivery-days.txt, and the chunks it drew on with their scores. The card-declined question runs twice, once at three chunks and once at one, so you can watch the answer narrow. “How much goodwill credit can I get?” answers from the internal runbook when nothing is filtered, and comes back with no answer once the query is pinned to audience = subscriber. The carbon-footprint question comes back with “I don’t know”.

Then change a document. Edit docs/delivery-days.txt to add a Wednesday run, run ./scripts/deploy.sh again, and ask again. The answer moves, because the deploy script re-uploads and re-syncs, and Bedrock re-embeds only the document that changed.

When you want the reference answer, deploy it with SRC=solution ./scripts/deploy.sh, or unfold it here:

def retrieve(question, k=3, audience=None):
    response = _agent.retrieve(
        knowledgeBaseId=KNOWLEDGE_BASE_ID,
        retrievalQuery={"text": question},
        retrievalConfiguration={
            "vectorSearchConfiguration": _vector_search_config(k, audience)
        },
    )
    return [
        {
            "text": r.get("content", {}).get("text", ""),
            "score": r.get("score"),
            "source": r.get("location", {}).get("s3Location", {}).get("uri"),
            "audience": r.get("metadata", {}).get("audience"),
        }
        for r in response.get("retrievalResults", [])
    ]

def answer(question, k=3, audience=None):
    response = _agent.retrieve_and_generate(
        input={"text": question},
        retrieveAndGenerateConfiguration={
            "type": "KNOWLEDGE_BASE",
            "knowledgeBaseConfiguration": {
                "knowledgeBaseId": KNOWLEDGE_BASE_ID,
                "modelArn": GEN_MODEL_ARN,
                "retrievalConfiguration": {
                    "vectorSearchConfiguration": _vector_search_config(k, audience)
                },
            },
        },
    )
    citations = []
    for citation in response.get("citations", []):
        for reference in citation.get("retrievedReferences", []):
            uri = reference.get("location", {}).get("s3Location", {}).get("uri")
            if uri and uri not in citations:
                citations.append(uri)
    return response["output"]["text"], citations

What it’s actually doing

The managed service runs the same steps you already built by hand. What it adds is incremental re-indexing, a store that outlives the process, and a set of seams worth knowing.

Chunking is a property of the data source. Not of the Knowledge Base, and not of the query. ChunkingStrategy: FIXED_SIZE with MaxTokens and OverlapPercentage sits under VectorIngestionConfiguration on AWS::Bedrock::DataSource, and every field in it is marked update-requires-replacement, so it cannot be edited in place. A different chunk size means replacing the data source and re-ingesting everything, which is why the choice is worth making deliberately the first time. The overlap is there so a sentence split across a boundary still appears whole in one of the two chunks.

Nothing happens until you sync. Creating the Knowledge Base and pointing it at a bucket indexes precisely zero documents. StartIngestionJob is what crawls the bucket, and it is an operation rather than a resource, so CloudFormation cannot do it for you. That is also the mechanism for keeping answers current: later jobs are incremental, re-ingesting only the documents added, modified or deleted since the last sync and skipping the rest, so a re-sync after one edited document is billed for one document’s worth of embedding.

Retrieve and RetrieveAndGenerate do different jobs. Retrieve returns chunks, scores, source URIs and metadata, with no model call and no generation charge. Use it when you are debugging why an answer is wrong, when you want to rerank the results yourself, or when the retrieved text is going into a prompt you control. RetrieveAndGenerate runs the search, generates prose over the results, and returns citations linking spans of the generated text to the chunks behind them. Returning both, as this lab does, separates “retrieval found the wrong chunks” from “the chunks were right and the generated text drifted off them”.

The filter is doing access control. The audience key exists because a sidecar file set it, and filtering at retrieval time means the internal runbook chunks are never eligible to come back. Instructing a model to ignore text already in its prompt is an instruction it can fail to follow; leaving the text out of the prompt removes the failure. The same mechanic is how one Knowledge Base serves several tenants without leaking between them.

Two IAM roles are doing separate jobs. The Knowledge Base service role is assumed by Bedrock to read your bucket, call the embedding model, and write vectors into the index. The Lambda role calls bedrock:Retrieve on one Knowledge Base ARN and bedrock:RetrieveAndGenerate, which AWS documents unscoped. Confusing the two produces access-denied errors at completely different moments: one at ingestion, one at query.

What’s worth remembering

1. A Knowledge Base runs chunk, embed, index and search for you; what you give up is control over how it chunks and ranks, and you still have to trigger every sync.
2. Chunking config lives on the data source under VectorIngestionConfiguration, and changing it replaces the data source and re-ingests everything.
3. Ingestion is an operation, not a resource: StartIngestionJob is what indexes anything, and re-running it is how a changed document reaches the index.
4. Syncs are incremental, re-ingesting only what was added, modified or deleted since the last one, which is what keeps a daily-changing corpus affordable.
5. Retrieve returns chunks and scores with no model call; RetrieveAndGenerate returns generated text with citations. Use the first to see, the second to answer, and both when you need to tell a retrieval fault from a generation fault.
6. numberOfResults is top-k: raising it retrieves more context and sends more tokens to the model on every query.
7. Metadata filters come from <filename>.metadata.json sidecars beside each document, and filtering at retrieval time beats instructing the model to ignore what is already in its prompt.
8. S3 Vectors keeps the vector store on the same bill as the documents and deletes with the stack. It does not support hybrid search, which needs OpenSearch Serverless, Aurora or MongoDB, and its queries run sub-second rather than at the lowest latencies available.