Finding the Documents That Never Reached the Knowledge Base

The situation

A question-answering assistant runs on an Amazon Bedrock Knowledge Base that ingests from four S3 buckets: a policy bucket, a product bucket, a bucket of scanned supplier agreements, and one the operations team drops ad-hoc spreadsheets into. Roughly forty thousand documents, re-synced nightly.

Support has started reporting a specific shape of failure. The assistant answers confidently about policies and products, and returns “I don’t have information about that” for questions whose answer is demonstrably in one of the supplier agreements. Not a wrong answer. No answer at all, as though the document does not exist.

The team already has monitoring. Bedrock’s CloudWatch metrics are on a dashboard, model invocation logging is enabled and writing prompts and completions to S3, and CloudTrail is recording API activity across the account. None of it says anything about the missing agreements. The nightly sync reports COMPLETE. What nobody can currently answer is which documents went in, which did not, and why not.

What actually matters

The first thing to separate is which layer the failure is on. Everything the team has instrumented watches inference: what was asked, what came back, how long it took, what it cost. A document that never reached the index fails hours earlier, in a pipeline that runs on a different schedule and emits a different set of signals. No amount of query-side logging will surface it, because from the retrieval side a document that was never indexed and a document that does not exist are the same thing.

The second is granularity. A sync either succeeded or failed is a useful signal for “did the job run”, and useless for “which of the forty thousand files is missing”. Ingestion is per-document work, so the diagnosis has to be per-document too. A job that scans forty thousand files, indexes 39,880, and fails on 120 will complete, because the job’s health is not the same as its output being right. A summary count gets you as far as knowing 120 went wrong, and stops before telling you which 120 or what happened to them.

The third is that failure is not binary at document level either. A file can be ignored before processing starts, embedded and then fail to index, index some chunks and not others, or succeed at content and fail at metadata. Those have different causes and different fixes: an unsupported format, a size limit, a vector store that rejected a write, malformed metadata JSON. A signal that reports only “failed” collapses four different problems into one, and the team ends up re-syncing and hoping.

The fourth is whether the record is a point-in-time state or a history. Asking “what is the status of this document right now” answers a question about today. Asking “what happened during Tuesday’s sync, and did it start then” needs events retained over time, which means the record has to be delivered somewhere durable rather than read back from the service on demand.

Underneath all of it, ingestion observability on Bedrock is off until somebody switches it on, and it is a different switch from the one the team has already flipped. Invocation logging and knowledge base logging are separate features with separate configuration, and having one does not give you the other.

What we’ll filter on

1. Which layer does it observe: ingestion, or inference?
2. What granularity: the job, or the individual document?
3. Does it distinguish the failure modes, or report a single “failed”?
4. Point-in-time state, or retained history you can query later?
5. Does it carry a reason, or only a status?
6. How much has to be built versus configured?

The landscape

The console sync history. Each data source has a Sync history panel listing every ingestion job with its outcome, and selecting a job offers View warnings for the reasons a sync event failed. It is the fastest look at a single recent job and it needs nothing enabled in advance. It is also a console view rather than something you can alarm on or query across jobs.

ListIngestionJobs and GetIngestionJob. The API form of the same thing. ListIngestionJobs gives the sync history for a data source, filterable by status and sortable by start time; GetIngestionJob returns one job with a statistics object containing numberOfDocumentsScanned, numberOfNewDocumentsIndexed, numberOfModifiedDocumentsIndexed, numberOfDocumentsDeleted, numberOfDocumentsFailed, numberOfDocumentsSkipped, numberOfMetadataDocumentsScanned, and numberOfMetadataDocumentsModified, alongside a failureReasons list for the job as a whole. This is where the count of 120 comes from. It does not name them.

ListKnowledgeBaseDocuments and GetKnowledgeBaseDocuments. Per-document detail, at last. Both return documentDetails entries carrying identifier, status, statusReason, and updatedAt, so you can ask what state a specific file is in or enumerate the documents in a data source. The limit is that it reports current state rather than history: it tells you a document is not indexed today, and not that it dropped out three weeks ago when somebody changed the bucket prefix.

Knowledge base logging. The purpose-built feature, and off by default. Bedrock supports one log type for knowledge bases, APPLICATION_LOGS, which tracks the status of each file during a data ingestion job. It uses the vended log delivery mechanism rather than a setting on the knowledge base itself: PutDeliverySource with the knowledge base ARN as resourceArn and logType set to APPLICATION_LOGS, PutDeliveryDestination pointing at CloudWatch Logs, Amazon S3, or Amazon Data Firehose, and CreateDelivery to join them. The console equivalent is editing the knowledge base to add a log delivery option and confirming the status reads Delivery active. The account needs bedrock:AllowVendedLogDeliveryForResource, and there are CloudFormation resources for all three pieces.

Two event types come out of it. StartIngestionJob.StatusChanged is the job-level event, carrying ingestion_job_status and a resource_statistics block. StartIngestionJob.ResourceStatusChanged is the per-document event, carrying document_location (with the S3 URI), a status, a status_reasons array, and a chunk_statistics block counting created, ignored, deleted, metadata_updated, failed_to_create, failed_to_delete, and failed_to_update_metadata. Every event has a level of INFO, WARN, or ERROR.

The status values are what make the distinction between failure modes legible. A document moves through SCHEDULED_FOR_INGESTION, EMBEDDING_STARTED, EMBEDDING_COMPLETED, INDEXING_STARTED, INDEXING_COMPLETED, and finishes on INDEXED. It can exit at RESOURCE_IGNORED before any work happens, fail at EMBEDDING_FAILED or INDEXING_FAILED, or finish on PARTIALLY_INDEXED, METADATA_PARTIALLY_INDEXED, or FAILED. Deletions and metadata updates have their own started, completed, and failed triples. In every failure case the reason lands in status_reasons.

CloudTrail. Records that somebody called StartIngestionJob, from which principal, at what time. Useful for “who kicked off an unscheduled sync” and worthless for “what happened to this PDF”, because the processing of individual documents is not an API call and never appears in a trail.

Model invocation logging. The feature the team already has, capturing full prompts and completions to S3 or CloudWatch Logs. It is the right tool for watching a production Bedrock app and it sits entirely on the inference side of the pipeline. A document that was never indexed produces no invocation to log.

CloudWatch metrics from Bedrock. The AWS/Bedrock namespace publishes invocation counts, latency, token counts, and throttles. All of it is runtime. There is no emitted metric for documents ingested or documents failed, so any alarm on ingestion health has to be built from the logs.

A polling job you write. A Lambda on a schedule calling GetIngestionJob and ListKnowledgeBaseDocuments, diffing against a previous run and raising alerts. It works, it can be shaped to whatever the team wants, and it is a service to own, deploy, and debug forever in exchange for something the platform delivers as configuration.

Evaluation

Side by side

Signal	Layer	Per document	Distinguishes failure modes	Retained history	Carries a reason	Build or configure
Console sync history	Ingestion	✗	Partly (warnings)	✓ (recent jobs)	Partly	Nothing to enable
GetIngestionJob statistics	Ingestion	✗	✗ (counts only)	✓ (job list)	Job-level only	Nothing to enable
ListKnowledgeBaseDocuments	Ingestion	✓	Partly	✗ (current state)	✓ statusReason	Nothing to enable
Knowledge base logging	Ingestion	✓	✓	✓	✓ status_reasons	Configure delivery
CloudTrail	Control plane	✗	✗	✓	✗	Usually already on
Model invocation logging	Inference	✗	✗	✓	✗	Configure delivery
AWS/Bedrock metrics	Inference	✗	✗	✓	✗	Emitted free
Custom polling job	Ingestion	✓	Partly	✓ (if you store it)	✓	Build and own

Reading it for the missing agreements: only two rows are per-document, and only one of those retains history. The three signals the team already has are all in the wrong column. Knowledge base logging is the row that answers the question the team is actually asking, and the reason it is not answering it today is that nobody turned it on.

The solution

Enable knowledge base logging with a CloudWatch Logs delivery, then query the resource-level events for the documents that never reached INDEXED. This is the feature built for the question, and the alternatives either report at the wrong granularity or watch the wrong layer.

Set up the delivery first. Get the knowledge base ARN from GetKnowledgeBase, call PutDeliverySource with that ARN as resourceArn and logType set to APPLICATION_LOGS, call PutDeliveryDestination pointing at a CloudWatch Logs group, and join the two with CreateDelivery. Confirm the console shows Delivery active rather than assuming the calls took. CloudWatch Logs is the right destination for this team because the diagnosis is interactive and Logs Insights can query it directly; S3 or Firehose suit a longer retention or downstream-analytics story, and the delivery mechanism supports all three.

Then run the sync again, because logging is not retrospective. The events describe jobs that run after the delivery is active, so the nightly sync has to come round once (or be triggered manually) before there is anything to read.

The query that finds the missing documents is a filter on the resource-level status. Start with filter event.status = "RESOURCE_IGNORED" for the files that were never processed at all, then filter event.status = "EMBEDDING_FAILED" and filter event.status = "INDEXING_FAILED" for the two ways processing can break, and read event.status_reasons on each hit for the cause. The broad net is filter level = "ERROR" or level = "WARN", which catches everything the job flagged in one pass. To follow one specific file across its whole lifecycle, filter on its URI with filter event.document_location.s3_location.uri = "s3://bucket/key" and read the status sequence in order.

Once the diagnosis is done, keep the logging on and turn it into an alarm. A metric filter on the log group counting ERROR-level events, with a CloudWatch alarm on the count exceeding zero, means the next batch of documents that quietly fails to index pages somebody instead of waiting for a support ticket. This is the piece that changes a sync from something reported as COMPLETE into something you can trust.

Why not the polling job. It arrives at roughly the same information by calling GetIngestionJob and ListKnowledgeBaseDocuments on a schedule, and it costs a Lambda, a state store to diff against, an alerting path, and the maintenance of all three. The configured delivery produces richer events (the full status ladder, chunk-level counts, reasons) with no code.

Why not lean on ListKnowledgeBaseDocuments alone. It is genuinely useful for confirming the state of a document you already suspect, and it is a point-in-time answer. It will tell you the agreement is not indexed. It will not tell you that it stopped being indexed on the night the operations team changed a prefix, which is the fact that leads to the fix.

Why not CloudTrail or invocation logging. They are the two signals most likely to be reached for, because they are the two most likely to be already switched on. Neither observes document processing: CloudTrail sees the API call that started the job, invocation logging sees queries arriving hours later. A document that failed to index is invisible to both.

Worked example

The team enables the delivery and triggers a manual sync. The job-level event arrives as expected, ingestion_job_status of COMPLETE, with resource_statistics showing 40,112 resources ingested and 118 failed. The same number the console has been showing all along, now with the resource-level events sitting underneath it.

The first query is the broad one, filter level = "ERROR" or level = "WARN", and it returns 118 hits that split cleanly into two groups.

Ninety-four are RESOURCE_IGNORED, and their document_location.s3_location.uri values are all in the supplier-agreements bucket. Reading status_reasons gives the cause: the files are scanned PDFs with no extractable text layer, and the default parser found nothing to pass downstream. They were not failures in any sense the job noticed. Each one was scanned, found to contain no text, and skipped, which is why the count of documents scanned looked healthy. The fix is a parser change on that data source rather than anything to do with the index, and it points straight at foundation-model parsing for the documents whose meaning lives in layout.

The remaining twenty-four are EMBEDDING_FAILED, all in the ad-hoc spreadsheet bucket, and their status_reasons name the same problem each time: the file exceeds the size limit for a single document. These need splitting before ingestion, which is a change to whatever drops them in the bucket.

Two different causes, in two different buckets, both reported by the job as one number. The chunk_statistics on the successful documents confirms the rest of the corpus is intact, with created counts in line with the document sizes and failed_to_create at zero throughout.

The team leaves the delivery in place, adds a metric filter counting ERROR events with an alarm at anything above zero, and adds a second alarm on RESOURCE_IGNORED appearing at all, since on this corpus an ignored document now means something has changed about the source rather than something being wrong with the pipeline.

What’s worth remembering

1. Knowledge base logging is a separate feature from model invocation logging, off by default, and it is the only signal that reports the status of individual files during ingestion; having invocation logging enabled gives you nothing on the ingestion side.
2. Enable it through vended log delivery rather than a knowledge base setting: PutDeliverySource with the knowledge base ARN and logType of APPLICATION_LOGS, PutDeliveryDestination for CloudWatch Logs, S3, or Firehose, then CreateDelivery.
3. The two event types answer different questions: StartIngestionJob.StatusChanged gives job status and resource_statistics counts, while StartIngestionJob.ResourceStatusChanged gives the file’s URI, its status, the status_reasons behind a failure, and chunk_statistics.
4. A document can exit at RESOURCE_IGNORED, EMBEDDING_FAILED, INDEXING_FAILED, or PARTIALLY_INDEXED, and those have different causes; a signal that reports only a failure count collapses them.
5. GetIngestionJob statistics count documents scanned, indexed, deleted, skipped, and failed, but never name them, and ListKnowledgeBaseDocuments names them with a statusReason but reports current state rather than history.
6. CloudTrail records who called StartIngestionJob and nothing about what happened to each document, because per-document processing is not an API call.

The assistant’s silence about the supplier agreements was never a retrieval problem. Ninety-four documents had been scanned and skipped every night for months, and the only signal that would have said so was the one nobody had switched on.