Verifying the Exam Room tracks

Every Exam Room track was drafted by agents and published without anyone checking what it claimed. On ANS-C01, checking found about seven wrong facts per post, and every post in both sample batches had at least one. Wrong quotas, inverted rules, a copy-pasteable IAM policy that did not do what the post said. So a track that has not been through this sweep should be assumed wrong, not assumed right.

This file is for picking the work up cold. Nothing here depends on the conversation it started in.

Where things stand

Run it, do not guess:

python3 scripts/exam-verification-status.py            # per-track table
python3 scripts/exam-verification-status.py --todo AIP-C01   # files still to do

State lives in _data/exam_verified.yml, keyed by track: completed for a track that is fully through, verified_slugs for one partway. Do not stamp a verified: field into post frontmatter. An earlier run did, and the extra line pushed nine pop quizzes and flash cards past the 450-word frontmatter ceiling measure-post.py enforces. Publication is not evidence of verification: AIF-C01 and AIP-C01 both finished publishing without ever being checked, which is why the status script counts live and unverified posts separately and flags that combination.

ANS-C01 is done: 113 posts verified, 799 facts corrected, 70 self-contradictions fixed, full coverage against the official exam guide.

AIF-C01, AIP-C01 and CLF-C02 are done too, at zero style warnings and zero rhythm failures. All three have been through the coverage gate, on 13 September 2026:

Track Objectives Covered Gaps Thin Gated Guide version
AIF-C01 69 69 0 0 13 Sep 2026 1.1, 30 Apr 2026
AIP-C01 98 96 0 2 13 Sep 2026 none stated, no revisions page
CLF-C02 135 121 1 13 13 Sep 2026 none stated
AIB-C01 58 56 0 2 17 Sep 2026 none stated, no revisions page
SAA-C03 189 187 0 2 20 Sep 2026 none stated, no revisions page

Weightings were confirmed from the domain pages in every case. Only AIF-C01 publishes a revisions page, at .../ai-practitioner-01/aif-01-revisions.html; AIP-C01, CLF-C02, AIB-C01 and SAA-C03 have no version, date or document history, so there is no cheap way to notice those guides moving and the gate has to be re-run periodically rather than on a revision signal.

AIB-C01 was gated before publication rather than after, which is the order to prefer: a gap means writing posts, and slotting one into a track with nothing live is a scheduling decision rather than a repair. It came back 56/58 with no hard gaps. Both thin objectives are the same shape, and the shape is worth recognising: an objective carried by a four-minute pop quiz plus paragraphs borrowed from posts written for something else, where every neighbouring objective has a 36-43 minute scenario of its own. Skill 2.3.2 (identifying business-model transformation opportunities) teaches the discriminating half well and the generative half not at all; Skill 4.3.5 (workforce development approaches) explains what each instrument produces but never sizes, sequences or measures a programme. Both were closed the same day with one scenario post each, so the track now teaches 58 of 58; the row above records what the gate found, not where the track ended up.

SAA-C03 was planned from the guide before it was written, and the gate shows what that is worth. 187 of 189 objectives taught on the first pass, no hard gaps, two thin, against AIB’s 56 of 58 with two thin on a track a third of the size. Domains 1 and 2 came back 32/32 and 43/43 with nothing thin at all. The difference is the order of work: scripts/workflows/plan-exam-track.js reads the guide domain by domain and hands the writers an objective inventory, so a bullet has to be deliberately dropped rather than quietly missed.

Both thin objectives landed in the same blind spot, which is the one the plan could not see: database choice. Skill 3.3.11 and its Domain 4 twin ask for MySQL compared with PostgreSQL, and every mention of the two in the track is either a migration story or a scattered behavioural fact, so a reader choosing an engine for a new application finds nothing. Skill 4.3.13 asks for cost-effective database types by shape, and the track teaches the columnar half at length and the time-series half not at all, naming Amazon Timestream only to say LiveAnalytics is closed to new customers and never naming Timestream for InfluxDB, which is the option a candidate can still pick. The lesson for the next plan: an objective naming two things (“time series format, columnar format”, “MySQL compared with PostgreSQL”) gets read as one, and the post that answers it teaches whichever half its scenario happened to need.

AIB-C01’s guide stem is ai-business-strategist-01, and the guide carries aib-01-in-scope-services.html, aib-01-out-of-scope-services.html and aib-01-technologies-concepts.html alongside the four domain pages. The gate reads the domain pages only, so the out-of-scope list is worth a separate cheap check: this is the one certification where teaching implementation detail is a defect rather than generosity.

AIB-C01 is done, on 17 September 2026: 57 posts in four batches, 856 documentation lookups, 1,495 claims checked, 346 wrong facts corrected, 6.1 per post, plus 179 arithmetic defects, 73 passages carrying implementation detail the guide puts out of scope, 45 service names that had moved on and 902 style fixes. 8.2M subagent tokens. The whole track finishes at zero style warnings and zero rhythm failures.

Four things that run’s findings are worth carrying to the next track.

Arithmetic is a defect class in its own right, and a read-through misses all of it. 179 across 57 posts, against 346 facts: a scenario states figures, a table repeats them, a worked example computes from them and a takeaway summarises them, and they drift. A population reading 430 in the prose and 508 in the costing table. A revenue ranking stated backwards in the prose, the table and the SVG at once. Give it its own numbered step in the brief and say to follow every number through the post, rather than hoping a careful reader notices.

The dominant staleness is not what the AIP track taught us to expect. On a generative-AI paper it is model IDs and token prices. Here it was serving and lifecycle: a customised Bedrock model was said to have to run on Provisioned Throughput, which stopped being true on 16 July 2025; the twelve-month availability promise was quoted as current when it governs only models launched before 7 September 2026; two posts quoted the AWS CAF Governance Perspective whitepaper, which now carries “This whitepaper is for historical reference only”. A stale framework page is worse than a stale price, because nothing about it looks out of date.

Non-AWS facts were the worst errors in the run, and nothing was checking them. four-thousand-letters-already-sent had the EU AI Act wrong in a way that changes the answer: it called 300 wrong credit decisions a widespread infringement and designed against the two-day Article 73 clock, when Article 3(61) needs harm across at least two Member States and the fitting limb, 3(49)(c), carries fifteen days. It also described UK automated decision-making as the Article 22 restriction, which the Data (Use and Access) Act 2025 replaced on 5 February 2026 with Articles 22A to 22D, and cited two FCA rules at the wrong paragraph. Brief for regulations, standards and jurisdictions explicitly; an agent told to check “AWS facts” will read past all of it. The same post’s scenario had been given a UK priority-services register while set in Australia, where the equivalent is the life-support register under the National Energy Retail Rules.

The currency rule cleared itself. The track carried 193 bare $ figures across 15 posts at the start and zero at the end. Naming the rule in the brief was enough; batch 1 fixed every affected post before anyone had counted them. Worth adding to check-exam-room.py so the next track never accumulates them.

CLF-C02’s findings cluster rather than scatter: five of its six Domain 4 thin items plus its one hard gap sit under Task 4.3 (AWS technical resources and Support options), and three more sit under Task 2.2 in a 30%-weighted domain. Do not re-derive these; the run is in the session transcript and the fixes are tracked separately.

The order of work

Set by which exams Craig is sitting, which beats every other consideration: he is the reader, and he is revising from this material.

  1. CLF-C02. 23 posts, publishing now, and he is studying for it. Done.
  2. AIP-C01. 205 posts, in progress. He is studying for this one too, which is why it gets the full treatment despite being a finished run. The calibration measured 12.1 wrong facts per post here, the worst of any track, because generative-AI content goes stale fastest. Ask the status script for the remaining count rather than trusting a number written here.
  3. Then in publication order, since that is the order he will revise in: ANS-C01 (done), AIB-C01 (January), SAP-C03 (March), DOP-C02 (June), SCS-C03 (September 2027). Each wants verifying before its window opens, while it is still preview: true and has no live URL to protect.
  4. AIF-C01 last, if there is budget. 96 posts, finished publishing on 3 September.

An earlier draft of this file put the live GenAI tracks first on the grounds that wrong content was reaching readers, and ranked a finished run as an archive not worth full verification. That was wrong about who the reader is.

PR and merge in batches of roughly 25 rather than one PR per track. A 200-post PR is unreviewable and one bad file blocks the rest.

Running it

Three workflow scripts live in scripts/workflows/. They are copies: the originals sat under a session directory that dies with its session, and rewriting the prompts from scratch loses the accumulated corrections below.

Exam guides live at https://docs.aws.amazon.com/aws-certification/latest/<stem>/<stem>-domainN.html. Read them; weightings and task statements change between versions.

The stem usually carries a numeric suffix, and guessing it wastes a round of 404s: AIF-C01’s is ai-practitioner-01, not ai-practitioner, and AIB-C01’s is ai-business-strategist-01 rather than anything built from “AI Business”. Probe candidates with curl -o /dev/null -w '%{http_code}' before launching agents, or search for the domain page; do not construct the URL from the certification’s name.

What went wrong before, so it does not again

Never set effort: 'low' on the agent options. A run with it set had all twelve agents blocked from every tool call by a permission-handler error and burned 813k tokens producing nothing. The symptom looks like a schema failure in the summary; the agents’ own transcripts say “every tool call was rejected”.

Judge each file on its own when committing. An early commit guard held the whole batch whenever one file failed a check, so a single flash card stalled everything. Skip files modified in the last 45 seconds (their agent is still writing), skip a file whose frontmatter fails yaml.safe_load or whose check-exam-room.py prints a warning, and commit the rest.

A rhythm failure is usually pre-existing. Flash cards overrun their frontmatter budget on main as well: nine of twelve sampled. Only treat measure-post.py failing as this sweep’s problem if the same file passes on origin/main.

Do not hand-edit a generated checklist. make reading-times rebuilds them from _data/exam_checklists/*.yml. Editing the post directly is overwritten.

On a published post, change content only. Never touch date:, preview:, the filename or the permalink. Re-dating a live post drops it off the listings and the feed while its URL still resolves, and _data/linkedin_shares.yml has public links with UTM parameters attached. scripts/check-published.py catches it; run it before every push.

The branch is deleted when its PR merges. That leaves a stale tracking ref, and git push --force-with-lease then refuses with “stale info” against a branch that no longer exists. git fetch --prune origin fixes it. After a merge, restart from main rather than pushing onto merged history.

A failed restart must not be followed by a push. When the rebase aborted on an unstashed file and the branch was pushed anyway, it stayed based on the old main. The PR went un-mergeable, and GitHub silently declines to run pull_request CI on a conflicted PR, so it presented as a five-hour Actions outage rather than an error. The script now exits non-zero and says so; if it does, resolve with git merge origin/main before pushing. The tell from the GitHub side is mergeable_state: "dirty" on the PR with zero check runs.

Restart the branch with scripts/verify-restart-branch.sh <merged-sha>, not git checkout -B. The agents keep running while CI does, so the commit cycle almost always lands commits after the sha the PR merged, and `checkout -B

origin/main` throws those away without a word. The script rebases them onto the new main instead, and stashes in-flight files across the move because git refuses to rebase over a file an agent is writing. ## SAA-C03 SAA-C03 was verified on 20 September 2026: **all 97 scenario, quiz, card, cheat-sheet and lab posts**, in six batches, every one of them coming back clean. The domain checklist is generated and carries nothing to look up. 1,884 documentation lookups against 4,606 claims produced **557 wrong facts corrected, 6.1 per post**, plus 296 arithmetic defects, 112 passages sitting above or below the associate design level, 40 stale service names and 637 style fixes. The two posts written to close the thin objectives were verified adversarially as they were written, and the checklist is generated. The weekly token limit cut the fifth batch off three posts short. They were finished when it reset, and they were worth the wait: all three are Domain 4, where the arithmetic carries the argument, and they came back at 6.7 facts a post against the track's 6.1. Two of the three had a rejected option rejected for the wrong reason, which reads as sound until somebody checks the number: an AWS WAF rate-based rule ruled out on a floor of 10 that argues nothing at 1,200 a window, when the real objections are the 600-second ceiling on the evaluation window and the documented detection lag; and an API Gateway account throttle called "the limit producing the bill" when at 600 requests a second it is not the binding limit at all. ### What this run says about planning first The coverage gate and the verification pass measure different things, and SAA-C03 is the cleanest demonstration of it in the corpus so far. Planning from the guide took the gate from AIB's 56 of 58 to 187 of 189, and did nothing at all for the error rate: 6.1 facts a post against AIB's 6.1. A track can teach exactly the right objectives and still be wrong about the figures inside them, and no amount of planning catches that. The defect that recurred most in this track was **a post contradicting itself**. One told a reader to raise an X-Ray sampling reservoir on a rule scoped to a child service, then explained in the next sentence why such a rule never fires. Another said a fully pinned RDS Proxy workload keeps connection reuse between sessions, which is the opposite of what pinning does. A third put an EventBridge rule for root sign-in in ap-southeast-2, where that event is never recorded. None of these needs external knowledge to spot; they need somebody to read the post as a whole and follow the claim through. That is what the arithmetic step does for numbers, and it is worth asking the same question of behaviour. ## SAP-C03, complete: the error rate is not the same everywhere 18 of 97 posts, on 20 September 2026, all clean at the end. The number that matters is **257 wrong facts, 14.3 a post**, against 6.1 on SAA-C03 and 6.1 on AIB-C01. Also 76 passages at the wrong level across 18 posts, where SAA-C03 had 112 across 97. So the six-to-eight-a-post figure this file has been quoting is not a property of agent-written posts in general. Two things separate this track. It is the oldest unverified one, drafted as SAP-C02 and retagged, so its claims have had the longest to go stale: 38 stale service names in 18 posts. And it is a professional exam, where a post has more room to be wrong, because the answer turns on mechanism and second-order consequence rather than on which service to pick. Budget a professional track at twice an associate one. ### The finished run confirms it **96 of 96 posts, complete on 28 September 2026. 1,036 wrong facts, 13.3 a post**, plus 283 arithmetic defects, 305 passages at the wrong level, 155 stale service names and 1,055 style fixes, from 1,567 documentation lookups against 2,990 claims. Every post clean at the end. 455 items went into `remaining`. The rate held across every batch rather than spiking in one, which is what makes it the track's property and not a sampling artefact: | Batch | Posts | Facts | Per post | |---|---|---|---| | 1 (2027-05-14 to 05-31) | 18 | 213 | 11.8 | | 2 (2027-06-02 to 06-19) | 18 | 237 | 13.2 | | 3 (2027-06-21 to 07-09) | 18 | 303 | **16.8** | | 4 (2027-07-10 to 07-28) | 18 | 226 | 12.6 | | 5 (labs, 2027-07-30 to 08-04) | 6 | 57 | **9.5** | The spread within the track is worth as much as the average. Batch 3 carried the pricing posts, and pricing is where a drafted post is least reliable: a figure that sounds plausible is wrong in a way no amount of internal consistency catches. Batch 5 was the labs, which came in lowest, because a lab's claims are mostly commands and a command either exists or does not. **The pricing findings from batch 3 are the ones to remember.** One post made two CloudFront claims that were both false. It said CloudFront egress is charged at a lower rate than direct egress from EC2 or S3; for Australian viewers it is identical, USD$0.114/GB, confirmed against the pay-as-you-go page and the Price List API (`APS2-DataTransfer-Out-Bytes`). The rate advantage only exists for offshore viewers hitting US or EU edges at USD$0.085/GB. And it said caching reduces the volume as well as the rate, when CloudFront origin fetches from S3, EC2 and ELB are free, so caching reduces a leg that costs nothing. The same batch had VPC peering offered as cheaper than Transit Gateway on the grounds that peering carries no per-GB charge. It carries USD$0.01/GB each way across zones, the same USD$0.02/GB as a Transit Gateway, and only undercuts a hub when traffic stays inside one zone. S3 RTC's SLA threshold was given as 99.99% in two places, where the SLA commits to 99.9% and 99.99% is the target. Plain CRR was given an invented "99% within 15 minutes under normal load", where AWS's own workload-requirements table says 24 to 48 hours with no SLA at all. **A scenario premise that could not exist**, from batch 2, is the other one worth keeping. The AZ-failure post opened on an ALB with "one subnet mapping"; an ALB must be given at least two subnets in different Availability Zones. The post was describing a configuration no reader could reproduce. Rewritten around two enabled zones with targets in only one, which is the defect that is actually reachable. In the same post RDS Multi-AZ failover was "a minute or so" against AWS's published 60 to 120 seconds, and its worked example completed in 51 — below its own floor. ### The kind of thing a first batch found The RDS cost post's central saving was **impossible**. It moved a 16 TB volume to gp3 and then reduced it to 4 TB; RDS cannot deallocate storage at all ("You can't deallocate space", USER_PIOPS.ModifyingExisting). The same post called the gp2-to-gp3 conversion "the free win", and the Sydney offer file has both at USD$0.138 per GiB-month, so the conversion changes nothing on that invoice. Worse, converting a 4 TiB gp2 volume **halves** baseline throughput, 1,000 MiB/s to 500, so holding throughput constant costs more than it saves. A reader following that post would have done three days of work to raise a bill. A Fargate quiz claimed the EC2 launch type is "the only way to get a GPU under ECS in AWS", which ECS Managed Instances has not been true of since it reached general availability on 30 September 2025. That post's option set predated a launch type AWS now lists among four. Both are the same shape and it is worth naming: **a claim that was true when the post was drafted**. Nothing in the post reads as wrong, the reasoning is sound, and the answer is no longer correct. A read-through cannot catch these, and neither can a coverage gate. Only a lookup does. ## Finishing a batch ``` make reading-times # restamps, regenerates checklists python3 scripts/check-exam-checklists.py --strict python3 scripts/check-exam-room.py <the track's posts> python3 scripts/check-preview-flags.py python3 scripts/check-published.py ``` Then commit, push, open a PR, and merge once CI is green. Add the batch's slugs to `verified_slugs` under its track in `_data/exam_verified.yml`, or the next run will do them again. **Commit as the agents finish, not at the end.** A batch of 25 runs about three at a time, so the slowest agent is an hour behind the first and holding the whole batch uncommitted risks losing all of it. `scripts/verify-commit-cycle.sh ""` commits every settled post and says which it held back and why: a file written in the last 45 seconds, one whose frontmatter will not parse, or one warning in `check-exam-room.py` where the `origin/main` copy does not. Run it on a timer against the workflow journal (`subagents/workflows//journal.jsonl`, one `"type":"result"` line per finished agent) and push once, at the end: every push cancels the in-progress CI run, so a push per cycle means CI never finishes. ## AIB-C01 splits, in progress: what the verifiers found in OTHER posts The sweep of the 29 AIB-C01 split posts is running. Nine are recorded and all nine came back clean on their own text, but seven of them reported a defect in a post they were told not to touch. That is the most valuable output of a sweep over posts that share a scenario, and it is only in the run's journal, so it is written down here before the journal rotates. Each is a claim by a verifier that has not been independently confirmed except where noted. They belong to the parent trim and the parent re-sweep. **Llama 4 Scout's context window on Bedrock is 3.5 million tokens, not 10 million.** `the-handbook-the-assistant-never-read` carries "Llama 4 Scout's model card lists a ten-million-token context window, so there the limit is the meter and not the window", and the figure AWS publishes for Bedrock is 3.5 million. This is the item the writing agents flagged for the sweep: a context-window claim taken off a model card rather than from what Bedrock supports. Its sibling has been corrected and the two now disagree. **Provisioned Throughput Model Unit pricing is published for some families.** `forty-users-then-four-thousand-what-breaks-that-is-not-the-bill` says "AWS publishes neither the throughput a model unit delivers nor its price, directing customers to their account manager". The throughput half is right; the price half is wrong for Nova and Titan, which carry published hourly Model Unit rates in the Price List API. `eleven-years-of-declinatures-and-1-2m-to-spend` has a defensible softer version ("for several model families AWS directs you to your account team for that hourly rate"). Worth knowing because the stronger sentence was quoted in PR #1837 as an example of a post correctly declining to invent a figure, and half of it is an error in the other direction. **A fine-tuning condition on serving a custom model on demand may not exist.** `eleven-years-of-declinatures-and-1-2m-to-spend` claims parameter-efficient versus full-rank fine-tuning decides whether a customised model can be served on demand. AWS's on-demand prerequisites list Region, the 16 July 2025 customisation date, model-access permission and KMS permission, and no such condition. Needs a verdict rather than a silent deletion. **Two Model Monitor dates appear on no AWS page.** `nine-systems-live-and-one-person-watching` states that Model Monitor and Clarify "moved to maintenance on 30 June 2026 and closed to new customers on 30 July 2026". Confirmed directly against `sagemaker/latest/dg/model-monitor.html`: the notice is real and reads "no longer open to new customers. Existing customers can continue to use the service as normal", and it is **undated**. Two invented dates in a track recorded as verified. All 14 posts mentioning Model Monitor do note the closure, so the service's status is handled correctly everywhere; it is only these dates that are wrong. **An adverse-selection claim is inverted, including inside an SVG.** `eleven-years-of-tagging-and-one-customer` reads "the figure is attractive precisely where students take 2.3 titles rather than 4.1" and, in an SVG aria-label, "because the early adopters are the light users". Its own figures refute both (AUD$40.02 a unit, AUD$96 flat, break-even 2.4 titles), so the early adopters are the heavy users. The aria-label matters: a correction that fixes the prose and leaves the accessible text is still wrong for anyone reading it that way. **Two parents resolve a decision their split also resolves.** `three-pilots-to-kill-and-one-to-fund` ends its worked example with three terminate, two pause, four scale, while its split minutes the same review as "continue, reduced scope" and terminates afterwards. Both cannot be true of one review, and every shared figure agrees, so this is the parent still carrying the split's decision. `the-tender-they-lost-on-a-feature-they-dont-have` states an availability baseline as an unqualified deliverable where its split establishes the baseline was never produced. That last pair is the duplication `check-exam-prose.py` found independently by word count, arriving at the same 19 parents from the other direction. A verifier reading for contradictions and a script counting words agreeing on the same posts is the strongest signal available that the parent trim is the right next job. ### The word budget undercounts duplication The 19 parents sent to the trim pass were chosen by `check-exam-prose.py`, on the reasoning that a post carrying two decisions must be over its section budget. That is wrong in one direction, and the verifiers found it. 29 parents spawned a split. Only 19 are over budget. A verifier working on `nine-months-of-prompts-and-nobody-set-a-retention-period` reported that its parent, `ten-percent-cheaper-and-out-of-the-country`, "still carries this post's decision in its own solution paragraph and its takeaway 5", and that parent is **under** the budget and so was never sent for trimming. A post that was already short can carry a second decision compactly and pass a word count. So the budget is a lower bound on duplication, not a detector for it. An earlier note in this file called the budget script and the verifiers "two independent methods landing on the same 19 posts", which overstates it: they agree on those 19, and the verifiers find more. The remaining 10 parents need the same trim, and the only reliable way to find them was to read the pair. What the budget IS good for is ordering the work and proving it finished: it cannot say whether the right half was removed, but it can say a post is still too long, and it cannot be argued with. ### Further cross-post findings from the AIB sweep Each is a verifier's claim about a file it was told not to edit. - `twelve-minutes-saved-and-a-cfo-who-wants-dollars` states "seventy-two case handlers across two sites work about 40,000 cases a quarter, 160,000 a year" while also funding an extension to a second shared-services site, which its split says is not yet staffed. One of the two counts a site twice. - `the-commitment-that-would-have-covered-a-quarter-of-the-bill` says Provisioned Throughput capacity is "sized either in model units or in input and output tokens a minute". AWS's prov-throughput page describes model units only. Check before editing: the Reserved tier does sell tokens a minute, so the sentence may be conflating two products rather than inventing a unit. - `four-systems-four-hundred-thousand-files` describes Amazon Quick's access control as coarse-grained at the knowledge-base level while its split establishes document-level ACLs synced from the source. Flagged as worth a look rather than as a contradiction. - `the-ad-the-model-wrote` lists "prompt attacks" among the harm categories a blurb "clears every one of", which reads as though a prompt-attack filter scores output. It scores input. The same looseness was fixed in its split. - The AWS Marketplace mechanics in `the-contract-under-the-marketplace-listing` (25 accounts, the SCMP addenda, the Vendor Insights counts, variable payments) are repeated in `flash-card-aws-marketplace` and `twelve-tools-and-one- signature`, neither of which has been swept since the split was written. ## AIB-C01 splits, complete: making agents cite their sources did not make them right The 29 split posts were drafted with a step the earlier tracks did not have. The brief made it an instruction rather than advice: before asserting any AWS price, limit, quota, retention maximum, default or behaviour, open the page that states it, read the figure off that page, and log the claim with its URL. The agents complied, returning several hundred cited claims. The sweep says it bought nothing. | | first 57 AIB posts, no cite step | the 29 splits, with it | |---|---|---| | claims checked per post | 26.2 | 31.7 (+21%) | | documentation lookups per post | 15.0 | 15.2 | | **wrong facts per post** | **6.1** | **7.2 (+19%)** | | arithmetic defects per post | 3.1 | 4.6 (+46%) | | above or below the exam's level, per post | 1.3 | 2.7 (+107%) | | service names that had moved on, per post | 0.8 | 0.7 | | **wrong facts per claim checked** | **23.1%** | **22.9%** | The last row is the finding. Accuracy per claim did not move. An agent that opens the AWS page, reads it, and writes the URL down beside what it wrote still gets about one claim in four wrong. What the cite step changed was density: the posts make a fifth more AWS claims, so each one carries a fifth more errors. Two consequences worth holding on to. **A cite step does not substitute for a sweep, and should not be budgeted as if it does.** Expect the same six to eight wrong facts a post whether or not the writer was told to cite. The sweep is where accuracy comes from. **On a non-technical track the cite step has its own failure mode.** Level defects more than doubled, and the mechanism is not mysterious: an agent sent to read AWS documentation reads *implementation* documentation, and then writes implementation detail. On AIB-C01 that is a defect rather than thoroughness, and the brief says so at length. Sending a business-track writer to the service guides pulls it toward the one failure the level note spends most of its words warning about. On a professional track the same step would probably be neutral or useful. What the cite step did produce is a target list. Each writing agent's logged claims told the sweep where to look first, and the two claims flagged at drafting time were both real: **Llama 4 Scout's context window, where AWS publishes two different numbers.** The Bedrock model card lists 10M tokens; the AWS News Blog launch post says "Amazon Bedrock currently supports a 3.5 million token context window for Llama 4 Scout". The post now names both and says which is which, rather than picking one. That is the right handling of a documented disagreement. **Provisioned Throughput comes in two forms, not one.** The purchase page states "Amazon Bedrock offers two types of Provisioned Throughput - by Tokens and by Model Units". Several posts presented Model Units as the only form. The same page also carries a procurement fact the posts had all missed: Model Units must be requested through the AWS support centre before a Provisioned Throughput can be bought, so a reservation is not a same-week decision. On a track where money and timing are the units, that absence mattered more than any rate. ### A fabricated date pair that propagated between services `30 June 2026` and `30 July 2026` appear across the corpus attached to different services, which is the signature of a date that was invented once and then reused. `nine-systems-live-and-one-person-watching` had SageMaker Model Monitor and Clarify "moved to maintenance on 30 June 2026 and closed to new customers on 30 July 2026". The parent trim deleted that passage, for an unrelated reason: its verified split establishes the account never used either service, so neither was selectable. The dates went with it. But the pair survives in `proving-where-ai-content-came-from` (AIP-C01, a track recorded as verified), where `30 July 2026` is now Clarify's closure date and `30 June 2026` is Titan Image Generator G1 v2's end of life. Checked directly, both AWS pages that carry the Model Monitor notice give **no date at all**: `sagemaker/latest/dg/model-monitor.html` and its dedicated `model-monitor-availability-change.html` both read "Amazon SageMaker Model Monitor is no longer open to new customers. Existing customers can continue to use the service as normal", undated. So a closure date for Model Monitor or for Clarify is not something AWS publishes on those pages, and a post giving one is supplying a figure rather than reading it. The Titan Image Generator date may well be real: Bedrock publishes model lifecycle dates on its own page, and that is a different kind of claim from a service closing to new customers. Check it there rather than assuming the pair is wrong together. Worth generalising: when the same date appears against two unrelated services, suspect the date rather than the services. A verifier checking one post cannot see the repetition, so this is a corpus-level check that no per-post sweep will catch. ## The parent trim, complete: 101 contradictions across 30 posts All 30 AIB-C01 parents that spawned a split have been trimmed back to one decision. 30 of 30 returned, none failed, all 30 came back inside the word budget, 74,312 words to 61,050. **101 contradictions resolved.** That is the number worth carrying forward. Each one is a place where a post and its split told a reader different things about the same event, and the splits had been through the documentation sweep while the parents had not, so the split won by default. A sample of what that looked like: - A renewal that one post declines and its split accepts, with different three-year sums behind each. Both cannot be true of one board paper. - A shift-handover summariser already proven at AUD$0.31 a shift in one post and still an open vendor pilot in the other. - One review approving a tariff change "in under two minutes" against the same post's own tier arithmetic, and the split's 485 reviewer-hours a week, both built on three minutes. - A go-live in month six against a February change window owned by another team, which puts first production use in March. - The same AUD$5.9m presented as new revenue in one post and as about minus AUD$500,000 against AUD$6.4m replaced in the other. Most were settled by deleting the parent's version rather than rewriting it, so the track states each fact once, in the post that owns the decision. Two things this says about the method. **A split-and-trim is a verification technique, not just an editing one.** Nobody set out to find 101 contradictions. They surfaced because one post was swept and its twin was not, and an agent was asked to read both and reconcile them. Any pair of posts sharing a scenario can be checked this way, and nothing else in this repository finds this class of defect: a per-post sweep reads one file and cannot see that the file next to it minutes the same meeting differently. **The word budget stops at the parents.** 10 posts are still over, all of them splits, and the growth is not padding: the verifiers added published AWS conditions, and eight of the ten fail on `### Evaluation` alone. The first guess was that the budget counts the side-by-side table as prose, and that is wrong. The table averages 48 words; the worst case carries 390 words of prose against 57 of table. Correcting a table row means explaining the correction beside it, and that prose is the post doing its job. So AIB-C01 is **not** in `_data/prose_budget.yml`, and forcing it in would mean asking an agent to cut prose from ten freshly verified posts, where the thing most likely to go is the AWS condition that made the section long. The budget has done what it can here; the remaining decision is editorial and belongs to whoever next reads those ten. ## AIB-C01 complete: 88 of 88, and what four groups say about accuracy The whole track is swept. 124 post-sweeps across four groups, 3,457 claims checked, 778 wrong facts corrected, 487 arithmetic defects, 259 passages above or below the exam's level. Zero style warnings, zero rhythm failures. Four groups, because the track was written and edited in four distinguishable ways, which makes it the only track that can compare them: | group | cite step | words | facts/post | **per claim** | arith/post | level/post | |---|---|---|---|---|---|---| | first 57, swept 17 Sep | no | ~3,300 | 6.1 | **23.1%** | 3.1 | 1.3 | | 29 splits, written to budget | YES | ~2,100 | 7.2 | **22.9%** | 4.6 | 2.7 | | 19 parents, cut then trimmed | no | ~2,200 | 6.3 | **23.1%** | 4.4 | 2.4 | | 19 parents, cut then trimmed | no | ~2,200 | 5.4 | **19.5%** | 4.8 | 3.4 | **The per-claim column is flat.** Four groups written and edited four different ways, and the rate at which an AWS claim is wrong sits between 19.5% and 23.1% in all of them. Pooled, the 95 posts drafted without a cite step come in at 22.4% and the 29 with it at 22.9%. About one AWS claim in four or five is wrong when a post reaches the sweep, and nothing tried so far moves that. That is the number to plan against. It does not depend on whether the writer was told to open the AWS page, and it does not depend on how heavily the post was edited afterwards. The sweep is where accuracy comes from. **Arithmetic and level defects are a different story, and they went up.** Against the original 57, every group written or edited after the word budget came in carries about 50% more arithmetic defects a post (3.1 to 4.4-4.8) and roughly twice the level defects (1.3 to 2.4-3.4). Two candidate explanations, and the evidence does not separate them cleanly. A deep cut can break arithmetic by removing one operand of a sum that survives elsewhere, which is exactly what the sweep brief was told to look for and what it found. And a shorter post states the same figures in less space, so a tighter-packed post gives arithmetic more chances to disagree with itself per page. The splits were freshly written rather than cut and still show 4.6, which argues for density over cutting damage, but they also make the most claims of any group at 31.7 a post. What it means in practice does not depend on resolving that. **Cutting a track to a word budget raises its arithmetic and level defect density, so sweep after the cut, not before.** Doing it in that order on this track is what surfaced 487 arithmetic defects; sweeping first would have certified prose that was about to change under it, which is the mistake this file already records once. ## SCS-C03 complete: 93 of 93, and the highest error rate measured anywhere | | | |---|---| | posts | 93, in six batches | | documentation lookups | 1,748 | | claims checked | 3,751 | | wrong facts corrected | **1,346** (14.5 a post) | | **per claim checked** | **35.9%** | | arithmetic defects | 243 | | passages off the exam's level | 353 | | stale service names | 127 | | style fixes | 1,232 | | claims reported unconfirmed | 541 | | agent failures | 0 of 93 | **35.9% per claim is half again the worst figure this file previously recorded.** AIB-C01's four groups held a 19.5 to 23.1% band however they were drafted, and that flatness was the last run's headline. SCS posts break it in two directions at once: they make more claims each, about 40 a post against AIB's 26 to 32, and a larger share of those claims is wrong. The per-post figure, 14.5, sits just above SAP-C03's 13.3, so by the measure this file has used until now the track looks ordinary. It is not. **Count claims as well as posts, or a denser track reads as a cleaner one.** The spread inside the track is wider than the spread between tracks. The cheat sheets and pop quizzes came in at 11.5 to 13.3 a post; the deep scenario posts and labs hit 17.4 and 18.7. The pattern matches what SAP-C03 showed and is worth stating generally: **a post whose answer turns on mechanism carries more wrong facts than one whose answer is which service to pick**, because there is more surface to be wrong on. ### A documentation fetch returned fabricated content three times This is the finding with consequences beyond this track. Three separate agents received plausible, well-formed content that the AWS page does not contain: - A complete condition-key table for the IAM Service Authorization Reference. The page is JS-rendered and a raw fetch returns a 2.4KB shell, so there was nothing to summarise and the summary was invented. - A CloudTrail pricing page describing a Lake 30-day new-customer trial, which cannot coexist with the 31 May 2026 closure to new customers. - A Config conformance-pack page carrying invented CLI syntax and invented quotas ("50 conformance packs, 50 rules per pack"); the real quotas are 50 packs, 130 rules per pack and 1,000 rules per Region per account. All three agents noticed and discarded what came back, two of them because the content contradicted a fact they already held. That is the discipline working, and it is also the reason to keep it: **the fetch step can manufacture a source, so "I read the page" is not by itself evidence.** It bears directly on the earlier finding that a cite step does not improve per-claim accuracy. A cite step that trusts what it receives can make a track worse, not merely fail to help it. ### Invented figures that read as entirely reasonable Four, none published by AWS, none catchable by reading the post: - "GuardDuty exports a finding to EventBridge within about five minutes." AWS says "in near real-time" and publishes no interval. - "Instance profile credentials rotate on a schedule measured in hours." AWS says only that new credentials are available at least five minutes before the old ones expire. - "Ninety days is where the public web settled." Public TLS validity is on a declining schedule, not settled. - Text transformations inside AWSManagedRulesSQLiRuleSet. AWS restricts access to the details of rules inside a managed rule group, so no one outside AWS can state them. ### Defects that invert a post's answer - **Security Lake does not support S3 Object Lock**, and enabling it interrupts delivery. A post recommended the lake for a tamper-evidence scenario on the strength of inheriting the estate's Object Lock, which it does not: Security Lake creates its own buckets, and editing their S3 Lifecycle rules directly can delete metadata and make the data unreachable. - **Name constraints and pathlen:0 are evaluated by the relying party** during certification path validation, not by the CA at signing. A post explained them as stopping an attacker who holds the issuing key from minting a sub-CA. The attacker signs it; the client is what rejects it. - **`aws:approve` never auto-approves on timeout.** It fails, with a 7-day default and a 30-day maximum. A post advised configuring the timeout "so silence means no", which is what the action already does. - **AWSCompromisedKeyQuarantineV2 is superseded by V3**, which also denies `cloudtrail:LookupEvents`, `s3:GetObject`, `s3:ListBucket` and `iam:ListUsers`. A post said the policy leaves enough intact for the owner to investigate; V3 specifically blocks investigation. - **The IAM Roles Anywhere credential helper cannot reach the Windows local machine store.** It searches the user's MY store, and `--system-store-name` selects only within CURRENT_USER. A post's entire Windows design was built on the machine store and could not have worked. - **An Athena query over WAF logs filtered on `action = 'COUNT'`** and could never return a row: `action` holds only the terminating action, and counted rule-group matches sit in `ruleGroupList[].nonTerminatingMatchingRules`. ### Security Hub is two services, and that was the commonest error AWS moved the plain name **Security Hub** in December 2025 to a new unified service, GA 2 December 2025, which correlates CSPM, GuardDuty, Inspector and Macie signals into OCSF exposure findings with attack-path and blast-radius views. The posture-management service that consumes ASFF findings, runs standards and derives control checks from Config service-linked rules is now **Security Hub CSPM**. Posts used one name for both, including inside SVG labels and aria-labels. Two posts carried the ambiguous name in their title, slug and permalink and were renamed. Feeding the confirmed renames into each later batch's staleness brief, rather than letting every agent rediscover them, is worth doing on any track: by batch five the brief carried eleven confirmed corrections and the agents spent their lookups on new ground. ### What the sweep could not do from inside a post Agents were told not to edit a file that was not theirs, which turned the `remaining` lists into a cross-post defect report. The pattern repeats the split-and-trim finding: **a defect spanning two posts is invisible to a sweep that reads one.** Among them, an access-analyzer post naming policy validation as the third analyzer type (AWS's third is internal access), a WAF capacity claim contradicting two already-verified posts in the same track, and a track-wide split between "organisation trail" and AWS's own "organization trail". Seven posts were renamed and one retitled afterwards, none of them published, so no redirect was needed. The checklist post is hand-maintained, with no `_data/exam_checklists/SCS-C03.yml`, so its link text does not regenerate on a retitle and `check-exam-checklists.py` does not look at link text: nine stale link labels were repaired by hand. **A generated checklist would have caught them; this track should get a plan file.** ### Decided, and worth not relitigating **Audit Manager stays.** It went into maintenance mode on 30 April 2026, but SCS-C03 Skill 6.3.2 names it explicitly and it remains selectable for an existing customer, so `choosing-between-config-conformance-packs-and-audit-manager` gets a clause rather than a retirement. It is a near-duplicate of `automating-compliance-evidence-for-an-audit` in scenario shape; after this pass the two no longer contradict each other, one organisation being a new customer and the other existing, and that distinction is itself exam-relevant. A merge is possible later and is not needed now. **Systems Manager Incident Manager closed to new customers on 7 November 2025**, with AWS pointing issue tracking at OpsCenter and paging at partner products. One SCS post recommended it and was rewritten. Two DOP-C02 posts were framed entirely on it. The scenario post was reframed and re-slugged as `standing-up-an-incident-path-with-no-native-pager`, teaching the path a new customer can actually build: two EventBridge rules splitting the record from the page, OpsCenter holding the OpsItem, Automation running the response, and a third-party pager for acknowledgement, rotation and escalation. Nothing was published, so there is no `redirect_from`. Its sibling pop quiz, which asked Incident Manager or OpsCenter, became `pop-quiz-opscenter-vs-paging-the-on-call`: what belongs in the work queue against what has to page a named human, which is the half of that comparison still standing. ## DOP-C02 complete: the corpus is swept, and the worst defect is not a wrong number | | | |---|---| | posts | 91, in six batches | | documentation lookups | 1,721 | | claims checked | 3,466 | | wrong facts corrected | **1,259** (13.8 a post) | | **per claim checked** | **36.3%** | | arithmetic defects | 254 | | passages off the exam's level | 347 | | stale service names | 124 | | style fixes | 1,082 | | claims reported unconfirmed | 541 | | agent failures | 0 of 91 | With this, **906 of 906 posts are verified** and every Exam Room track has been through a documentation sweep. ### The per-claim rate looks like a property of the level | track | level | facts/post | per claim | |---|---|---|---| | AIB-C01 | business | 6.1 to 7.2 | 19.5% to 23.1% | | SCS-C03 | specialty | 14.5 | 35.9% | | DOP-C02 | professional | 13.8 | 36.3% | Two professional and specialty tracks landing within 0.4 points of each other, against a business track that held a 19.5 to 23.1% band however it was drafted, suggests that band is **the associate and foundational rate** rather than a universal one. Plan a professional track against one claim in three being wrong, not one in four or five. ### Seven posts recommended a capability AWS does not offer This is the finding of both sweeps, and it is not what the sweep was built to catch. Every figure in such a post can be right while the design it teaches cannot be built: - An **X-Ray sampling rule** taking successes at 1% and faults at 100%. Sampling is head-based, decided at the root service before the outcome is known, and matches on service name, service type, host, HTTP method, URL path, resource ARN and attributes. There is no status, fault or error field. - **"Application Insights + Enhanced tier."** There is no basic or enhanced tier; a tier in that service is a component's technology type. The post carrying it came back at 22 wrong facts, the worst single result in the corpus, most of it invented capability rather than wrong numbers. - **`DefaultValue: 0` keeping a metric filter reporting through calm periods.** AWS reports that value only when logs are ingested and none match; with no ingestion there is no data point. The mechanism that does the job, `TreatMissingData: notBreaching`, was set in the lab's own template and absent from its prose. - **An ephemeral AWS account per pipeline run.** Organizations sets a four-day minimum age before removal, three concurrent closures, and a closed account keeps counting until permanently closed. - **One CodeArtifact repository proxying both npmjs and pypi.** "Each CodeArtifact repository can only have one external connection." - **CloudFormation Hooks as unbypassable "regardless of origin"**, with a worked example of a hook blocking a laptop deployment. Hooks run before CloudFormation and Cloud Control API operations only; a direct service API call never reaches one. That is what an SCP is for. - **Snapshot tests catching regressions.** The CDK testing guide says the opposite, because a library upgrade or a changed context value rewrites the synthesised template. **Ask whether AWS documents the capability, or only the pieces it is being assembled from.** That question is now in the verification brief. It is cheap and it is not implied by checking figures. ### The fetch fabricated six times, and once it reached a post Across both tracks a documentation fetch returned plausible, well-formed content the AWS page does not contain: - A complete condition-key table for the IAM Service Authorization Reference, a JS-rendered page whose raw fetch is a 2.4KB shell. - A CloudTrail pricing page describing a Lake new-customer trial that cannot coexist with the 31 May 2026 closure. - A Config conformance-pack page with invented CLI syntax and invented quotas. - A Security Hub controls-config-rules page returning an empty 281-byte shell. - A Security Hub pricing figure of USD$3.75 per resource unit, which appears nowhere in the page: the value is a `{priceOf...}` template placeholder, and 3.75 occurs only in an unrelated partner example. - An AWS DevOps blog post saying CodeCommit "returned to general availability on 24 November 2025", at a URL that returns 404. Five were caught inside the agent that received them, usually because the content contradicted something already held. **The sixth reached a post** and was found only when the claim was checked by hand afterwards. AWS's pricing page still says, verbatim, "AWS CodeCommit is no longer available to new customers." Two rules came out of it. **A citation you cannot re-open is not a source.** And **availability is checked on the page that would carry the closure**: the CodeCommit user guide carries no notice at all and reads as a live service, which is exactly how that mistake was made. An absent notice on one page is not evidence a service reopened. ### Three posts reframed rather than retired All three taught a decision nobody can make any more, and none of this track is published, so each was a clean rename with no `redirect_from`. `running-incident-response-with-systems-manager-incident-manager` became `standing-up-an-incident-path-with-no-native-pager`. The verifier's report is why a clause was not enough: the facts were right and the closure was stated throughout, "but the post still teaches a decision a new DOP-C02 candidate cannot make". Incident response is Domain 5 and only the service is dead. `pop-quiz-incident-manager-vs-opscenter` became `pop-quiz-opscenter-vs-paging-the-on-call`, which is the half of that comparison still standing. `aws-config-vs-audit-manager` became `config-conformance-packs-vs-security-hub-cspm-standards`. Audit Manager is in maintenance mode, and decisively **DOP-C02's exam guide does not name it anywhere**: not in Domain 6's task statements, not in the in-scope services list. That is the opposite of SCS-C03, where Skill 6.3.2 names it and the equivalent post survived with a clause. The same service, two tracks, two different answers, and the exam guide is what separates them. One caution for the next reframe: the rewritten pop quiz came back at 574 words of frontmatter against a 450 budget and 43% of sentences over 30 words. A brief that asks for a rebuilt argument gets an essay unless it also names the format's budget. `measure-post.py` caught it. All three keep their `exam_verified.yml` entries while carrying mostly new prose. Every claim was sourced from AWS during the reframe, but that is not the same as having been through the sweep, and re-queueing the three is cheap. ## A verified post is not a correct post: the residual, measured The corpus reached 906 of 906 verified on 2 October 2026, and nobody had ever swept a track twice, so what that number was worth was unknown. This measures it. A stratified sample of 28 already-verified posts, four from each of the seven tracks swept in September 2026, went through a second independent pass against AWS's own documentation. | | | |---|---| | posts re-swept | 28 | | came back clean | **4 (14%)** | | AWS claims re-checked | 988 | | documentation lookups | 537 | | wrong facts found | **72, or 2.57 a post** | | **wrong facts per claim** | **7.3%** | | mechanism defects | 16 | | unpublished figures | 10 | | availability defects | 4 | | arithmetic | 9 | | bad documentation fetches | 66 | Per post, worst first. The ordering is the exam's level, as the first sweep's was: SAP-C03 at professional level is worst, the two foundational tracks are mildest. | Track | Posts | Clean | Claims | Wrong facts | Per post | |---|---|---|---|---|---| | SAP-C03 | 4 | 0 | 174 | 16 | 4.00 | | AIP-C01 | 4 | 1 | 136 | 15 | 3.75 | | AIF-C01 | 4 | 1 | 111 | 11 | 2.75 | | SAA-C03 | 4 | 1 | 185 | 9 | 2.25 | | AIB-C01 | 4 | 0 | 124 | 9 | 2.25 | | CLF-C02 | 4 | 1 | 126 | 7 | 1.75 | | ANS-C01 | 4 | 0 | 132 | 5 | 1.25 | The first sweep took these tracks from roughly one AWS claim in four wrong to 7.3% wrong. That is a real improvement and it is not "correct". Nothing in the sample was checked against the four evidence rules that went into the brief in October, each earned by a defect a September sweep missed, and those rules are where most of this came from: 16 recommended capabilities AWS does not document, 10 plausible figures AWS publishes nowhere. The mechanism defects are the ones worth reading, because every figure in them was right. An idempotency key on an EventBridge `PutEvents` call, credited with stopping a client retry creating a second order, where the Classic custom bus has neither an idempotency token nor deduplication. A Fargate task-count service quota to watch in the DR Region, where what caps a Fargate service is the On-Demand vCPU quota. A declarative policy setting `internet_gateway_block.mode` to `block_bidirectional` and leaving `exclusions_allowed` at `enabled`, so the guardrail the post calls unbypassable is bypassable as written. "Zonal target groups" as the name of a control, where the control is a cross-zone load balancing setting. A 3,000-page document through Bedrock Data Automation's splitter, where AWS caps an individual document at 20 pages and 3,000 is the account's page quota. ### The fetch is the weakest link in the chain 66 bad documentation fetches across 537 lookups, so one lookup in eight came back unusable, and the failure has one dominant shape: a `docs.aws.amazon.com` page renders client-side or has been renamed and 302s to its guide index, so a rendered fetch returns the service name and nothing else. Mostly the summariser says so. Twice it did not. An EventBridge page returned title-only and the fetch answered from its own memory, stating "Maximum retry attempts: 2, Maximum event age: 3600 seconds" as AWS documentation, where the published figures are 185 attempts over 24 hours. A Bedrock vector-store page that does not exist came back with a supported-store list that is wrong in both directions against the `StorageConfiguration` enum. The worst single instance is a loop. A search for a Snowball Edge end-of-support date returned "On December 31, 2026, AWS will discontinue support for AWS Snowball devices in all AWS commercial Regions", with invented supporting detail, citing a page on which "2026" does not appear. That is the sentence the post under review was being corrected for. In the agent's own words: *this is exactly the post's wrong claim coming back as a source.* `scripts/aws-doc.py` is the answer to the dominant shape. Every page under `docs.aws.amazon.com` is served as markdown at the same path with `.md`, which returns text rather than a render and 404s honestly when a page has moved, and `--find` resolves a renamed page out of the guide's own `toc-contents.json`. Both verification briefs now say to use it. ### A correction can be wrong in its own way Spot-checking two of the sample's corrections against AWS afterwards: the Step Functions quotas hold exactly as corrected (StartExecution 800 bucket and 150 a second outside us-east-1, us-west-2 and eu-west-1; StartExpressExecution 6,000 and 6,000 everywhere). The Snowball closure date did not. The corrected post read "closed to new customers on 7 November 2025", which is the Incident Manager row immediately below Snowball's on the Services in Maintenance list. Snowball Edge's own announcement date there is 7 October 2025, and the availability-change guide publishes no date at all. Right table, adjacent row. A sweep cannot catch that about itself, which is the argument for the spot-check rather than for a third sweep. ### What it costs 2.63M subagent tokens for 16 posts, so about 165,000 a post. Re-sweeping all 722 September posts is roughly 119M subagent tokens. Work the tracks in defect order rather than all at once. ## SAP-C03 re-swept to the October standard: 96 of 96, and the sample was right SAP-C03 measured worst in the residual sample at 4.00 wrong facts a post, and it is a live exam, so it went first. The 92 posts not already in the sample were re-swept in two parallel runs. With the sample's four, the whole track is now at the 2026-10 standard. | | Batch A | Batch B | Total | |---|---|---|---| | posts | 46 | 46 | **92** | | claims re-checked | 2,015 | 2,069 | **4,084** | | documentation lookups | 1,450 | 1,537 | **2,987** | | wrong facts | 185 | 178 | **363** | | per post | 4.02 | 3.87 | **3.95** | | came back clean | 1 | 3 | **4 (4.3%)** | | mechanism defects | 56 | 57 | **113** | | unpublished figures | 37 | 38 | **75** | | availability defects | 3 | 10 | **13** | | arithmetic | 38 | 37 | **75** | **The four-post sample predicted 4.00 and the track came in at 3.95.** A stratified sample of four posts is enough to size a track's residual, which makes the remaining six tracks a costing exercise rather than a guess. 16.8M subagent tokens for 92 posts, about 182,000 a post. **Mechanism defects outnumber everything else, 113 of them.** Every figure in those posts could have been right while the design stayed unbuildable, and a read-through does not catch them because nothing on the page contradicts anything else on the page. A permissions boundary built only from `Deny` statements. A child-policy lock built from `@@assign` alone, where the child control operator is the mechanism Organizations actually documents. One hand-locked backup vault serving copy actions in two Regions, when vaults and vault locks are regional, so the second copy job had no vault behind it. A CloudFormation template holding three named IAM roles and deployed into two Regions of every account, which collides on the role names because IAM is global. Aurora cross-Region read replicas offered for PostgreSQL, where AWS documents the managed replica for MySQL only. Backup Audit Manager delivering a report to another account's bucket, which it cannot do. Two were the post getting a service's own documentation backwards: Istio ambient mode called single-cluster only, where multi-network is the only supported ambient multicluster topology, and `VirtualService` recommended as the waypoint configuration mechanism on a page that says not to use it that way. **13 availability defects**, in a track swept eleven days earlier: Amazon FSx File Gateway offered as a live Storage Gateway option where its own user guide opens by saying it is closed, and ARC readiness checks recommended in three places including a quota line. ### The fetch tool earned its place, and gained two fixes `scripts/aws-doc.py` went into this run new. Across 2,987 lookups the agents report it refusing shells and redirects rather than letting a summariser answer from memory, repeatedly on the exact URLs a sweep would otherwise have trusted: `list_amazondynamodb.html` and `list_amazonrds.html` both 302 to the Service Authorization Reference index and extract to 31 characters, where the real pages are `list_dynamodb.html` and `list_rds.html`; a `capacity-reservation-billing.html` that no longer exists, on which the zonal-RI-versus-ODCR distinction rested. One agent put it plainly: a rendered fetch of that URL *would have looked like a page and invited an answer from memory*. Two usability defects came out of being used in anger, both now fixed: `--find` needed the full guide stem. An agent called `--find compute-optimizer idle`, got a 404 on the table of contents, and went back to guessing page names by hand. It now probes the eight common guide directories and says which it resolved to, so a bare service name finds the page in one command. `--grep` is a Python regex, and an agent passed grep's `\|` alternation, which matches a literal pipe and so reported "no match" on a page that did contain the term. A false negative there reads exactly like a fabricated page. It now warns on `\|` and exits cleanly on an invalid pattern. ## AIP-C01 re-swept: 207 of 207, and a cutoff that cost nothing AIP-C01 is the largest track in the corpus and sampled second-worst at 3.75 wrong facts a post. All 203 posts not already in the sample have been re-swept; with the sample's four, the whole track is at the 2026-10 standard. | | | |---|---| | posts re-swept | 203 | | claims re-checked | 7,142 | | documentation lookups | 5,022 | | wrong facts | **684, or 3.37 a post** | | **wrong facts per claim** | **9.6%** | | came back clean | **7 (3.4%)** | | mechanism defects | 173 | | unpublished figures | 102 | | availability defects | **49** | | arithmetic | 82 | | bad documentation fetches | 234 | **The sample predicted 3.75 and the track came in at 3.37.** With SAP-C03's 4.00 against 3.95, two tracks now say a four-post stratified sample sizes a track to within about 10%, which is close enough to cost the remaining five rather than guess at them. ### The weekly limit hit mid-run, and the ledger made it a non-event 34 of the 203 agents died unstarted when the account hit its weekly limit. Because `resweep.py --record` had written each returned post as it landed, the ledger knew exactly which 34 had never run. The fix was to re-launch those 34 and nothing else: no re-sweeping 169 posts to find the gap, no guessing, no work repeated. That is the whole argument for keeping progress in the repo rather than in the session, tested by an actual cutoff rather than reasoned about. ### 49 availability defects, against SAP-C03's 13 This track had drifted hardest, and the reason is in the subject matter: AWS has been closing, renaming and reshuffling GenAI surfaces faster than anything else in the corpus. QuickSight became Amazon Quick, with the old user guide path 302ing to a different guide stem entirely. Five Stability AI and Luma model-card URLs that exist by analogy with every other model do not exist at all, because those models carry only inference-parameter pages. A Titan Image Generator model card links its own service card to the *text embeddings* card, so following AWS's own link returns a well-formed page with no watermarking section, which reads as evidence that watermarking is undocumented. ### Two AWS pages that contradict themselves Worth recording because no read-through catches either. `knowledge-base-supported.html` lists Amazon Nova Multimodal Embeddings as a Knowledge Bases embedding model at 1024 dimensions, while that model's own card marks Knowledge Bases as Not Supported. And the Command R row of `model-lifecycle-legacy.html` has its columns shifted by one, so the provider cell holds the model name, the model cell holds the ID, and the regions cell holds a date; the dates had to be read from the Command R+ row below it. ### What the fetch tool caught, and the two gaps it has 234 bad fetches across 5,022 lookups, and the agents are explicit that `aws-doc.py` refused the dangerous ones rather than letting a summariser answer from memory: `list_amazonbedrock.html` 302ing to an index with a 31-character body where the live page is `list_bedrock.html`, `converse-api.html` likewise, and `nova/latest/userguide/customize-fine-tune.html` which AWS's own Bedrock pages link to twice and which no longer exists. Two limits came out of this run and are worth knowing rather than fixing blindly: `--find` only covers `docs.aws.amazon.com`. An agent looking for the Amplify AI kit got a listing of Amplify Hosting pages and could reasonably have concluded the kit is undocumented; it is documented, on `docs.amplify.aws`. **A thin page is not a shell, and silence on it is not evidence.** Bedrock's `playgrounds.html` is a real 675-character page saying its content moved into the console, and `models-supported.html` is 264 characters pointing at `model-cards.md`. Neither trips the shell check, because neither is a failed render, and a grep for a model name on either returns nothing. One agent nearly recorded a correct console detail as an invented one on that basis. ## AIB-C01 and CLF-C02 re-swept, and the rules have a gap Both tracks are at the 2026-10 standard. AIB-C01 is the business track and CLF-C02 the foundational one, and both were chosen over the higher-rate AIF-C01 because CLF-C02's 26 posts are all live. | | AIB-C01 | CLF-C02 (with AIB remainder) | |---|---|---| | posts | 88 | 25 | | claims re-checked | 1,365 (first 42) | 2,066 (63-post batch) | | wrong facts | 97 in 42 | 146 in 63 | | per post | 2.31 | 2.32 | | came back clean | 7 of 42 | 10 of 63 | | availability defects | 7 | 17 | **A third and fourth track inside 10% of their sample.** AIB sampled 2.25 and measured 2.31. Across SAP (4.00 against 3.95), AIP (3.75 against 3.37) and these, a four-post stratified sample is now a reliable way to size a track. ### The evidence rules only cover AWS, and these tracks are not only AWS This is the finding worth keeping. The four rules all concern AWS's own documentation, and both tracks cite sources that are not AWS at all. Those sources fail in ways the rules do not describe: **A statute whose text renders client-side.** `legislation.gov.au` returns over a megabyte of HTML for the Fair Work Act in which the table of contents is present and the section bodies are not. An agent caught it because the entry for s205 was there while "major change" and "significant effect" appeared nowhere in the body. `ato.gov.au` returns 403 to this proxy and austlii serves a Cloudflare block under an HTTP 200. **An official register that answers 202 with an empty body.** EUR-Lex returned HTTP 202 and zero bytes for all three URL forms of the EU AI Act amendment, so the primary source was simply unavailable. **A marketing page whose head is navigation.** `aws.amazon.com/compliance/iso-42001-faqs/` is 348KB of which the first 4KB is a menu naming unrelated services, which to a summarising read is indistinguishable from page content. The per-programme compliance paths are case-sensitive: `/FedRAMP/` and `/SOC/` return 200 where `/fedramp/` and `/soc/` return 404. **A 404 served as a 322KB page.** The AWS what's-new URL for the ISO/IEC 42001 certification returns 404 with a body full of real AWS product copy; the only tell is one "Error - 404 - Not Found" line. **And a fetch that fabricated against a real page.** A WebFetch of an AWS security blog post returned an ISO/IEC 42001 Annex A control list the page does not support, including an A.8.6 that does not exist and an A.10.3 titled "denial-of-service", having conflated a STRIDE-style risk mapping with the real control titles. Caught only because AWS's own compliance PDF gives A.8 a different title. The rules need a fifth that covers a non-AWS primary source: what counts as one, what to do when it cannot be fetched, and that a legislative table of contents is not the legislation. The agents handled these correctly by instinct this time, reattributing rather than sourcing, which is not something to rely on twice. ## AIF-C01 and SAA-C03 re-swept in full (October 2026) Both tracks are through a second independent pass against AWS's own documentation and promoted to the 2026-10 standard. Eight of the nine written tracks are now at it; ANS-C01's 109 posts are the whole remaining backlog. | | AIF-C01 | SAA-C03 | |---|---|---| | Posts | 92 | 93 | | AWS claims checked | 2,440 (26.5 a post) | 4,898 (52.7 a post) | | Documentation lookups | 2,074 | 2,789 | | Wrong facts corrected | 231 (2.51 a post) | 261 (2.81 a post) | | Wrong per claim | 9.5% | 5.3% | | Posts clean on the second pass | 9 of 92 (10%) | 6 of 93 (6%) | | Mechanism defects (rule A) | 63 | 92 | | Unpublished figures (rule B) | 26 | 25 | | Availability defects (rule D) | 14 | 6 | | Arithmetic | 31 | 54 | | All findings | 365 (3.97 a post) | 438 (4.71 a post) | | Subagent tokens | 12.6M | 16.0M | Every one of these 185 posts was already recorded verified. **15 came back clean.** That is the measurement the ledger's `standard` field exists to carry: a sweep is a record that someone looked, not a claim that the post is right. ### A four-post sample sizes a track, within about 25% Three full tracks have now been measured against their own four-post samples. SAP-C03 predicted 4.00 wrong facts a post and measured 3.95, AIF-C01 predicted 2.75 and measured 2.51, SAA-C03 predicted 2.25 and measured 2.81. So sampling is good enough to cost a track before committing to it, and SAP-C03's 1% was luck rather than the method's precision. Plan on plus or minus a quarter. ### Format dominates, so a sample has to be stratified by it AIF-C01 ran as two batches that happened to split by format, and they came in at **1.57 and 3.46 wrong facts a post**: a factor of 2.2 within one track, one standard, one brief. The short formats (pop quiz, flash card, both under a 450-word frontmatter ceiling) carry about 20 claims a post. The long formats carry 32 on AIF-C01 and 53 on SAA-C03, and they are wrong more often per claim as well. A sample drawn without regard to format predicts the format it drew, not the track, which is why the half-track figures looked at first like the sampling method failing when the whole-track figures confirm it. ### Mechanism defects live in the long formats SAA-C03 returned 92 mechanism defects in 93 posts and AIF-C01 63 in 92, almost all of them in the scenario and exploratory posts rather than the quizzes. This is the defect class no read-through catches, because nothing on the page contradicts anything else on the page: the figures are right and the architecture cannot be built. For SOA-C03, DVA-C03, DEA-C01 and MLA-C02, still unwritten, this is where the verification effort belongs. ### What the sweep says about the tooling `scripts/aws-doc.py` earned its place again. Across both tracks the agents reported dozens of fetches it refused rather than answered: renamed pages that 302 to a guide index and extract to 20-odd characters of shell (`about-endpoints-healthcheck-options.html`, `sticky-sessions.html`, `storage-classes.html`, `Composite_Alarms.html`), AWS pricing pages whose every rate is an unrendered `{priceOf!...}` placeholder (S3, Bedrock, RDS Proxy, ElastiCache, Step Functions, CloudWatch, Timestream, Glue, Comprehend, Storage Gateway), and guessed exam-guide stems that 404 behind a page titled "Amazon Web Services Documentation". Two failures are worth naming because the tool does not catch them: - **`--find` with a bare service name resolves to whichever guide answers first, and reports "no match" from there.** `--find wellarchitected 'rel.*retry'` searched the user guide's 138 pages and found nothing, which reads as evidence the practice does not exist; the Reliability pillar needs the full stem `wellarchitected/latest/reliability-pillar`. Same shape with `AmazonRDS` resolving to the API reference instead of the user guide. A false negative here is indistinguishable from a page that genuinely says nothing. - **A genuine page can be too thin to be the page you wanted.** `bedrock/latest/userguide/models-supported.html` returns 264 characters that are a real pointer to `model-cards.html`, and `bedrock/latest/userguide/quotas.html` returns a 1,823-character index with no quotas on it. Both pass the shell check because both are real content. The guard is implausibility: a models-supported page cannot be 264 characters. One AWS self-contradiction is recorded rather than resolved: Titan Image Generator G1 v2's model card gives an EOL date of 30 June 2026 and marks it Legacy, while the Legacy table omits it, the region-compatibility page shows it plainly supported, the pricing page still sells it and Service Terms section 50.10 still indemnifies it. Asserting either way would be inventing a fact, so four AIF-C01 posts naming that model are left as they stand, with the conflict noted here for the next sweep.