Verifying the Exam Room tracks

Every Exam Room track was drafted by agents and published without anyone checking what it claimed. On ANS-C01, checking found about seven wrong facts per post, and every post in both sample batches had at least one. Wrong quotas, inverted rules, a copy-pasteable IAM policy that did not do what the post said. So a track that has not been through this sweep should be assumed wrong, not assumed right.

This file is for picking the work up cold. Nothing here depends on the conversation it started in.

Where things stand

Run it, do not guess:

python3 scripts/exam-verification-status.py            # per-track table
python3 scripts/exam-verification-status.py --todo AIP-C01   # files still to do

A post records its own state in frontmatter, verified: YYYY-MM-DD, set on the day it was checked against the documentation. A post without that field has not been checked. Publication is not evidence of verification: AIF-C01 and AIP-C01 both finished publishing without ever being checked, which is why the status script counts live and unverified posts separately and flags that combination.

ANS-C01 is done: 113 posts verified, 799 facts corrected, 70 self-contradictions fixed, full coverage against the official exam guide.

The order of work

Set by which exams Craig is sitting, which beats every other consideration: he is the reader, and he is revising from this material.

CLF-C02. 23 posts, publishing now, and he is studying for it.
AIP-C01. 197 still unverified. He is studying for this one too, which is why it gets the full treatment despite being a finished run. The calibration measured 12.1 wrong facts per post here, the worst of any track, because generative-AI content goes stale fastest.
Then in publication order, since that is the order he will revise in: ANS-C01 (done), AIB-C01 (January), SAP-C03 (March), DOP-C02 (June), SCS-C03 (September 2027). Each wants verifying before its window opens, while it is still preview: true and has no live URL to protect.
AIF-C01 last, if there is budget. 96 posts, finished publishing on 3 September.

An earlier draft of this file put the live GenAI tracks first on the grounds that wrong content was reaching readers, and ranked a finished run as an archive not worth full verification. That was wrong about who the reader is.

PR and merge in batches of roughly 25 rather than one PR per track. A 200-post PR is unreviewable and one bad file blocks the rest.

Running it

Three workflow scripts live in scripts/workflows/. They are copies: the originals sat under a session directory that dies with its session, and rewriting the prompts from scratch loses the accumulated corrections below.

verify-exam-track.js is the full treatment, one agent per post: facts against the docs, then style, rhythm, structure, links and Jekyll. Pass an array of repo-relative post paths as args. Roughly 161k tokens per post, measured, which is the number to budget from.
check-internal-consistency.js finds where a post contradicts itself (a table cell against the prose that argues the case, a worked example computing from a different number, an SVG description naming a structure other than the one drawn). No web access, roughly 24k tokens per post. On ANS it found 70 real defects for an eighth of the price, so it is worth running across a whole track before committing to full verification.
coverage-gate.js checks every “Knowledge of” and “Skills in” bullet in the official guide against the track. Read-only. Cheap, four agents.

Exam guides live at https://docs.aws.amazon.com/aws-certification/latest/<stem>/<stem>-domainN.html. Read them; weightings and task statements change between versions.

What went wrong before, so it does not again

Never set effort: 'low' on the agent options. A run with it set had all twelve agents blocked from every tool call by a permission-handler error and burned 813k tokens producing nothing. The symptom looks like a schema failure in the summary; the agents’ own transcripts say “every tool call was rejected”.

Judge each file on its own when committing. An early commit guard held the whole batch whenever one file failed a check, so a single flash card stalled everything. Skip files modified in the last 45 seconds (their agent is still writing), skip a file whose frontmatter fails yaml.safe_load or whose check-exam-room.py prints a warning, and commit the rest.

A rhythm failure is usually pre-existing. Flash cards overrun their frontmatter budget on main as well: nine of twelve sampled. Only treat measure-post.py failing as this sweep’s problem if the same file passes on origin/main.

Do not hand-edit a generated checklist. make reading-times rebuilds them from _data/exam_checklists/*.yml. Editing the post directly is overwritten.

On a published post, change content only. Never touch date:, preview:, the filename or the permalink. Re-dating a live post drops it off the listings and the feed while its URL still resolves, and _data/linkedin_shares.yml has public links with UTM parameters attached. scripts/check-published.py catches it; run it before every push.

The branch is deleted when its PR merges. That leaves a stale tracking ref, and git push --force-with-lease then refuses with “stale info” against a branch that no longer exists. git fetch --prune origin fixes it. After a merge, restart from main rather than pushing onto merged history.

Finishing a batch

make reading-times                                   # restamps, regenerates checklists
python3 scripts/check-exam-checklists.py --strict
python3 scripts/check-exam-room.py <the track's posts>
python3 scripts/check-preview-flags.py
python3 scripts/check-published.py

Then commit, push, open a PR, and merge once CI is green. Stamp verified: YYYY-MM-DD into the frontmatter of every post the batch checked, or the next run will do them again.