Verifying a track

One track a week, in a fresh context. This is the whole procedure; the prompt that starts a run is three lines and points here.

EXAM-VERIFICATION.md is the record of what past runs found and what the findings mean. This file is how to do one.

The prompt

Paste this, with the certification code substituted:

Verify the SCS-C03 Exam Room track. Follow VERIFICATION-RUNBOOK.md. Work through as many batches as the session’s budget allows, commit as posts settle, and stop with an honest count rather than a claim of completion.

Nothing else is needed. The level note, the staleness list and the batch of paths all come from the repository.

What one batch looks like

python3 scripts/verify-track-args.py SCS-C03 --status   # how much is left
python3 scripts/verify-track-args.py SCS-C03            # args for the next batch

Feed that JSON to the workflow:

Workflow({ scriptPath: 'scripts/workflows/verify-exam-track-v2.js', args: <the JSON> })

Two workflows at once is the ceiling. The box has four cores and the runner allows CPUs - 2 agents per workflow, so two workflows is four agents and a third only makes all three slower. A post takes most of an hour, so a batch of 18 is about nine hours of wall clock and roughly 3.3M subagent tokens.

While they run, commit what settles:

./scripts/verify-commit-cycle.sh "Verify SCS-C03: more settled posts"
git push

That script holds back a file written in the last 45 seconds, one whose frontmatter will not parse, and one that fails check-exam-room.py where the origin/main copy does not. When it says a file is still being written, wait for it. Committing a half-rewritten post is the one failure this whole arrangement exists to avoid. Push once per cycle rather than per commit: every push cancels the in-progress CI run.

Recording what was done

_data/exam_verified.yml is the only state. After each batch:

python3 scripts/record-verified.py SCS-C03 <the batch's post paths>
python3 scripts/record-verified.py SCS-C03 --complete   # once nothing is left

It takes paths, or --from-journal <run-dir> to read them out of a workflow run’s journal.jsonl. It is idempotent, it refuses a post whose cert_code belongs to another track, and --complete refuses while any post in the track is unrecorded.

verify-track-args.py skips what is recorded, so the next call gives the next batch and a batch that died half way through is picked up by whatever it did manage to record.

Do this even if the session is about to end. A run that verified forty posts and recorded none of them has done the work twice.

Finishing a track

make reading-times                               # restamps, regenerates checklists
python3 scripts/check-exam-checklists.py --strict
python3 scripts/check-preview-flags.py
python3 scripts/check-published.py
python3 scripts/check-currency.py --strict
python3 scripts/check-exam-room.py
python3 scripts/exam-verification-status.py

make reading-times renames nothing but rewrites many files, so run it only once every agent has stopped. The same goes for python3 scripts/relay-exam-queue.py --apply, which renames files and will pull the ground out from under an agent mid-edit.

Then add the track’s row and its findings to EXAM-VERIFICATION.md, open a PR, and merge once CI is green.

What comes back that is not a fix

Two kinds of finding need a person, and the verifier is told to report them rather than act:

A framing built on a dead choice. Where a post’s whole argument is “should we adopt X” and X has closed to new customers, a clause is not the fix. The post either gets retired and its URL redirected to whatever covers the same ground, or rewritten around the service AWS points at instead. Both change a URL, so both are decisions. CloudTrail Lake is the live example: several SCS-C03 posts pick it as the answer, and it closed to new customers on 31 May 2026.

A claim that could not be confirmed. The verifier is told to put these in remaining rather than leave them in the post looking sourced. Read that list at the end of a run; it is usually short and it is usually the most interesting thing in the report.

What to expect

Every track measured so far comes back at six to eight wrong facts per post, whether or not it was planned from the exam guide first. Planning fixes coverage and does nothing for accuracy: SAA-C03 gated at 187 of 189 objectives and still carried 6.1 wrong facts a post, the same rate as a track written without a plan. Treat a batch that finds nothing as a weak check rather than a clean batch.

The defect that recurs most is a post contradicting itself a few sentences apart, and it needs no external knowledge to catch: an X-Ray sampling rule scoped to a child service and then explained as a rule that never fires, a claim that a fully pinned RDS Proxy workload keeps connection reuse, an EventBridge rule for root sign-in in a Region where that event is not recorded. The arithmetic step catches the numeric version of this. Ask the same question of behaviour.