SAA Lab 01 - Rotate a secret without restarting the application

Scaffold: 4/5. The VPC, the Secrets Manager interface endpoint, the Aurora PostgreSQL cluster, the superuser secret, the application secret and the stand-in application are all built and wired. What is missing is the rotation schedule. You write it, then watch the application serve traffic straight through a rotation on a credential it cached an hour’s worth of ago.

The scenario

An application reads its database password from Secrets Manager once and keeps it in memory. That is the right shape: a credential fetched on every request puts an API call in the request path, and the Secrets Manager Python caching client exists so you do not have to. Its default refresh interval is 3600 seconds, so a rotation that happens now is invisible to a running process for up to an hour.

Which leaves the question this lab is about. If the password in the database has changed and the process is still holding the old one, what happens to the next connection it opens? The answer depends entirely on which rotation strategy is configured, and you can watch both answers happen.

The requirement

AppSecret holds the credential for the appuser role in labdb. Give it a rotation schedule that uses the alternating users strategy, so that during and after a rotation a process holding the previous credential can still log in.

The gap is a TODO block in src/template.yaml, and it is one resource: an AWS::SecretsManager::RotationSchedule with a HostedRotationLambda. The block spells out the properties and the four details that decide whether it works.

Run it

./scripts/deploy.sh          # deploys src/template.yaml, then packages app.py
./scripts/test.sh            # creates appuser, rotates, keeps calling the app
./scripts/teardown.sh        # deletes the stack and force-deletes the secrets

The first deploy creates an Aurora Serverless v2 cluster, so allow about ten minutes. test.sh prints one line per call while the rotation runs.

Then break it on purpose

Change PostgreSQLMultiUser to PostgreSQLSingleUser in your template, drop MasterSecretArn, redeploy and run test.sh again. Single-user rotation changes the password of the one user the cache is holding, so the calls that land after the change are denied until the cache refreshes. That is the failure the alternating-users strategy is there to avoid, and it is worth seeing once.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

Cost

An Aurora Serverless v2 cluster at 0.5 ACU, an interface VPC endpoint, two secrets and a handful of Lambda invocations. Small, but not free, and the cluster bills while it exists. Tear down when you are finished; the lab reaper is a backstop, not a plan.