Exam Room · Solutions Architect

Lab: Rotate a Secret Without Restarting the Application

· 20 min read

Cloud Architecture · part of The Exam Room

This is the first lab in the Solutions Architect Associate hands-on track. The reference posts argue the decisions; this one stands a decision up and makes it behave. The full lab is in lab-saa-01-secret-rotation.zip.

If this is your first lab, do the one-time, once-per-account setup first: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every lab tags its stack for the reaper on deploy. This one creates an Aurora cluster, which bills for as long as it exists, so tear down when you are finished.

The scenario

An application needs a database password. Fetching it from Secrets Manager on every request puts an API call in the request path and a per-call charge on the bill, so the application fetches it once and holds it. The Secrets Manager Python caching client exists for exactly this, and its default refresh interval is 3600 seconds. A process that started an hour ago may be holding a value nobody has re-read since.

Now rotate the secret. The password in the database changes. The value in Secrets Manager changes. The value in the running process does not, and will not for up to an hour. The next connection that process opens uses a credential that is, by the clock, out of date.

Whether that connection is accepted decides whether rotation is something you schedule or something you announce. Secrets Manager documents two rotation strategies for database secrets, and this case is where they differ. In this lab you build the schedule, force a rotation, and keep calling the application all the way through it.

What you’re given

CloudFormation builds the world around the rotation. A VPC with two private subnets, no internet gateway and no NAT. An interface VPC endpoint for Secrets Manager, with private DNS on, so code inside those subnets reaches the service without a route to the internet. An Aurora PostgreSQL Serverless v2 cluster with the RDS Data API turned on, so the test script can run SQL over HTTPS with no driver and no bastion host. Two secrets: a superuser credential for the cluster’s master user, and the application’s own credential for a database role called appuser.

The application is a Lambda function in the same subnets. It reads its credential through the caching client and opens a real PostgreSQL connection with it:

_client = botocore.session.get_session().create_client("secretsmanager")
_cache = SecretCache(config=SecretCacheConfig(), client=_client)

def lambda_handler(event, context):
    creds = json.loads(_cache.get_secret_string(SECRET_ARN))
    conn = pg8000.native.Connection(
        user=creds["username"], password=creds["password"],
        host=creds["host"], port=int(creds["port"]),
        database=creds["dbname"], ssl_context=True, timeout=10)

The cache is built at module scope, outside the handler, which is what makes it a cache: it survives between invocations of a warm execution environment. Build it inside the handler and every invocation is a fresh GetSecretValue call and the lab has nothing to show you. The execution role grants both secretsmanager:GetSecretValue and secretsmanager:DescribeSecret, because the caching client calls DescribeSecret first to work out which version carries AWSCURRENT, and only then fetches that version. Grant GetSecretValue alone and the first fetch fails, not the first refresh.

The application’s secret carries more than a username and password:

SecretStringTemplate: !Sub |
  {"engine": "postgres",
   "host": "${DbCluster.Endpoint.Address}",
   "port": 5432,
   "dbname": "labdb",
   "username": "appuser",
   "dbClusterIdentifier": "${DbCluster}",
   "masterarn": "${MasterSecret}"}

A rotation function reads those keys to find the database. engine has to read postgres or aurora-postgresql, the two values the PostgreSQL templates accept. masterarn is the ARN of the superuser secret. It sits in the secret rather than in the code because CloudFormation does not hardcode secret ARNs into a rotation function, so one function can rotate many secrets.

The superuser secret cannot be built that way. The cluster resolves its master password out of that secret, so the secret cannot reference the cluster: CloudFormation rejects the cycle. The template breaks it with an AWS::SecretsManager::SecretTargetAttachment, which runs after both resources exist and adds the database connection information to the secret’s JSON.

The gap is the rotation schedule. As shipped, the secret sits there being fetched and never changing.

Your task

Fill in the TODO block in src/template.yaml with one resource:

AppSecretRotation:
  Type: AWS::SecretsManager::RotationSchedule
  DependsOn: MasterSecretAttachment
  Properties:
    SecretId: !Ref AppSecret
    RotateImmediatelyOnUpdate: false
    RotationRules:
      ScheduleExpression: 'rate(4 hours)'
      Duration: '1h'
    HostedRotationLambda:
      RotationType: PostgreSQLMultiUser
      RotationLambdaName: !Sub '${AWS::StackName}-rotate'
      MasterSecretArn: !Ref MasterSecretAttachment
      VpcSubnetIds: !Join [',', [!Ref SubnetA, !Ref SubnetB]]
      VpcSecurityGroupIds: !Ref LambdaSecurityGroup

PostgreSQLMultiUser is the alternating-users strategy, and it is the one line this lab turns on. On the first rotation the function clones appuser into appuser_clone with a generated password and makes the clone the current version. On every rotation after that it alternates which of the two users it changes the password for. After a rotation, both credentials are valid, which is why a process holding the previous one keeps working. PostgreSQLSingleUser changes the single user’s password and has no second user to fall back to.

Cloning needs a privilege the application’s own role does not have, which is what MasterSecretArn is for. CloudFormation grants the rotation function GetSecretValue on that secret; the masterarn key inside the application’s secret is how the function finds it at run time. Both are required, and missing either one produces a different error.

Three details are easy to get wrong. VpcSubnetIds and VpcSecurityGroupIds are comma-separated strings rather than YAML lists. That is why two subnets need a !Join and a single security group is a plain !Ref. Leave both properties out and the rotation function is created outside the VPC, with no path to the database. The Transform: AWS::SecretsManager-2024-09-16 at the top of the template expands HostedRotationLambda into a real function, and a template with a transform needs CAPABILITY_AUTO_EXPAND on deploy. And rate(4 hours) is the floor: four hours is as often as Secrets Manager will rotate a secret on a schedule, inside a window you can set no smaller than an hour.

RotateImmediatelyOnUpdate: false stops the deploy from running a full rotation before the appuser role exists. It does not mean nothing runs. Secrets Manager still executes the testSecret step to check the configuration, creating an AWSPENDING version and then removing it. Nor does it reliably hold back a rotation already due: on a rate() schedule the previously scheduled rotation may still happen, which is worth knowing before you redeploy. A cron() expression is what gives you exact control over when a window opens.

Run it

./scripts/deploy.sh          # deploys src/template.yaml, then packages app.py
./scripts/test.sh            # creates appuser, rotates, keeps calling the app
./scripts/teardown.sh        # deletes the stack and force-deletes the secrets

Allow about ten minutes for the first deploy, because it creates an Aurora cluster. deploy.sh then packages app.py with pg8000 and aws-secretsmanager-caching, both pure Python, and uploads it over the placeholder handler in the template.

test.sh creates the appuser role through the Data API, because Secrets Manager stores credentials and does not create database users; a first rotation against a user who does not exist fails at the second step. Then it calls the application once for a baseline, starts a rotation with rotate-secret --rotate-immediately, and calls the application every five seconds until the AWSPENDING label is gone. With the unedited template it stops at the baseline and tells you there is no rotation schedule. With the schedule in place it prints a line per call and finishes with:

AWSCURRENT  is now appuser_clone
AWSPREVIOUS is now appuser
The warm application is still using appuser, from its cache.

PASS: the secret rotated to a different database user, and the application
served 9 calls through the rotation on a credential it had already cached.

Then break it on purpose. Change PostgreSQLMultiUser to PostgreSQLSingleUser, drop MasterSecretArn, redeploy and run test.sh again. Now the rotation changes the password of the one user the cache is holding, and the calls that land after that change come back denied until the cache refreshes, up to an hour later. That denial is what alternating users is there to avoid.

If it fails

  • The rotation function stalls and times out. It cannot reach a Secrets Manager endpoint, and where the log stops narrows down which piece is missing. A long invocation with nothing after Found credentials in environment variables means no route to the service at all: check that the interface endpoint exists and has private DNS on. Nothing after createSecret means the request leaves and does not arrive. Check three things: that the endpoint’s security group admits the VPC CIDR on 443, that the endpoint policy allows the function, and that the subnets’ network ACLs allow HTTPS both ways.
  • setSecret: Unable to log into database. Either the network or the credential. Check that the database’s security group admits the rotation function on 5432 and that the function’s group allows outbound to it. If the invocation was short rather than long, it is the credential instead: the appuser role does not exist yet, or its password does not match the current version of the secret.
  • Key is missing from secret JSON. The rotation function looked for a key the secret does not have. For alternating users that is almost always masterarn. Key names are case-sensitive.
  • The stack will not deploy: Requires capabilities: [CAPABILITY_AUTO_EXPAND]. The template has a transform. deploy.sh passes it; a hand-rolled cloudformation deploy needs it too.
  • Calls are denied during the rotation. The schedule is single-user, or MasterSecretArn is missing so the clone never got created. Check which username AWSCURRENT carries: if it is still appuser after a rotation, no clone exists.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

  • Caching a secret is the normal, recommended shape, and it puts a window between the moment a credential changes and the moment a running process finds out. The Python caching client refreshes every 3600 seconds by default; the AWS Parameters and Secrets Lambda Extension caches for 300 seconds. Neither number is zero.
  • Single-user rotation updates one password. Open connections are not dropped. Between the password changing in the database and the secret being updated, though, there is a short window where a call using the rotated credential can be denied, and retries with backoff cover it. Alternating-users rotation keeps two users alive and changes one at a time, so a process holding the previous credential still logs in. That is the strategy to reach for when an application must stay available through a rotation.
  • Alternating users needs a superuser secret, because a database user cannot usually clone itself. AWS recommends single-user rotation where the clone would not end up with the same permissions as the original, and notes that changing the original user’s permissions later does not change the clone’s.
  • A rotation function inside a VPC needs a path to Secrets Manager, and an interface endpoint is the way to give it one without a NAT gateway or an internet gateway. The alternative exposes the VPC to the internet for the sake of one HTTPS call.
  • AWSCURRENT, AWSPENDING and AWSPREVIOUS are how rotation stays atomic from the caller’s side. GetSecretValue with no version returns AWSCURRENT, and the label only moves once the new credential has been written and tested.
  • Secrets Manager charges USD$0.40 per secret per month and USD$0.05 per 10,000 API calls, so the cache is a cost decision as well as a latency one. The rate quota on GetSecretValue is 10,000 requests per second per Region. An application of this shape never approaches it, so the per-call charge is the reason to cache, not the quota.

Next

The rest of the Solutions Architect Associate lab track is in the README.

These posts are LLM-aided. Backbone, original writing, and structure by Craig. Research and editing by Craig + LLM. Proof-reading by Craig.