Lab: Immutable Backups With S3 Object Lock

This is the third lab in the SA Pro hands-on track. The reference posts argue the decisions; this stands one up. It pairs with the post on ransomware-proof backups across an organisation, which walks the whole picture: accounts, Regions, and where the immutable copy sits. Here we build the smallest honest slice of it, one source bucket and one guarded destination in a single stack. The full lab is in lab-sa-pro-03-immutable-backup.zip.

If this is your first lab, do the one-time, once-per-account setup first: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every SA Pro lab tags its stack for the reaper on deploy.

Everything runs in ap-southeast-2, and this is one of the cheap ones. S3 charges storage on both copies, one replication PUT per replicated object, and, for cross-Region replication only, inter-Region data transfer out. Same-Region replication of one small test object is a handful of requests and a few bytes held twice, so the bill rounds to nothing, and Object Lock has no line of its own on the S3 pricing page. Run teardown.sh anyway. CloudFormation will not delete a bucket that still holds objects, and the reaper’s own emptying pass sends no governance bypass, so a forgotten stack here can outlive the 24-hour clock.

The scenario

A backup in the same account and Region as the thing it protects shares that thing’s fate. An attacker who compromises the account can delete the data, delete the backups, and then delete the versions of the backups, because every one of those deletes is a call the compromised account is allowed to make. A copy counts as a backup only when the account cannot destroy it while fully owned: a delete that is not permitted, for anyone, until a retention clock runs out.

S3 gives you both halves. Replication makes the second copy automatically on every object written after the rule lands, with no batch job to forget. Object Lock holds that copy, so a version under retention cannot be overwritten or deleted until the lock expires. A nightly copy script and a bucket policy leave both halves inside reach of the same credentials. The design that holds is replication into an Object Lock bucket, ideally in another account, so the copy sits outside the blast radius of whatever wrote it.

Replication does not backfill. Objects already in the source when the rule lands stay where they are, and moving them takes a Batch Replication job.

What you’re given

CloudFormation builds the destination bucket with versioning and S3 Object Lock, and the source bucket with versioning. Both carry explicit names, which matters later. The destination is already guarded: its Object Lock configuration sets a default retention on every new version.

DestinationBucket:
  Type: AWS::S3::Bucket
  Properties:
    BucketName: !Sub '${AWS::StackName}-dest-${AWS::AccountId}-${AWS::Region}'
    VersioningConfiguration:
      Status: Enabled
    ObjectLockEnabled: true
    ObjectLockConfiguration:
      ObjectLockEnabled: Enabled
      Rule:
        DefaultRetention:
          Mode: GOVERNANCE
          Days: 1

Three things about that block are worth pinning down. Object Lock requires versioning, which is why both properties sit together, and it goes on a new bucket or an existing versioned one. The switch is one-way: once Object Lock is on you cannot disable it and you cannot suspend versioning on that bucket, and turning it on locks nothing already sitting there.

The mode is GOVERNANCE, not COMPLIANCE, on purpose. GOVERNANCE lets a caller holding s3:BypassGovernanceRetention delete a locked version by sending x-amz-bypass-governance-retention:true, which is --bypass-governance-retention on the CLI, and that is how the teardown script clears the bucket in minutes. Production uses COMPLIANCE, which no principal can bypass or shorten, including the account root user. AWS documents exactly one way out before the retain-until-date, and it is deleting the AWS account. That is real immutability, and it is also why a COMPLIANCE bucket cannot be torn down on demand.

The third thing is whose retention wins. A replica takes the Object Lock settings of the source object, and the destination’s default applies only where the source object carries none. The source bucket here has no Object Lock, so every replica picks up the destination’s one-day GOVERNANCE default. Turn Object Lock on at the source and two things follow: the destination must have it enabled too, and the replication role needs s3:GetObjectRetention and s3:GetObjectLegalHold to carry the retention across.

The gap is the source bucket. As shipped it has versioning but no replication rule, so writes land in the source and never reach the guarded destination.

Your task

Add two things to src/template.yaml. First, an IAM role S3 assumes to replicate, trusting the s3.amazonaws.com service principal for sts:AssumeRole. Its policy is the standard replication set in three statements: s3:GetReplicationConfiguration and s3:ListBucket on the source bucket ARN; s3:GetObjectVersionForReplication, s3:GetObjectVersionAcl and s3:GetObjectVersionTagging on that ARN plus /*; and s3:ReplicateObject, s3:ReplicateDelete and s3:ReplicateTags on the destination ARN plus /*. Second, a replication rule on the source bucket:

SourceBucket:
  Type: AWS::S3::Bucket
  Properties:
    BucketName: !Sub '${AWS::StackName}-source-${AWS::AccountId}-${AWS::Region}'
    VersioningConfiguration:
      Status: Enabled
    ReplicationConfiguration:
      Role: !GetAtt ReplicationRole.Arn
      Rules:
        - Status: Enabled
          Priority: 0
          DeleteMarkerReplication:
            Status: Disabled
          Filter:
            Prefix: ''
          Destination:
            Bucket: !GetAtt DestinationBucket.Arn

The empty-prefix Filter matches every object, and a rule that carries a Filter has to carry a DeleteMarkerReplication element as well. Disabling it stops a delete on the source propagating a delete marker to the guarded copy. A delete that names a version ID is never replicated either way, so the destination keeps what the source loses. Encryption stays at the bucket default, SSE-S3, so there is no KMS key to grant the role.

One detail decides whether the stack even builds. The source’s replication config references the role, and the role’s policy needs the source bucket’s ARN. Write that ARN with !GetAtt SourceBucket.Arn and CloudFormation fails the stack with a circular-dependency error: source needs role, role needs source. Build the source ARN from a !Sub string using the same name expression the bucket uses, and the cycle goes, because the role no longer points back at the source resource.

Run it

./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # writes an object, checks replication and the lock
./scripts/teardown.sh        # empties both buckets and deletes the stack

test.sh writes one object to the source, polls the destination twelve times at five-second intervals for the replicated copy, then takes that copy’s version and tries to delete it without a bypass. With the starter template nothing replicates, so the test stops and tells you the rule is missing. With the role and rule in place it prints:

PASS: the object replicated to the guarded bucket, and Object Lock refused to delete the protected version.

The delete comes back as an Access Denied error, HTTP 403, which is what S3 returns for a versioned delete of a protected version. A delete with no version ID would have returned 200 and left a delete marker, because delete markers are not WORM-protected whatever the retention underneath them says.

teardown.sh needs s3:BypassGovernanceRetention on whatever principal you run it as. Without it the destination’s locked versions stay put and the stack delete fails on a non-empty bucket.

If it fails

• Nothing replicates. Replication is asynchronous, and AWS publishes a time bound only for rules with S3 Replication Time Control on, where 99.9% of objects arrive within 15 minutes. Without it there is no published figure, so the script’s minute is a convenience rather than a verdict: re-run test.sh before concluding anything. If the copy still never arrives, the source bucket has no replication rule, or the role is missing or lacks the read permissions on the source.
• The stack will not build (circular dependency). The role’s policy is referencing the source bucket with !GetAtt SourceBucket.Arn. Build the source ARN from a !Sub string instead so the role does not depend on the source resource.
• The delete succeeds. The object replicated but Object Lock is not guarding the destination. Confirm the destination has ObjectLockEnabled: true, an ObjectLockConfiguration with a default retention, and versioning on. Confirm too that you passed a version ID: a delete without one adds a delete marker and returns 200 on a locked bucket.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• A backup inside the same account and Region as its source shares that source’s blast radius. Separation and immutability are what make it a backup rather than a copy.
• Replication copies every object written after the rule lands, with no schedule to miss, and backfills nothing: existing objects need a Batch Replication job. S3 assumes the role itself, reading on the source and writing on the destination.
• Object Lock protects a version, not a bucket. GOVERNANCE gives way to s3:BypassGovernanceRetention plus an explicit bypass header; COMPLIANCE gives way to nobody in the account.
• Object Lock goes on a new bucket or an existing versioned one, and then it is one-way: no disabling it, no suspending versioning, and nothing already in the bucket is locked by the switch.
• A replica takes the source object’s retention where it has one, and the destination bucket’s default where it does not.
• Moving the destination to another Region changes one ARN. Moving it to another account also adds a bucket policy there granting the replication role s3:ReplicateObject and s3:ReplicateDelete, plus a decision on whether to override replica ownership.

Next

The rest of the SA Pro lab track is in the README.