SAA Lab 05 - Move ten thousand objects down a lifecycle ladder

Scaffold: 4/5. The versioned bucket, its cost allocation tag, the Storage Lens dashboard, the storage class analysis filter, the loader and the three measuring scripts are all built and wired. What is missing is src/lifecycle.json, which ships as an empty rule list.

This lab runs in two sittings about a day apart. NoncurrentDays cannot be zero, so the version rules cannot act until tomorrow, and both surfaces that report what lifecycle did (Storage Lens and Cost Explorer) run a day behind it. deploy.sh sets LAB_TTL_HOURS=48 so the reaper leaves the stack alone in the meantime.

The scenario

Sounding Line Hydrographics runs multibeam sonar surveys out of Hobart and has never deleted anything. One bucket, ten thousand current objects under three prefixes: 2,000 raw swath files at 160 KB, 1,000 gridded products at 192 KB with three noncurrent versions behind each of them, and 7,000 chart tiles at 24 KB. The swaths are read once and then not again, the grids weekly, the tiles all day. Five hundred of the swaths belong to a survey leg under dispute, and counsel has asked that those stay where they can be read in a click, with no restore step and no retrieval charge to argue about later.

Everything runs in ap-southeast-2, and every price in this lab is the Sydney list price in USD$.

The requirement

Four rules in src/lifecycle.json. The _TODO block in that file spells each one out; apply-lifecycle.sh refuses to run while it is still there.

archive-swaths moves the swaths to Glacier Instant Retrieval at day zero, except the disputed leg. Lifecycle has no way to say “everything except”, so the exception is expressed by tagging the objects the rule should catch, and a filter combining a prefix with tags has to be wrapped in And.
grid-versions transitions noncurrent versions to Standard-IA after one day and expires everything beyond the two newest. Both actions land on the same versions on the same day, and S3 resolves that with a documented precedence: permanent deletion wins over transition.
archive-tiles moves the tiles to Glacier Instant Retrieval at day zero. It is valid, it will be accepted, it will be enabled, and tomorrow it will have moved nothing at all. Work out why before you read the failure list.
housekeeping aborts incomplete multipart uploads after seven days and cleans up expired object delete markers. Neither action is allowed in a rule whose filter uses object tags, so they live in a rule of their own.

Then activate the tag. In Billing and Cost Management, under Cost allocation tags, activate cost-centre, at the start of the first sitting. An applied key can take up to 24 hours to appear on that page and another 24 to activate, and a tag attaches to usage only from the moment it is on the resource. cost-centre is on the bucket: cost allocation tags label buckets. The retention tags the lifecycle rules filter on are object tags, they never reach the bill as a dimension, and they carry a small charge of their own.

Run it

./scripts/deploy.sh                 # bucket, Storage Lens dashboard, analytics config
./scripts/load.py                   # 13,000 PUTs, ~1.2 GiB
./scripts/apply-lifecycle.sh        # put-bucket-lifecycle-configuration
./scripts/count.py                  # the baseline
#  --- come back the next day ---
./scripts/count.py                  # what moved
./scripts/cost.py                   # Cost Explorer, by usage type, filtered on the tag
./scripts/teardown.sh

Apply the configuration early in your day. S3 evaluates rules daily and stamps the resulting action at midnight UTC, which is 08:00 in Perth.

If it fails

Nothing moved overnight. A configuration is replaced wholesale on each PUT, never merged, so check with get-bucket-lifecycle-configuration that the rules on the bucket are the ones you wrote. Actions are queued asynchronously; hours between a rule being satisfied and an object moving is normal.
InvalidRequest when applying. NewerNoncurrentVersions was specified without a Filter element in that rule.
tiles/ is still entirely in S3 Standard. Objects smaller than 128 KB are not transitioned to any storage class by default, and S3 has applied that floor to all transitions since September 2024. An ObjectSizeGreaterThan in the filter will move them. Do the arithmetic first: 7,000 transitions cost USD$0.14, and Glacier Instant Retrieval bills a 128 KB minimum per object, so 7,000 tiles of 24 KB would be billed as 0.85 GiB instead of 0.13 GiB. The transition raises the storage bill and charges USD$0.14 to do it.
The disputed swaths moved too. The filter is missing its And wrapper, or the tags are not what you think. Check one object with get-object-tagging.
The object count went up. Expiring a current version in a versioning enabled bucket writes a delete marker, and delete markers count as objects.
Cost Explorer shows nothing under the tag. The key is not activated yet, or it was activated after the charges were incurred. Use cost.py --no-tag in the meantime; usage types need no activation.
A Deny in the bucket policy did not stop a rule. Lifecycle actions do not go through DeleteObject; they run on internal S3 endpoints. Disable or delete the rule instead.

Reveal the solution

SRC=solution ./scripts/apply-lifecycle.sh && ./scripts/count.py

Cost

About USD$0.20 all in: 13,000 PUTs, 3,500 lifecycle transitions, a day or two of storage on 1.2 GiB, and a Storage Lens dashboard on the free tier. Tear down the day after the second sitting; early-deletion charges on Standard-IA and Glacier Instant Retrieval are a cent or so here and would not be on forty terabytes.

Untested

Every script here is syntax-checked, the templates parse and both lifecycle configurations are valid JSON, but this bundle has not been deployed against a real AWS account end to end. Treat the first run as the shakedown.