The situation
A document management business spends AUD$34,000 a month on S3 across about 900 TB in eleven buckets. Three years ago it was AUD$9,000. Nobody has made a decision that explains the difference; the business has roughly doubled in that time and the bill has nearly quadrupled.
What is known: the main bucket holds customer-uploaded documents, currently 640 TB, all in S3 Standard. Versioning was enabled four years ago after an incident and no expiry was ever configured for noncurrent versions. A second bucket holds thumbnails generated from those documents, regenerable in about six hours if lost. A third holds application logs going back to the beginning, because “logs are small” was true when it was said. There is a nightly analytics job that lists a large prefix and reads a subset of objects.
Finance has asked for a 40% reduction. The compliance team requires that customer documents remain retrievable within one business day for seven years.
What actually matters
The first thing that matters is that an S3 bill is storage, requests, retrieval and transfer, and a bucket can be dominated by any of them. A bucket of small objects with a heavy LIST and GET pattern can cost more in requests than in storage, in which case moving it to a cheaper storage class saves nothing and may cost more. Knowing the split decides which lever applies.
The second is that “how much data” and “how much of it is accessed” are different questions, and only the second determines the storage class. Seven years of retention with a one-business-day retrieval requirement is a specification that permits archival classes; it is the access pattern, not the retention period, that decides which one.
The third is that the cheaper classes carry conditions that turn into charges when the data does not match the assumption. Minimum storage durations mean deleting or transitioning an object early still bills the remainder of the minimum. Per-object overhead means small objects cost more than their size suggests. Retrieval fees mean a class that is cheap to store can be expensive to read. A transition applied to the wrong data produces a bigger bill, not a smaller one.
The fourth is that some of this is not data at all. Noncurrent versions with no expiry, delete markers with nothing beneath them, and incomplete multipart uploads all occupy paid storage and appear in no application’s view of the bucket. Incomplete multipart uploads in particular are invisible in the console object listing and can account for a startling share of a long-lived bucket.
Underneath it, the bill grew because nothing was measuring it per bucket and per prefix. Whatever the fix, the durable part is the measurement.
What we’ll filter on
- Is this bucket’s cost dominated by storage, requests, retrieval, or transfer?
- What is the actual access pattern, per prefix rather than per bucket?
- Is the data regenerable, and at what cost?
- What does the retention requirement actually specify, retrieval time included?
- Does the object size distribution suit the class being considered?
- Is any of this not data anybody meant to keep?
The landscape
S3 Storage Lens. The measurement layer, and where this work starts. Free metrics cover usage across the estate; the advanced tier adds activity metrics, prefix-level aggregation and cost-optimisation recommendations. It answers which buckets and which prefixes hold the data, how much is noncurrent, and how much is incomplete multipart uploads, which is the question the console cannot answer.
S3 Inventory. A scheduled report of every object with its size, storage class, version status and encryption, delivered as a file you can query with Athena. Where Storage Lens gives aggregates, Inventory gives the per-object detail needed to build a targeted rule rather than a blanket one.
Lifecycle rules. The mechanism for transitions and expiry, scoped by prefix, tag, size or age. They handle transitions between classes, expiry of current and noncurrent versions, cleanup of expired delete markers, and abortion of incomplete multipart uploads. Rules act on age, which makes them right for data with a predictable cooling curve and wrong for data whose access is unpredictable.
S3 Intelligent-Tiering. Moves objects between access tiers automatically based on observed access, for a small per-object monitoring charge. There is no retrieval fee between the frequent and infrequent tiers, which is the property that makes it safe for unpredictable access. It suits data where you cannot predict the cooling curve; it costs slightly more than a correct manual lifecycle rule for data where you can, and it is poor value for objects small enough that the monitoring charge dominates.
Standard-IA and One Zone-IA. Cheaper storage, a retrieval charge per GB, and a minimum storage duration of 30 days. One Zone-IA stores in a single Availability Zone at a lower price and is the correct answer for regenerable data, where losing a zone means regenerating rather than losing.
Glacier Instant Retrieval. Archival pricing with millisecond retrieval, a 90-day minimum, and a higher retrieval charge. Suits data that is rarely read but must come back immediately when it is.
Glacier Flexible Retrieval and Deep Archive. Cheapest per GB, with retrieval measured in minutes to hours for Flexible and up to twelve hours for Deep Archive, and minimums of 90 and 180 days respectively. A one-business-day retrieval requirement is comfortably satisfied by either, which is the fact that makes the compliance requirement much cheaper than it sounds.
Versioning controls. Noncurrent version expiry, a limit on how many noncurrent versions to keep, and expired object delete marker cleanup. On a bucket with versioning and no expiry rule, this is frequently the largest single saving and it removes nothing anybody is using.
Incomplete multipart upload abortion. A lifecycle rule aborting uploads not completed within a few days. Costs nothing, removes storage nobody knows exists, and is worth applying to every bucket as a default regardless of the rest of the analysis.
Requests and the analytics job. LIST operations on a large prefix are billed per request and are the usual reason a bucket’s request cost exceeds its storage cost. S3 Inventory replaces a nightly LIST with a scheduled report, which is both cheaper and faster.
Evaluation
Side by side
| Lever | Removes | Retrieval cost | Minimum duration | Risk if data does not match |
|---|---|---|---|---|
| Abort incomplete uploads | Invisible storage | n/a | n/a | None |
| Expire noncurrent versions | Storage nobody reads | n/a | n/a | Loses history |
| Intelligent-Tiering | Manual guessing | ✗ between access tiers | n/a | Monitoring fee on small objects |
| Standard-IA | ~45% of storage cost | ✓ per GB | 30 days | Retrieval fees if warm |
| One Zone-IA | More again | ✓ per GB | 30 days | Loss on zone failure |
| Glacier Instant Retrieval | More again | ✓ higher | 90 days | Early-delete charges |
| Glacier Flexible / Deep Archive | Most | ✓ and slow | 90 / 180 days | Retrieval time |
| S3 Inventory instead of LIST | Request cost | n/a | n/a | Report is a day stale |
The first two rows have no downside at all and should be applied before any analysis. Everything below them depends on the access pattern being known, which is why measurement comes first: a transition to Standard-IA on data that turns out to be read weekly costs more than leaving it in Standard.
The solution
Measure with Storage Lens, take the free wins, then transition by measured access pattern rather than by age alone.
Turn on Storage Lens advanced metrics across the estate and let it accumulate. While it does, apply the two rules that need no evidence: abort incomplete multipart uploads after seven days on every bucket, and clean up expired object delete markers. Neither removes anything anyone is using and both are pure configuration.
Then deal with versioning, which on a four-year-old bucket with no expiry rule is likely to be the largest single line. Decide with the compliance team how many noncurrent versions and how long, then set noncurrent version expiry accordingly. The requirement is seven years of retrievability for the customer document, which is not the same as seven years of every intermediate version of it.
With Storage Lens data in hand, split the main bucket by prefix rather than treating it as one thing. Documents uploaded this month behave differently from documents uploaded three years ago, and the transition should follow the observed curve. Where the curve is clear, a lifecycle rule moving objects to Glacier Instant Retrieval and then to Deep Archive is cheaper than Intelligent-Tiering, because there is no per-object monitoring charge. Where access is genuinely unpredictable, Intelligent-Tiering is worth its monitoring fee precisely because it has no retrieval charge between access tiers and therefore cannot punish a wrong guess.
The compliance requirement is the part worth reading carefully. One business day is a generous retrieval window, and Glacier Deep Archive satisfies it comfortably. The seven-year retention is a floor on how long to keep the data, not a statement about how it must be stored, and treating it as a reason to keep 640 TB in Standard is the expensive misreading.
The thumbnail bucket goes to One Zone-IA, because the data is regenerable in six hours and the class exists exactly for that trade. If a zone is lost, the thumbnails are regenerated rather than restored.
The log bucket gets an expiry, which is a conversation rather than a technical decision. Whatever retention the business actually needs becomes a lifecycle rule, and everything past it is deleted rather than archived, because archiving logs nobody will read is paying to defer the decision.
The analytics job stops listing. An S3 Inventory report delivered daily, queried with Athena, replaces the nightly LIST of a large prefix and removes a request charge that had grown with the bucket while nobody attributed it to anything.
Finally, keep the measurement. Storage Lens stays on, per-bucket cost allocation tags go on, and a budget alert on the S3 line means the next quadrupling is a notification.
Why not move everything straight to Deep Archive. It satisfies the compliance requirement and it breaks the product, because recent documents are read constantly and a retrieval measured in hours is not a document management system.
Why not use Intelligent-Tiering everywhere and stop thinking. It is a good default and it charges per object per month to monitor. On a bucket of many small objects that charge is significant, and on data with a predictable cooling curve a lifecycle rule does the same job for less.
Worked example
Storage Lens produces two surprises within a week. Incomplete multipart uploads account for 71 TB, none of which appears in any object listing and all of which has been billed for years. And noncurrent versions account for 218 TB against 640 TB of current objects, because a bucket with versioning and no expiry keeps every version of every re-uploaded document forever.
The multipart rule removes 71 TB immediately. Noncurrent version expiry, set to keep the last three versions for 90 days after agreement with compliance, removes 194 TB over the following fortnight.
The access analysis on what remains shows a clean curve: 91% of reads are against objects uploaded in the last 60 days, and objects older than a year are read at a rate of roughly 0.02% a month. A lifecycle rule moves objects to Glacier Instant Retrieval at 90 days and Deep Archive at 365. Standard shrinks to about 40 TB of genuinely warm data.
The thumbnails move to One Zone-IA, saving less than expected because the objects are small enough that per-object overhead matters, and it is still the right class for regenerable data.
The log bucket turns out to hold four years of debug-level output. The business needs 90 days. Deleting the rest removes 47 TB and a slightly uncomfortable conversation about what else nobody had questioned.
The analytics job’s LIST charges vanish with the Inventory switch, and the job also runs faster because reading a manifest beats paginating through a prefix.
The bill lands at AUD$12,800, a 62% reduction against a 40% target, and about half of it came from data nobody meant to be storing.
What’s worth remembering
- An S3 bill is storage, requests, retrieval and transfer, and a bucket of small objects with a heavy LIST pattern can be dominated by requests, where a cheaper storage class saves nothing.
- Start with the levers that need no analysis: abort incomplete multipart uploads and expire noncurrent versions. Both remove storage nobody can see and neither loses anything in use.
- Retention period and access pattern are different questions; a seven-year requirement with a one-business-day retrieval window is a Deep Archive specification, not a reason to keep everything in Standard.
- The cheaper classes carry minimum storage durations (30 days for IA, 90 for Glacier Instant Retrieval and Flexible, 180 for Deep Archive) and retrieval charges, so a transition applied to warm data increases the bill.
- Intelligent-Tiering is right where the access pattern is unpredictable, because it has no retrieval charge between access tiers; a lifecycle rule is cheaper where the cooling curve is known, and neither suits objects small enough for per-object charges to dominate.
- One Zone-IA is the correct class for regenerable data, because the failure mode of losing a zone is regeneration rather than loss.