Paying for Seven Years of Logs

The situation

A wealth management firm is required to retain security-relevant logs for seven years. The compliance wording says the records must be available to a regulator on request, with no stated turnaround, and must be demonstrably unaltered.

Today everything goes to CloudWatch Logs with retention set to Never expire, which is what somebody chose when the platform was small. The log groups now hold 340 TB: CloudTrail management events, VPC flow logs from twelve VPCs, application logs at debug level, ALB access logs, and the output of a security agent on every instance.

The bill is AUD$61,000 a month and rising with the fleet. Two facts about usage: the security team queries the last 30 days constantly, and in four years nobody has queried anything older than 90 days except during one regulator request, which asked for a specific date range and gave six weeks to respond.

What actually matters

The first thing that matters is that “retain for seven years” says nothing about where. It is a statement about existence and integrity, not about which service holds the bytes, and reading it as a requirement to keep everything in the store it landed in is what produces a bill like this one.

The second is that retrieval time is a separate requirement from retention period, and here it is generous. Six weeks to answer a regulator is a specification that permits archival storage with retrieval measured in hours. Where the requirement instead said “within one hour”, the answer would be different and considerably more expensive.

The third is that query frequency decides the store rather than the age of the data. The last 30 days are queried constantly and want to be in something interactive. Everything older is queried approximately never, and paying for it to be instantly queryable is paying for a capability nobody uses.

The fourth is that “demonstrably unaltered” is an integrity requirement, not a storage one, and it is satisfied by specific mechanisms: log file validation, object lock, and restrictive policies. It is worth separating because a design that puts everything in expensive storage on integrity grounds has confused the two.

Underneath it, some of these 340 TB should not be retained at all. Debug-level application logs are not security-relevant records, and retaining them for seven years is applying a compliance requirement to data it does not cover.

What we’ll filter on

1. Is this log actually in scope for the retention requirement?
2. How often is it queried, and at what age?
3. How fast must it come back when it is needed?
4. What proves it has not been altered?
5. What does it cost per GB-month in each candidate store?
6. Who deletes it at seven years, and how?

The landscape

CloudWatch Logs Standard. Interactive, queryable with Logs Insights, priced for ingestion plus storage per GB-month. It is the right store for recent data that gets queried and the wrong one for an archive, because you pay interactive-store prices for data nobody interacts with.

CloudWatch Logs Infrequent Access. A cheaper log class with reduced ingestion pricing, supporting Logs Insights queries with a subset of features. It suits logs that must be queryable but are read rarely, and it is chosen at log group creation, so it is a decision about a stream rather than a lifecycle stage.

Export to S3. CloudWatch Logs can export to S3 on a schedule or via a subscription filter through Firehose, which is the continuous route. Once in S3 the data is subject to lifecycle rules, storage classes and object lock, and it is queryable with Athena rather than Logs Insights.

S3 storage classes for the archive tail. Standard-IA for data read occasionally, Glacier Instant Retrieval for archival with millisecond access, Glacier Flexible Retrieval for minutes-to-hours, and Deep Archive for the cheapest storage with retrieval up to twelve hours. Minimum storage durations of 30, 90, 90 and 180 days respectively, which matters when the lifecycle transitions are close together.

S3 Object Lock. Write-once-read-many retention on objects, in governance mode (overridable with a specific permission) or compliance mode (overridable by nobody, including the root user, until the retention period expires). This is the mechanism that satisfies “demonstrably unaltered” for an archive, and compliance mode is the one a regulator-facing requirement usually wants.

CloudTrail log file validation. Produces digest files allowing verification that trail files have not been modified or deleted since delivery. It is specific to CloudTrail and it is free, and it is the integrity evidence for that particular log source independently of where the files end up.

CloudTrail Lake. A managed event data store for CloudTrail and related events, with SQL querying and retention configurable up to ten years. It removes the export-and-Athena machinery for CloudTrail specifically, at its own per-GB ingestion and retention pricing, and it suits an organisation that queries audit history rather than merely keeping it.

Security Lake. Normalises logs from AWS services, on-premises and third parties into OCSF in an S3 data lake you own, with lifecycle management built in. It is the answer when the requirement is analysis across many sources rather than retention of one.

Athena over S3. Queries the archive in place with no ingestion cost, priced per TB scanned. Partitioning by date and converting to a columnar format is what keeps that number sane, and it is how a six-week regulator request gets answered from Deep Archive after a restore.

Evaluation

Side by side

Store	Query latency	Relative storage cost	Integrity mechanism	Suits
CloudWatch Logs Standard	Seconds	Highest	IAM and resource policies	Last 30 days
CloudWatch Logs IA	Seconds	Lower	IAM and resource policies	Rarely-read but queryable
S3 Standard	Seconds via Athena	Much lower	Object Lock, versioning	30–90 days
S3 Glacier Instant Retrieval	Milliseconds to fetch	Lower again	Object Lock	90 days to a year
S3 Glacier Deep Archive	Up to 12 hours	Lowest	Object Lock	Year one to seven
CloudTrail Lake	Seconds, SQL	Its own pricing	Immutable by design	Audit history queried regularly

The retrieval column against the requirement settles it. Six weeks to answer a regulator makes Deep Archive comfortable for the long tail, and the only reason to hold years of data anywhere more expensive is a retrieval requirement this organisation does not have.

The solution

Scope the requirement first, keep 30 days hot, move the rest to S3 continuously, tier it down to Deep Archive, and lock it.

Scoping comes first because it is the largest single reduction and it costs nothing but a decision. Debug-level application logs are not security-relevant records; establish with compliance which sources are actually in scope, and set ordinary short retention on the rest. Expect this to remove a large fraction of the 340 TB before any storage engineering happens.

For the in-scope sources, set CloudWatch Logs retention to 30 days rather than Never expire, and add a subscription filter through Firehose to S3 so nothing is lost when the log group expires it. The subscription is continuous, which matters: a scheduled export task is a thing that fails silently on a weekend.

In S3, write to a prefix partitioned by source and date, because Athena’s cost is per TB scanned and a partition scheme is the difference between scanning a day and scanning seven years. Convert to a columnar format on ingest if the volume justifies the pipeline.

Then lifecycle it: Standard for 90 days, Glacier Instant Retrieval to a year, Deep Archive to seven years, and expiry at seven years and one day so deletion happens by rule rather than by somebody remembering. Watch the minimum storage durations when setting the transition ages, since a transition earlier than the previous class’s minimum still bills the remainder.

Apply Object Lock in compliance mode with a seven-year retention period, on a bucket with versioning enabled. That is the mechanism that makes “demonstrably unaltered” a property of the store rather than an assertion about access control, and compliance mode specifically means nobody, including the account root, can shorten it.

Keep CloudTrail log file validation on for the trail, because it is free and it gives per-file cryptographic evidence independent of the bucket’s configuration.

For the security team’s daily work, leave the last 30 days in CloudWatch Logs where Logs Insights is fast, and put Athena over the S3 archive for anything older. Where CloudTrail audit history specifically is queried often enough to be annoying through Athena, CloudTrail Lake is worth its pricing for that one source.

Then test a restore. An archive nobody has retrieved from is an archive of unknown quality, and the first time to discover that a Deep Archive restore takes twelve hours and needs a specific IAM permission is not during the regulator’s six weeks.

Why not keep everything in CloudWatch Logs with long retention. It satisfies the requirement and prices seven years of never-read data at interactive-store rates, which is the current AUD$61,000 a month.

Why not go straight to Deep Archive from ingestion. The last 30 days are queried constantly, and a twelve-hour retrieval on data the security team uses daily is not a saving, it is an outage of their working practice.

Worked example

The scoping conversation removes 190 TB. Debug application logs go to 14-day retention, ALB access logs to 90 days in S3 with no archive tail, and the security agent’s verbose output is turned down at source rather than retained. Nobody had asked whether the requirement covered them.

The remaining 150 TB moves to the tiered design over two months. The subscription filters go in first, then retention on the log groups is shortened one source at a time with a week of overlap each, so nothing is expired before it has been seen in the bucket.

The lifecycle transitions produce one surprise: the first month’s bill includes early-deletion charges, because the Standard-to-Glacier-Instant transition at 90 days was set alongside a Glacier-Instant-to-Deep-Archive transition at 120, which is inside Glacier Instant Retrieval’s 90-day minimum. The transitions are re-spaced to 90 and 365.

Object Lock in compliance mode causes the expected moment of nervousness, because it genuinely cannot be undone, and the answer is a test bucket with a one-day retention to prove the behaviour before applying it to the real one.

The restore test takes 11 hours for a 40 GB range from Deep Archive, and finds that the role the security team uses lacks s3:RestoreObject. That is a five-minute fix discovered in a rehearsal rather than during a regulator request.

The bill lands at AUD$9,400 a month. Roughly half the saving came from the scoping conversation and half from the storage design, which is worth remembering the next time the instinct is to start with lifecycle rules.

What’s worth remembering

1. A retention period says how long data must exist, not where it lives; reading it as a requirement to keep everything in the store it landed in is what produces the bill.
2. Retrieval time and retention period are separate requirements, and a generous retrieval window (weeks, not hours) makes Deep Archive viable for the long tail.
3. Query frequency decides the store: the recent window that gets read wants CloudWatch Logs, and years of never-read records want object storage with Athena over it.
4. Scope the requirement before engineering it. Debug logs are usually not security-relevant records, and removing them is the largest single saving available.
5. S3 Object Lock in compliance mode is what makes “demonstrably unaltered” a property of the store rather than a claim about access control, and nobody including root can shorten it.
6. Watch the minimum storage durations when spacing lifecycle transitions, and rehearse a restore, because an archive nobody has retrieved from is of unknown quality.