When Telemetry Grows Faster Than Traffic

The situation

A travel booking company spends AUD$38,000 a month on CloudWatch across four accounts. Traffic has grown 40% in eighteen months and the observability bill has grown 260%.

The breakdown, once somebody looked: AUD$19,000 of log ingestion, AUD$11,000 of custom metrics, AUD$4,000 of log storage, AUD$2,500 of dashboards and alarms, and the rest scattered. Nobody owns the number, and each team’s contribution is invisible because none of the log groups are tagged.

Three specifics emerged from a first pass. One service emits a log line per database query at INFO level, which is about 60% of the total ingestion. A metrics library publishes a custom metric per endpoint per status code per instance, producing roughly 240,000 unique metrics of which about 400 appear on a dashboard or in an alarm. And debug logging was enabled on two services during an incident in 2025 and never turned off.

What actually matters

The first thing that matters is that the bill has parts with different drivers, and the largest one here is ingestion rather than storage. Ingestion is charged per GB accepted and is a function of what applications emit; storage is a function of retention. Cutting retention reduces the smallest line and leaves the largest untouched, which is why the obvious move disappoints.

The second is that custom metric cost is driven by cardinality rather than volume. A metric published once a minute costs the same as one published constantly; publishing per instance turns one metric into forty. 240,000 metrics of which 400 are used is not a monitoring system, it is a bill.

The third is that nobody owns the number, and a shared cost with no owner grows. Attribution is a prerequisite for anything else, because “reduce observability spend” applied to a platform team is a request to remove other teams’ telemetry, which is a conversation that goes badly.

The fourth is that some of this telemetry is genuinely needed and the answer is to move it rather than delete it. A log that must be retained for compliance can live somewhere cheaper; a metric that is charted monthly does not need to be a custom metric at all.

Underneath it, the two services still emitting debug output are the clearest signal that nothing measures this. A change nobody reversed for two years is a change nothing was watching.

What we’ll filter on

1. Which line of the bill does this reduce: ingestion, storage, metrics, or queries?
2. Is this telemetry used by a dashboard, an alarm, or an investigation?
3. Is the cost driven by volume or by cardinality?
4. Can it move somewhere cheaper rather than being deleted?
5. Who owns it, and can they see what it costs them?
6. What stops it growing back?

The landscape

Log ingestion versus storage. Ingestion is charged per GB accepted into CloudWatch Logs and is the larger number for most estates; storage is per GB-month. Reducing what applications emit reduces both; reducing retention reduces only the second.

Log level and sampling at source. The largest available reduction and the one that needs application changes. Moving per-query logging from INFO to DEBUG, or sampling it at 1%, removes the volume before it is charged. AppConfig feature flags make log level changeable without a deployment, which is what stops the next incident’s debug setting becoming permanent.

CloudWatch Logs Infrequent Access. A cheaper log class with reduced ingestion pricing, supporting Logs Insights with a subset of features. It suits logs that must exist and are rarely read, and it is chosen per log group at creation.

Subscription filters to S3. Route logs to S3 through Firehose for long retention at object-storage prices, with Athena for querying. This moves the storage line rather than the ingestion line, and it is how a compliance retention requirement stops being a CloudWatch bill.

Metric filters versus custom metrics. A metric filter extracts a metric from log data you are already paying to ingest, which costs nothing extra per metric. A custom metric published directly is billed per unique metric. Where the data is already in the logs, the filter is the cheaper route.

Embedded Metric Format. Emits structured logs that CloudWatch converts to metrics at ingestion, so the cost is log ingestion rather than per-metric publishing. It changes the cost model for dimensioned metrics substantially.

Dimension discipline. Every unique combination of dimension values is a separate metric. Publishing per instance is the usual cardinality explosion, and it is rarely needed: aggregate at the service level and rely on Container Insights or logs for per-instance detail.

Retention policies. Per log group, and the setting most often left at Never expire by accident. Cheap to fix and, as above, addressing the smaller line.

Cost allocation tags on log groups. What makes attribution possible. Without them, no team can see their own contribution and the total belongs to nobody.

Contributor Insights and Logs Insights query cost. Queries are billed per GB scanned, so a dashboard running an expensive query on a short refresh interval is its own line item, and it is one people forget exists.

Evaluation

Side by side

Lever	Reduces	Relative size here	Needs app change	Reversible
Lower log level at source	Ingestion and storage	Largest	✓	✓
Sample high-volume logs	Ingestion and storage	Large	✓	✓
Turn off leftover debug	Ingestion and storage	Moderate	Config only	✓
Cut metric cardinality	Custom metrics	Large	✓	✓
Metric filters instead of custom metrics	Custom metrics	Moderate	✓	✓
Logs IA class	Ingestion	Moderate	✗	Per group
Subscription to S3	Storage	Small here	✗	✓
Set retention	Storage	Smallest	✗	✓

The ordering is the finding. Retention is at the bottom and is where most teams start, because it needs no application change and no negotiation. The top three require touching services and account for the majority of the bill.

The solution

Tag for attribution first, then reduce at source in order of size, then move what must be retained, and put a budget on the line so it cannot drift back.

Attribution comes first because everything else is a conversation with a team. Tag every log group and metric namespace with an owning team, and produce a per-team cost report. The report is what converts “the observability bill is too high” into “your service is AUD$11,000 of it”, which is a conversation that produces changes rather than resistance.

Then take the largest reduction: the per-query INFO logging that is 60% of ingestion. Moving it to DEBUG, and sampling it at 1% when enabled, removes roughly AUD$11,000 a month. Put the log level behind an AppConfig feature flag so raising it during an incident is a configuration change with a visible state, rather than a deployment that somebody forgets to reverse.

Turn off the two services still emitting debug from 2025, which is a configuration change and a small embarrassment. Add an alarm on log group ingestion rate per service so the next one is visible within a day.

Then the metrics. 240,000 unique metrics against 400 in use is a cardinality problem, and the cause is the per-instance dimension. Aggregating at the service level takes the count down by roughly the instance count, and per-instance detail moves to Container Insights and logs, where it belongs and where it is already being paid for. Where a metric is derived from data already in the logs, replace the custom metric with a metric filter, which costs nothing per metric.

Move the logs that must be retained but are rarely read. A subscription filter to S3 through Firehose, with a lifecycle policy, satisfies retention requirements at object-storage prices, and Logs IA covers the ones that must stay queryable in CloudWatch.

Set retention on every log group last, because it is the smallest line and it is worth doing anyway. Never expire on a log group is a decision nobody made.

Then stop it growing back. A budget with an alert on the CloudWatch cost line per account, the per-team report circulated monthly, and an alarm on log ingestion rate per log group so a change in emission is visible when it happens rather than in the quarterly bill.

Why not start with retention. It is the easiest change and the smallest line, and starting there produces a modest saving and the impression that the problem has been addressed.

Why not delete the unused metrics. Custom metrics are billed on publication rather than on existence, so the fix is to stop publishing them, which means changing the library’s dimension set rather than deleting anything.

Worked example

Tagging takes a fortnight, most of it spent finding log groups created before the tagging convention existed. The per-team report immediately changes the conversation: one team’s share is 61% of the bill and they had assumed they were a minor contributor.

The log level change on the query logging ships in three weeks, mostly waiting for a release train. Ingestion falls from about 4.2 TB a month to 1.6 TB, and the bill falls AUD$10,800.

The 2025 debug settings are turned off in an afternoon and remove AUD$2,100. The ingestion-rate alarm is added at the same time, and it fires legitimately six weeks later when a team enables verbose logging to debug a problem, which is noticed and reversed the same day rather than in two years.

The metrics library change takes longer, because per-instance dimensions had been used in three dashboards nobody had questioned. Rebuilding those on Container Insights takes a sprint and reduces the metric count from about 240,000 to 6,300. That line falls from AUD$11,000 to AUD$900.

Moving the audit-relevant logs to S3 with a lifecycle policy takes AUD$4,000 of storage to about AUD$400.

The total lands at AUD$12,600, a 67% reduction, and the ratio that matters is that it now grows roughly with traffic rather than at six times traffic’s rate. The budget alert has fired once since, on a legitimate increase during a peak season.

What’s worth remembering

1. Ingestion is usually the largest line and retention the smallest, so cutting retention first produces the least saving for the most satisfying-feeling change.
2. Custom metric cost is driven by cardinality, not volume: a per-instance dimension multiplies every metric by the fleet size, and unused metrics still cost because billing follows publication.
3. Reduce at source before moving or deleting. A log line that is never emitted costs nothing to ingest, store, or query.
4. Where the data is already in the logs, a metric filter produces the metric at no extra per-metric cost, and Embedded Metric Format changes dimensioned metrics from a per-metric charge to log ingestion.
5. Attribution is a prerequisite: without tags nobody owns the number, and “reduce observability spend” aimed at a platform team is a request to delete other people’s telemetry.
6. Put log level behind a runtime flag so an incident’s debug setting has a visible state, and alarm on ingestion rate per log group, or the next one lasts two years.