The situation
A ticketing platform ran a 90-minute incident where roughly 4% of checkouts failed. The dashboards showed a green service: average latency normal, error rate at 0.4%, CPU and memory unremarkable, every health check passing.
The 4% turned out to be customers in one region hitting one of six payment providers, through one of forty task instances that had a stale credential cached. Every one of those dimensions was invisible: the metrics were service-wide aggregates, the logs were unstructured strings, and there was no way to follow one failing request from the load balancer to the payment call.
Diagnosis took 90 minutes and happened by someone eventually noticing a pattern in the support tickets. The requirement is that the next one takes minutes, without tripling the observability bill.
What actually matters
The first thing that matters is that an average hides exactly the failures that matter. A service-wide error rate of 0.4% is 100% for the customers affected, and no aggregate will show it. Being able to slice by the dimensions a failure follows, region, provider, instance, customer tier, is what turns a flat line into a signal.
The second is that the dimensions have to be chosen in advance, because you cannot slice by something you did not record. That is a design decision about which attributes matter, and it is bounded by cost, since every dimension multiplies the cardinality of what is stored.
The third is that joining across services requires something that survives the hop. A trace identifier propagated through every call, and included in every log line, is what makes “show me everything that happened to this request” a query rather than a reconstruction.
The fourth is that unstructured logs cannot be queried by dimension. A log line that concatenates a message with values cannot be filtered on those values without regular expressions that break when the message changes. Structured logging is the cheapest large improvement available here.
Underneath it, the health check passing while the service was failing for some customers is its own defect. A health check answering “is the process up” is a different question from “is this instance serving correctly”.
What we’ll filter on
- Can this signal be sliced by the dimensions a failure follows?
- Does it join to other services’ signals?
- Is the cost proportional to traffic, or to cardinality?
- Does it answer a question during an incident, or only reassure between them?
- Is it available in seconds, or after a pipeline?
- What does it cost to add one more dimension?
The landscape
Structured logging. Emitting logs as JSON with named fields rather than concatenated strings. CloudWatch Logs Insights can then query on fields directly, and a filter on provider = "stripe" and region = "ap-southeast-2" is a query rather than a regular expression. It costs nothing beyond the change and it is usually the largest single improvement.
Embedded Metric Format. A log format that CloudWatch extracts custom metrics from at ingestion, so a service emits one structured log line and gets both the log record and the metric with its dimensions. This is the mechanism for high-cardinality-ish metrics without a separate PutMetricData call per dimension combination, and the cost model is log ingestion rather than per-metric.
Custom metrics with dimensions. Metrics published with dimension sets, billed per unique combination. Precise and expensive at high cardinality, since every combination is a separate metric. Right for a bounded dimension set (six providers, four regions) and wrong for anything unbounded like customer identifier.
AWS X-Ray. Distributed tracing: a trace identifier propagated across services, with segments and subsegments showing where time went and which call failed. The service map is the visual, and the useful part during an incident is filtering traces by annotation to find the failing subset. Sampling is configurable, and the default sampling means a rare failure may not be captured, which is worth setting deliberately.
Trace annotations and metadata. Annotations are indexed and filterable; metadata is stored and not indexed. Putting the dimensions that matter (provider, region, tier) into annotations is what makes “show me traces where provider is X and the result was an error” possible.
Application Signals. Automatic service-level monitoring built on OpenTelemetry, producing standard latency and error metrics per service and operation, with service level objectives on top. It removes a chunk of the manual instrumentation for the common signals.
Correlation identifiers. A request identifier generated at the edge and propagated, included in every log line. Where full tracing is not in place, this alone makes cross-service investigation possible, and it is a header and a logging convention rather than a service.
Container Insights and enhanced observability. Per-task and per-container metrics for ECS and EKS, which is what would have shown one task behaving differently from the other thirty-nine.
Health check design. Liveness versus readiness, and a readiness check that exercises the dependencies the instance needs, so an instance with a stale credential fails readiness rather than passing a check that only proves the process is running.
Evaluation
Side by side
| Signal | Sliceable | Joins across services | Cost driver | Would have found it |
|---|---|---|---|---|
| Service-wide metrics | ✗ | ✗ | Per metric | ✗ |
| Metrics with bounded dimensions | ✓ | ✗ | Per combination | ✓ region and provider |
| Embedded Metric Format | ✓ | Via fields | Log ingestion | ✓ |
| Structured logs | ✓ | Via correlation id | Log ingestion | ✓ |
| X-Ray traces with annotations | ✓ | ✓ | Per trace, sampled | ✓ all four dimensions |
| Container Insights | Per task | ✗ | Per metric | ✓ the one bad task |
| Readiness check on dependencies | n/a | n/a | Free | Would have prevented it |
The last row matters most: the cheapest intervention on the page is not observability at all. An instance whose credential is stale failing readiness would have been removed from the load balancer and the incident would not have happened.
The solution
Structured logs with a correlation identifier first, then dimensioned metrics on a bounded set, then tracing with annotations, and fix the readiness check.
Start with structured logging and a correlation identifier, because together they cost almost nothing and they convert the log estate from prose into a queryable dataset. Generate the identifier at the load balancer or the edge service, propagate it as a header, and include it plus the request’s key attributes (region, provider, customer tier, task identifier) in every log line as named fields. After this change, the 90-minute diagnosis is a Logs Insights query that groups errors by provider and region.
Add dimensioned metrics for the bounded dimensions using Embedded Metric Format, so the same structured log line produces the metrics without a separate publishing call. Six providers, four regions and three tiers is 72 combinations, which is affordable; customer identifier is not and belongs in logs and traces rather than in metrics. Being explicit about which dimensions go where is what keeps the bill proportionate.
Then enable X-Ray with annotations for the same dimensions, and set the sampling rule deliberately rather than accepting the default. A 4% failure rate under default sampling may produce very few captured failing traces, so a rule that samples errors at a much higher rate than successes is what makes tracing useful for rare failures. The trace is what answers “where in the chain did this go wrong”, which the logs and metrics cannot.
Turn on Container Insights so per-task metrics exist. The incident involved one task out of forty behaving differently, and no service-level aggregate will ever show that.
Then fix the readiness check, which is the cheapest and most valuable change. A check that verifies the instance can reach its dependencies with its current credentials, rather than one that returns 200 if the process is alive, would have taken the bad task out of rotation automatically. Keep liveness separate so a dependency blip does not cause a restart loop.
Finally, write the query. An incident-response runbook containing the three or four Logs Insights queries that slice errors by each dimension turns the investigation from an exercise in inspiration into a checklist. The instrumentation is only useful if somebody knows what to ask it.
Why not add more dashboards. The dashboards were green and accurate. Adding more views of the same aggregates would have shown the same reassuring picture, because the problem was the aggregation rather than the presentation.
Why not put every dimension into metrics. Customer identifier as a metric dimension produces a metric per customer, which is a bill that grows with the customer base and a set of metrics nobody can chart. High-cardinality dimensions belong in logs and traces.
Worked example
The structured logging change ships in two weeks across nine services, mostly waiting for the shared library version to propagate. The correlation identifier is generated at the ALB using its request identifier, which removes the need to invent one.
The first test of the new instrumentation is unplanned, eleven days later: a 1.2% error rate appears. The query that groups errors by provider and region returns an answer in under a minute, one provider in one region, and the investigation moves straight to that provider’s status page, which is showing an incident. Total time from alert to cause: four minutes.
The EMF metrics produce a surprise in the first week. One customer tier has a consistently higher error rate than the others, at about 0.9% against 0.2%, and it has been that way for months without anybody knowing, because the service-wide average absorbed it. That becomes a separate ticket and turns out to be a validation difference on that tier’s payload.
X-Ray sampling takes two attempts. The first rule samples everything at 5%, which captures almost no failures. The second samples successes at 1% and anything with a fault at 100%, which is a large improvement in usefulness and a reduction in volume.
The readiness check change is the one that gets pushback, because an instance failing readiness on a dependency blip is a capacity reduction. The compromise is a readiness check that verifies credential validity and a dependency reachability check with a generous threshold, which catches the stale-credential case without flapping on latency.
The observability bill rises about 18%, almost all of it log ingestion, and the same quarter’s incident minutes fall by roughly two thirds.
What’s worth remembering
- An average hides the failures that matter: 0.4% service-wide is 100% for the affected customers, and no aggregate will show it. Sliceability is the property to design for.
- You cannot slice by a dimension you did not record, so choosing which attributes to attach is a design decision made before the incident.
- Structured logging plus a correlation identifier propagated across services is the cheapest large improvement, converting the log estate from prose into a queryable dataset.
- Put bounded dimensions in metrics (Embedded Metric Format gets both from one log line) and unbounded ones in logs and traces, because metric cost scales with unique dimension combinations.
- Set X-Ray sampling deliberately: default sampling may capture almost none of a rare failure, and sampling faults at a much higher rate than successes is what makes tracing useful during an incident.
- The cheapest fix here is not observability. A readiness check that exercises the instance’s dependencies would have removed the bad task automatically, and a check that only proves the process is alive will pass throughout an outage.