Cutting the Data Transfer Bill Nobody Can Explain

The situation

A data analytics business spends AUD$29,000 a month on data transfer across seven accounts, up from AUD$11,000 eighteen months ago. Nobody can attribute it. The architecture has not changed in any way anyone connects to the increase, and the platform team’s working theory is that it is “just the internet egress from the API”, which the invoice does not support.

What is known about the estate: a shared services account holds a Transit Gateway connecting five workload VPCs. Two workload accounts run Kubernetes clusters with pods spread across three Availability Zones. A nightly job copies about 4 TB from S3 in ap-southeast-2 to an EC2 fleet in us-east-1 for processing. Application logs from every account ship to a central logging account. A partner pulls a daily 900 GB extract over the public internet. And every VPC reaches S3 and DynamoDB through a NAT gateway, because nobody added endpoints when the VPCs were built.

Finance wants the increase explained before it approves next year’s budget.

What actually matters

The first thing that matters is that data transfer is not one charge. Egress to the internet, transfer between Availability Zones, transfer between regions, transfer through a NAT gateway, and transfer through a Transit Gateway attachment are separate usage types at separate rates, and they respond to completely different fixes. Treating the total as a single number is what makes it unattributable.

The second is that the most expensive paths are usually the ones nobody chose. Traffic to S3 through a NAT gateway is billed for NAT processing on top of everything else, and a gateway endpoint makes it free. Cross-zone chatter between pods that could have been zone-local is billed in both directions. Neither appears in a diagram, because both are consequences of routing rather than of design.

The third is that attribution has to come before optimisation, and the tooling for it is specific. Cost Explorer with usage-type grouping separates the charges; VPC Flow Logs with the right fields identify which resources are talking; and the Cost and Usage Report is where per-resource detail lives. Guessing which of the six candidate causes is dominant wastes the effort on the wrong one.

The fourth is that some transfer is worth paying for and the question is only whether it should take a cheaper path. The partner’s 900 GB extract has to leave AWS. What is negotiable is whether it leaves through EC2 egress at the standard rate, through CloudFront at a lower one, or through S3 with requester-pays so the partner covers it.

Underneath it, several of the fixes reduce the bill and improve something else at the same time. A gateway endpoint removes cost and removes a dependency on the NAT gateway. Zone-aware routing removes cost and reduces latency. Those are easier to fund than a change that only saves money.

What we’ll filter on

1. Which usage type is this, and at what rate?
2. Is the traffic necessary at all, or an artefact of routing?
3. Can it take a cheaper path without changing what the application does?
4. Does the fix reduce a dependency as well as a cost?
5. Who is the traffic for, and could they pay for it?
6. Is it reversible if it turns out to hurt latency?

The landscape

Gateway endpoints for S3 and DynamoDB. A route table entry rather than a network interface. They cost nothing, and traffic to S3 or DynamoDB through them skips the NAT gateway entirely, removing both the per-GB NAT processing charge and the NAT’s hourly share of the work. For an estate reaching S3 through NAT, this is usually the single largest saving available and it takes a route table change per subnet.

Interface endpoints (PrivateLink). For services without a gateway option. They bill hourly per endpoint per zone plus per GB processed, so they are not free, and the arithmetic is against the NAT gateway charge for the same traffic. They also reach on-premises networks over VPN or Direct Connect, which gateway endpoints do not, and that is often the deciding factor rather than the price.

Availability Zone placement. Traffic between zones is billed in both directions. A service mesh or a Kubernetes cluster spreading pods evenly across three zones generates a great deal of it by design. Topology-aware routing keeps traffic zone-local when a healthy endpoint exists in the same zone, falling back across zones only when it does not, which preserves the availability property while removing most of the charge.

NAT gateway placement. One NAT gateway shared across zones means every private subnet in the other zones pays cross-zone charges to reach it, on top of NAT processing. One gateway per zone with zone-local routes removes the cross-zone hop. It costs an extra hourly charge per gateway and usually pays for itself on the transfer line alone.

CloudFront for egress. Data transfer out through CloudFront is charged at a lower rate than direct egress from EC2 or S3, and transfer from AWS origins into CloudFront is free. For any workload serving substantial volume to the internet, putting CloudFront in front reduces the rate and adds caching, which reduces the volume as well.

S3 requester-pays. Shifts the transfer and request costs of a bucket to the account making the request. Suited exactly to a partner pulling a daily extract: they are the beneficiary, they can be given access, and the cost lands with them. It requires the requester to authenticate and to acknowledge the charge, so it is a commercial conversation as much as a technical setting.

Moving compute to the data. The 4 TB nightly copy from ap-southeast-2 to us-east-1 is a cross-region charge paid every night to put data next to compute. Running the compute in the region where the data already lives eliminates it entirely. Where that is impossible, S3 Cross-Region Replication moves only the changes rather than the whole set, and moves them once.

Transit Gateway attachment charges. A Transit Gateway bills per attachment-hour and per GB processed. Traffic between two VPCs that talk constantly may be cheaper over VPC peering, which has no per-GB processing charge, at the cost of the mesh that a Transit Gateway exists to avoid. This is a real trade rather than a free win.

Log volume itself. Central logging generates transfer, and much of it is debug output nobody reads. Reducing what is shipped is the only fix that reduces the storage and ingestion charges at the same time.

Evaluation

Side by side

Fix	Usage type it removes	Cost of the fix	Also improves	Reversible
S3/DynamoDB gateway endpoints	NAT processing	Nothing	Removes a NAT dependency	✓
Interface endpoints	NAT processing	Hourly + per GB	Reaches on-premises	✓
Zone-aware routing	Cross-AZ, both directions	Config	Latency	✓
NAT gateway per zone	Cross-AZ to the gateway	Hourly per gateway	Removes a single point of failure	✓
CloudFront in front of egress	Internet egress rate	Per-request charges	Caching, latency, WAF	✓
S3 requester-pays	Internet egress for that bucket	A conversation	Attribution	✓
Move compute to the data	Cross-region	Migration effort	Latency	Hard
Peering instead of Transit Gateway	TGW per-GB processing	Mesh complexity	Latency	✓
Ship fewer logs	Ingest and transfer	Engineering	Storage and query cost	✓

Two things stand out. The top four rows cost almost nothing and remove charges generated by routing rather than by design, which makes them the first work regardless of what attribution finds. And the cross-region row is the largest single number here while also being the hardest change, which is the usual shape: the biggest saving needs the most negotiation.

The solution

Attribute first with Cost Explorer and flow logs, then take the free routing wins, then negotiate the expensive ones.

Attribution is the step teams skip and the one that decides everything else. In Cost Explorer, filter to the data transfer usage types and group by usage type and then by linked account. This separates internet egress from cross-AZ from cross-region from NAT processing, and it will usually contradict the working theory within an hour. Where per-resource detail is needed, the Cost and Usage Report carries it; where the question is which resources are actually talking, VPC Flow Logs published to S3 and queried with Athena will answer it, with the caveat that flow logs are themselves a cost and should be scoped and time-boxed.

Then take the routing wins, in this order. Add gateway endpoints for S3 and DynamoDB in every VPC and repoint the route tables; it costs nothing, removes NAT processing charges on that traffic, and reduces how much depends on the NAT gateway staying up. Add a NAT gateway per zone with zone-local routes, which removes the cross-zone hop to a shared gateway and removes a single point of failure at the same time. Turn on topology-aware routing in the Kubernetes clusters so pod-to-pod traffic prefers a same-zone endpoint, keeping the cross-zone fallback for when there isn’t one.

Put CloudFront in front of the public API and the partner extract. Egress through CloudFront is billed at a lower rate than direct egress, transfer from the origin into CloudFront is free, and the caching reduces the volume as well as the rate.

The partner extract then gets a commercial conversation rather than a technical one. Requester-pays on the extract bucket moves 900 GB a day of egress onto the account that benefits from it, and the negotiation is easier than it sounds when the alternative is the charge appearing in their contract renewal instead.

The nightly cross-region copy is the biggest number and the last to move, because it needs the analytics workload to run where the data is. Where that is possible, the charge disappears. Where it is not, replace the full 4 TB copy with S3 Cross-Region Replication so only changed objects cross the boundary, once, rather than the whole set nightly.

Finally, make it stick. Data transfer grew because nothing watched it. Tag the resources that generate it, put the usage types on a Cost Explorer report, and set an AWS Budget with an alert on the transfer line specifically rather than on the total, so the next increase is a notification rather than a discovery during budget season.

Why not start with CloudFront. It is the fix most people reach for and it addresses internet egress, which the invoice may show is not the largest component. Attribution first is what stops the effort landing on the wrong usage type.

Why not replace the Transit Gateway with peering. The per-GB processing charge is real and so is the reason the Transit Gateway is there. Peering five VPCs is ten connections and no transitive routing, and unpicking that to save a per-GB charge is a poor trade unless two specific VPCs dominate the traffic.

Worked example

Attribution takes two days and reverses the working theory. Internet egress is AUD$4,100 of the AUD$29,000. The largest single component is cross-AZ transfer at AUD$9,800, almost all of it from the two Kubernetes clusters. NAT processing is AUD$6,200, and flow logs show 71% of that traffic has an S3 prefix as its destination. Cross-region is AUD$5,400, all of it the nightly copy. The Transit Gateway accounts for AUD$2,300 and the rest is scattered.

The gateway endpoints go in over a week, one VPC at a time, and remove AUD$4,400 a month. The change is a route table entry and nothing in any application notices.

Topology-aware routing in the clusters takes cross-AZ from AUD$9,800 to AUD$3,100. Median service-to-service latency also falls by about 40%, which is the outcome the engineering team cares about and the reason the change gets prioritised at all.

NAT gateways per zone add AUD$390 a month in hourly charges and remove about AUD$900 in cross-zone transfer, which is a modest net win on cost and a real win on availability.

CloudFront in front of the public API takes internet egress from AUD$4,100 to AUD$2,300, with a cache hit ratio of 61% on a workload nobody thought was cacheable.

The partner conversation takes six weeks and ends with requester-pays enabled, moving 900 GB a day off the bill. The partner accepts because the alternative offered was a line item in the renewal.

The cross-region copy survives, because the analytics workload has a dependency in us-east-1 that cannot move this year. Switching from a nightly full copy to Cross-Region Replication takes it from 4 TB a night to about 240 GB of daily change, removing roughly AUD$4,800.

The bill lands near AUD$11,500, and the budget alert on the transfer usage types means the next increase gets noticed in the month it happens.

What’s worth remembering

1. Data transfer is several charges at several rates (internet egress, cross-AZ, cross-region, NAT processing, Transit Gateway processing) and each has a different fix, so the total is unattributable until it is split by usage type.
2. Attribute before optimising: Cost Explorer grouped by usage type takes an hour and routinely contradicts the team’s theory about where the traffic goes.
3. Gateway endpoints for S3 and DynamoDB cost nothing, remove NAT processing charges, and remove a dependency on the NAT gateway; in an estate without them this is usually the largest free saving available.
4. Cross-AZ traffic is billed in both directions, and an evenly-spread cluster generates a lot of it by design; topology-aware routing keeps it zone-local without giving up the cross-zone fallback.
5. Egress through CloudFront is billed at a lower rate than direct egress and transfer from AWS origins into CloudFront is free, so it lowers the rate and the volume together.
6. Some transfer should be paid for by whoever benefits: S3 requester-pays moves a partner’s extract onto the partner’s account, and the conversation is easier than the engineering alternative.