Cutting the Data Transfer Bill Nobody Can Explain

The situation

A data analytics business in Melbourne spends AUD$29,000 a month on data transfer across seven accounts in one AWS Organization, up from AUD$11,000 eighteen months ago. Nobody can attribute it. The platform team’s working theory is internet egress from the public API, and the invoice does not support that.

What is known about the estate: a shared services account holds a Transit Gateway connecting five workload VPCs, all in ap-southeast-2. Two workload accounts run Kubernetes clusters with pods spread evenly across three Availability Zones. A nightly job copies about 1.2 TB from S3 in ap-southeast-2 to an EC2 fleet in us-east-1 for processing. Application logs from every account ship to a central logging account. A partner pulls a 900 GB extract over the public internet every day. And every VPC reaches S3 and DynamoDB through a NAT gateway, because nobody added endpoints when the VPCs were built.

Finance wants the increase explained before it approves next year’s budget.

What actually matters

Data transfer is not one charge. Internet egress, transfer between Availability Zones, transfer between Regions, NAT gateway processing and Transit Gateway processing are separate usage types at separate rates, and each one responds to a different fix. In ap-southeast-2 the published rates run from USD$0.01/GB on each side of a zone boundary to USD$0.114/GB for the first 10 TB out to the internet, an order of magnitude apart. A single total hides which of them grew.

The expensive paths are the ones nobody chose. Traffic to S3 through a NAT gateway carries USD$0.059/GB of NAT processing in Sydney; the same traffic through a gateway endpoint carries nothing, because gateway endpoints have no hourly or per-GB charge and same-Region transfer to S3 is not billed. Cross-zone chatter between pods is billed on both sides, so a gigabyte between zones costs USD$0.02. Neither path appears in a diagram, because both are consequences of routing.

Attribution across seven accounts is a different exercise from attribution inside one, and the tooling is account-specific in ways worth knowing before starting. Cost Explorer and the Cost and Usage Report cover the whole organisation from the management account, whose primary billing view holds every member account’s cost data. A member account’s primary billing view holds only its own, which is why the workload team’s investigation stalled. Short of handing out management-account access, the way round that is a custom billing view: the management account builds one over a chosen set of accounts and shares it through AWS Resource Access Manager. Cost allocation tags have no equivalent: AWS documents that only a management account, or a standalone account outside an organisation, can reach the cost allocation tags manager, and a newly applied tag key takes up to 24 hours to appear there and up to another 24 to activate.

Some transfer should be paid for, and the live question is which path it takes. The partner’s daily extract has to leave AWS. What is negotiable is whether it leaves as egress billed to the estate, through CloudFront at a rate set by the viewer’s geography, or from a Requester Pays bucket the partner is charged for.

Several of the fixes change more than the bill. A gateway endpoint removes a charge and removes a dependency on the NAT gateway staying up. Zone-local routing removes a charge and drops a network hop. Those are easier to fund than a change that only saves money.

What we’ll filter on

1. Which usage type is this, and at what rate in this Region?
2. Is the traffic necessary, or an artefact of routing?
3. Can it take a cheaper path without changing what the application does?
4. Does the fix remove a dependency as well as a charge?
5. Who benefits from the traffic, and can they be charged for it?
6. Is it reversible if it turns out to hurt latency?

The landscape

Gateway endpoints for S3 and DynamoDB. A route table entry rather than a network interface, and those two services are the only ones that have one. AWS is explicit that there are no data processing or hourly charges for gateway endpoints, so traffic moved off the NAT gateway and onto one stops accruing NAT processing altogether. The NAT gateway’s hourly charge continues: AWS’s pricing note is that once you provision a NAT gateway you are charged for each hour it is available, whatever it processes. The endpoint removes a per-GB line and nothing else until the gateway itself is deleted.

Interface endpoints (PrivateLink). For everything without a gateway option. In ap-southeast-2 they bill USD$0.013 per endpoint-hour in each Availability Zone plus USD$0.01/GB for the first petabyte, against the NAT gateway’s USD$0.059/GB. That is roughly six times cheaper per gigabyte, and the hourly charges decide whether the arithmetic still holds at low volume across many VPCs. Inter-AZ data transfer through an interface endpoint has been free since 1 April 2022. Interface endpoints also reach on-premises networks over VPN or Direct Connect, which gateway endpoints do not, and at organisation scale one endpoint in a shared VPC can serve several accounts instead of one endpoint per VPC. That is often what decides it, rather than the rate, and it is the same primitive a SaaS vendor exposes to its customers.

Availability Zone placement. Cross-zone traffic is billed USD$0.01/GB on each side. A Kubernetes cluster spreading pods evenly across three zones generates that volume by design. The spread is wanted, so the fix is to prefer a zone rather than to pin to one. Topology Aware Routing, enabled per Service with the service.kubernetes.io/topology-mode: Auto annotation, hints endpoints to zones in proportion to each zone’s allocatable CPU. The safeguards matter as much as the mechanism. Below three endpoints per zone there is roughly an even chance the controller cannot allocate evenly. When it cannot, it assigns no hints and routing goes back to cluster-wide. The annotation also cannot be combined with internalTrafficPolicy: Local. AWS records that this mechanism is still in beta, and that its attempt to even out load across zones made behaviour unpredictable. It points instead at Traffic Distribution: trafficDistribution: PreferSameZone on the Service, generally available since Kubernetes 1.33 and spelled PreferClose before 1.35. That routes same-zone first and falls back cluster-wide, with no endpoint threshold and no attempt to balance load, so the zone carrying the traffic is the zone that has to be scaled.

NAT gateway placement. One NAT gateway serving three zones means two zones’ traffic crosses a boundary to reach it and is billed for the crossing on top of NAT processing. AWS’s advice is to keep resources in the gateway’s zone, or to create a gateway in each zone that has resources. Regional availability mode does the second without the bookkeeping: one NAT gateway ID routable from every zone, expanding and contracting with the workload footprint, billed per AZ-hour exactly as separate gateways are, with 32 IP addresses per zone against 8 for a zonal gateway and no public subnet to hold it. Two limits. Expansion into a new zone takes up to 60 minutes, and until it completes that zone’s traffic is processed cross-zone. Regional mode does not support private NAT, so private NAT stays zonal.

CloudFront. CloudFront bills data transfer out by the geography of the edge location serving the viewer rather than by the origin’s Region, and origin fetches from S3, EC2 or an Elastic Load Balancer are free. For Australia and New Zealand the published rate after the always-free first terabyte is USD$0.114/GB for the next 9 TB, which is exactly Sydney’s own first-tier egress rate. The rate advantage shows up when the viewers are somewhere cheaper: US and European edges are USD$0.085/GB. Caching reduces origin fetches, which were already free, so it does not reduce the bytes billed to viewers. CloudFront also sells flat-rate plans, from a free tier through USD$10,000 a month at the top configurable Premium level, which converts a variable line into a fixed one. AWS states plainly that there are no overage charges however far usage runs past the allowance. On substantial excess sustained over two or three months, what happens instead is that traffic may be served from fewer or more distant edge locations until the plan is upgraded. One plan covers one distribution and one apex domain.

S3 Requester Pays. The requester pays for the request and the download; the owner keeps paying for storage. Anonymous access to the bucket stops working, and requesters must send x-amz-request-payer on DELETE, GET, HEAD, POST and PUT. Two edges are worth knowing. The owner is billed for a request that returns AccessDenied when it originated inside the owner’s own account or organisation, and a Glacier restore charges the requester for the request while the owner pays the retrieval.

Moving compute to the data. The nightly copy out of ap-southeast-2 is billed outbound from Sydney at USD$0.098/GB, with inbound to us-east-1 free. Running the analytics where the data already sits removes the charge. Where a dependency pins the compute, S3 Cross-Region Replication moves new and updated objects rather than the whole set, at the same USD$0.098/GB on the bytes it does move, plus replication requests. Live replication does not replicate objects that existed before the rule was set up, and S3 Batch Replication is the on-demand job that seeds those.

Transit Gateway against VPC peering. A Transit Gateway bills USD$0.07 per VPC attachment-hour in Sydney plus USD$0.02/GB processed, and inter-AZ transfer across it has been free since April 2022. Peering has no attachment charge and no processing charge, and is free within a zone, but across zones it bills USD$0.01/GB in each direction, which comes to the same USD$0.02/GB. Peering therefore undercuts a hub only where the traffic stays inside one Availability Zone, and it brings back the mesh a Transit Gateway was built to avoid.

Log volume. Central logging generates cross-account transfer, and flow logs published to S3 add vended-log ingestion and archival charges of their own. Shipping less debug output is the one change that reduces transfer, ingestion, storage and query cost together.

Evaluation

Side by side

Fix	Usage type it removes	Cost of the fix	Also changes	Reversible
S3/DynamoDB gateway endpoints	NAT processing, per GB	Nothing	Removes a NAT dependency	✓
Interface endpoints	NAT processing, per GB	USD$0.013/AZ-hour, USD$0.01/GB	Reaches on-premises	✓
Topology Aware Routing	Cross-AZ, both sides	Three endpoints a zone	One network hop fewer	✓
Regional NAT gateway	Cross-AZ to a single gateway	An AZ-hour per added zone	Port and IP limits	✓
CloudFront in front of egress	Nothing, at Australian rates	Request charges	Latency, TLS, WAF	✓
S3 Requester Pays	Internet egress for that bucket	A commercial conversation	Attribution	✓
Compute in the data’s Region	Cross-Region, outbound	Migration effort	Latency	Hard
Peering instead of Transit Gateway	TGW processing, same-zone only	A mesh to maintain	Nothing	✓
Ship fewer logs	Ingest and cross-account transfer	Engineering	Storage and query cost	✓

The first four rows remove charges created by routing rather than by design, and none of them changes what an application does, which makes them the first work whatever attribution turns up. CloudFront is the row most teams reach for and the only one that removes nothing here, because the Australian edge rate matches the Sydney egress rate to the cent. The cross-Region row is the largest number and the hardest change. So the sequence is settled before the numbers arrive: cheap and certain first, negotiation last.

The solution

Attribute from the management account, take the routing fixes, then negotiate the two expensive ones.

Attribution is the step teams skip and the one that decides everything after it. In Cost Explorer, signed in to the management account, filter to the data transfer usage types and group by usage type, then by linked account. That splits internet egress from cross-zone from cross-Region from NAT and Transit Gateway processing, and it usually contradicts the working theory inside an hour. Where the answer needs finer grain than an account, the Cost and Usage Report carries the usage type and the operation on every line item, and Athena will query it in place. It will not name the resource: AWS documents line_item_resource_id as blank for usage types with no instantiated host behind them, data transfers among them, so the column is empty on exactly the lines in question. Which workloads are talking is a question for VPC Flow Logs delivered to a bucket in the logging account, at vended-log ingestion and archival rates, so scope them to the VPCs under suspicion and turn them off afterwards.

Then the routing fixes, in this order. Gateway endpoints for S3 and DynamoDB in every VPC, with route tables repointed subnet by subnet: nothing to pay, and no application change. Topology Aware Routing on the Services carrying the most cross-zone traffic, after checking that each has three or more endpoints in every zone, because below that the controller is about as likely as not to assign no hints and leave routing cluster-wide. For the Services that cannot meet it, trafficDistribution: PreferSameZone prefers the zone with no endpoint threshold at all. Then NAT gateways in every zone that holds workloads, or one gateway in regional mode, which gives the same zonal affinity without a route table per zone and raises the per-zone IP limit from 8 to 32.

CloudFront is where this scenario departs from the reflex. At Australian list prices the edge rate equals the Sydney egress rate, so putting CloudFront in front of an API served to Australian viewers moves the charge between usage types instead of reducing it. Take it for the latency, the TLS termination and the WAF, take it for viewers on cheaper continents, or take a flat-rate plan to cap the line. Do not book a saving the rate card does not support.

The partner extract is a commercial conversation rather than a technical one. Requester Pays on the extract bucket moves the download onto the account that benefits from it, at the cost of anonymous access and a header on every request, and the negotiation is easier when the alternative on the table is a line item at renewal.

The nightly cross-Region copy moves last, because it needs the analytics workload to run where the data sits. Where it can, the charge goes to zero. Where a dependency in us-east-1 pins it, Cross-Region Replication sends only new and updated objects, seeded once with a Batch Replication job covering everything older than the rule, and the outbound charge then tracks the daily change rather than the full set.

Then make the next increase visible. Tag the resources that generate transfer, activate those tag keys from the management account, and set an AWS Budget on the transfer usage types rather than on the total, so an increase arrives as an alert instead of as a discovery in budget season.

Why not replace the Transit Gateway with peering. Across zones the two cost the same USD$0.02/GB, so the trade is the attachment-hours, about AUD$380 a month across five attachments. Peering five VPCs is ten connections, at least twenty route table entries and no transitive routing, and every VPC added makes it worse. That is a poor trade unless two specific VPCs dominate the traffic and sit in the same zone.

Worked example

Attribution takes two days and reverses the working theory. Internet egress is AUD$5,700 of the AUD$29,000, on about 36 TB a month. The largest line is cross-zone transfer at AUD$9,300, almost all of it the two Kubernetes clusters. NAT processing is AUD$6,200, and flow logs show 71% of those bytes have an S3 prefix as their destination. Cross-Region is AUD$5,400, all of it the nightly copy. Transit Gateway processing is AUD$2,100. The remaining AUD$300 is scattered.

The gateway endpoints go in over a week, one VPC at a time, and take AUD$4,400 off NAT processing, which is the 71%. The change is a route table entry per subnet. The NAT gateways stay, so their hourly charge does not move.

Topology Aware Routing takes cross-zone transfer from AUD$9,300 to AUD$3,100. Two of the eleven Services have too few endpoints per zone to qualify and keep routing cluster-wide, which is visible in the EndpointSlice hints and not in the bill.

Regional NAT gateways then cost slightly more than they save. With 71% of the NAT traffic gone, the cross-zone hop to a shared gateway is worth about AUD$400 a month, and covering three zones across five VPCs adds about AUD$650 in AZ-hours. The change goes ahead for the availability and the port limits; the transfer effect is now incidental. Sequencing did that: run the same comparison before the endpoints and the numbers reverse.

Requester Pays on the extract bucket takes six weeks to agree and removes 27 TB a month. Internet egress falls from AUD$5,700 to about AUD$1,600. The partner accepts because the alternative offered was a line item at renewal.

CloudFront stays on the list and off this quarter’s work. The remaining 9 TB a month would run about AUD$1,600 direct against about AUD$1,400 through CloudFront once the free first terabyte is counted, and the HTTPS request charge at Australian edges is USD$0.0125 per 10,000, which at the API’s 150 million requests a month comes to about AUD$280 against a transfer difference of AUD$200.

The nightly copy survives, because the analytics workload has a dependency in us-east-1 that cannot move this year. Cross-Region Replication replaces the full 1.2 TB copy with about 240 GB of daily change, taking cross-Region from AUD$5,400 to about AUD$1,100, after a one-off Batch Replication job of roughly AUD$180 for the standing set.

The transfer usage types land near AUD$9,600: AUD$1,600 internet egress, AUD$2,700 cross-zone, AUD$1,800 NAT processing, AUD$1,100 cross-Region, AUD$2,100 Transit Gateway and AUD$300 scattered. The new NAT gateway AZ-hours add about AUD$650 on a line that is not data transfer at all, so the monthly total moves from AUD$29,000 to about AUD$10,250. The Budget on the transfer usage types means the next increase surfaces in the month it happens.

What’s worth remembering

1. Split the total first. Rates run from USD$0.01/GB per zone side to USD$0.114/GB internet egress from Sydney; a total hides which usage type grew.
2. Attribute from the management account. Only it reaches the cost allocation tags manager; a member account needs a custom billing view shared to it.
3. NAT hourly charge survives gateway endpoints. Free S3 and DynamoDB gateway endpoints remove NAT processing; the gateway’s hourly charge runs until it is deleted.
4. Cross-zone traffic bills both sides. Topology Aware Routing wants three or more endpoints per zone; trafficDistribution: PreferSameZone prefers the zone with no threshold.
5. CloudFront bills by viewer geography. Australian viewers pay Sydney’s own rate, so nothing is saved there; caching only cuts origin fetches that were already free.
6. Replication saves bytes, not rate. Cross-Region Replication costs the same USD$0.098/GB outbound as the copy; objects older than the rule need a Batch Replication job.