Four Hundred VPCs and a Route Table Limit

The situation

One AWS Organization, 412 member accounts, and a rule written into the account-vending pipeline four years ago: every account gets a VPC. Four hundred of them now have one in ap-southeast-2, each with a /20 cut from 10.0.0.0/8 and handed out in account-creation order from a spreadsheet. Product lines are scattered across the range, so nothing summarises.

Three hundred and four of those four hundred accounts hold a single application with exactly two dependencies: an internal identity API over HTTPS, and a shared metadata store speaking the PostgreSQL wire protocol on 5432. They talk to nothing else east-west, and between all of them they push about 6 TB a month to those two services. The remaining ninety-six are platform and environment VPCs across five product groups, which call each other in every direction, at around 40 TB a month.

Connectivity is VPC peering, added a pair at a time by whoever needed it. The shared-services VPC that fronts identity now holds 123 active peering connections; the one fronting the metadata store holds 118. Both are against a ceiling of 125 that no quota increase moves. The identity service has already been cloned into a second VPC to make room, and a config change to it now ships twice. Three of the platform VPCs have route tables at 500 entries, and the next route added to any of them comes back with RouteLimitExceeded. The networking team has a ticket asking for a transit gateway and a diagram that assumes four hundred attachments.

What actually matters

Start by sorting the quotas into the ones that move and the ones that do not, because a growth plan that ends in a support case depends on which case gets granted. Routes per route table defaults to 500 and can be raised to 1,000, with AWS attaching a plain warning that network performance might suffer at that size. Active peering connections per VPC defaults to 50 and stops at 125 whatever you ask for. Attachments per transit gateway defaults to 5,000 and is adjustable. A design whose next hundred accounts run into the adjustable quota can be extended with a support case; a design whose next hundred accounts run into the hard one has no next hundred accounts. There is a smaller trap in the same arithmetic. A managed prefix list referenced from a route table counts its maximum entry count against that table, not the number of prefixes currently in it. The on-premises list sized at 100 and holding 61 takes 100 route slots in every table that references it.

The second question is who is allowed to break reachability. Four hundred VPCs means four hundred route tables in four hundred accounts. Every one of them is editable by whoever has administrator rights in that account, which for a single-application account is usually the two people who ship the application. Nobody set out to give three hundred teams a routing domain each. It arrived as a side effect of a rule about accounts. Whether an application team should own a route table at all is a separable question from whether it should own an account, and separating them is what makes the topology governable rather than merely smaller.

The third is which way the bill grows. Peering has no hourly charge and bills only for data crossing an Availability Zone boundary, at a cent a gigabyte each way, so its cost tracks traffic. Hub and endpoint shapes invert that. At the US rates AWS uses in its own pricing examples, a transit gateway is five cents an hour for every attachment plus two cents a gigabyte processed. An interface endpoint is a cent per Availability Zone per hour plus tiered data processing from a cent a gigabyte. Sydney’s figures differ; the shape of the charge does not. Four hundred attachments is about USD$14,600 a month before a packet moves, and 304 VPCs each holding two interface endpoints across three zones is about USD$13,300 a month for six terabytes of traffic. Both numbers are driven by the count of VPCs rather than by anything a user did. Reduce the count and both fall.

The fourth is what those 304 applications actually need from the network. They need account-level isolation, because that is where IAM boundaries, billing separation and blast radius live. They need to reach two services in one direction. They do not need a routing domain, an internet gateway, a NAT gateway, a DHCP option set or a /20. Those arrived in the same bundle, and the bundle is the thing worth taking apart. A one-way service call has a simpler and safer shape than a routed network, and an account keeps its IAM and billing isolation while running in subnets a network account owns.

What we’ll filter on

1. Survives the next four hundred accounts without the growth path running into a quota that cannot be raised.
2. Keeps the number of separately-owned route tables small, and puts them in an account whose owner is on call for routing.
3. Cost grows with bytes moved rather than with the number of accounts.
4. Carries a plain TCP protocol, not only HTTP and TLS.
5. Gives a one-way path where the traffic only goes one way, so a shared service cannot open a connection back into a caller.
6. Slows down consumption of the address plan instead of using a /20 per new account.

The landscape

Five shapes are available for connecting accounts inside one Organization, and they are not alternatives so much as different layers.

A full peering mesh is the incumbent and fails on the arithmetic. Four hundred VPCs fully meshed is 400 x 399 / 2, or 79,800 connections, and each VPC would need 399 of them against a hard 125. Peering is also non-transitive, so no amount of partial meshing lets A reach C through B; every path is its own connection and its own pair of route entries. There is no hourly charge and a cent a gigabyte across Availability Zones, which is why nobody noticed the shape was wrong until the quota was reached.

A transit gateway replaces N x (N-1) / 2 edges with N attachments and one routing domain. It takes 5,000 attachments by default, 20 route tables, and 10,000 routes combined across all of them, which is where the per-VPC prefixes go once they leave the VPC route tables. Route propagation means attaching a VPC installs its CIDR without anyone editing a table. Separate route tables give segmentation: a production table that carries production and shared services, a non-production table that carries neither, and no path between them. It supports an MTU of 8,500 bytes between VPCs, gives up to 100 Gbps each way per VPC attachment per zone, and charges per attachment-hour plus per gigabyte processed.

PrivateLink endpoint services publish one service rather than joining two networks. The provider puts a Network Load Balancer in front of the service and publishes it; each consumer creates an interface endpoint whose network interfaces live in the consumer’s own subnets. Traffic only flows in the direction the consumer opens, there are no routes, and the two sides may have identical CIDRs because nothing is being routed. It carries anything TCP, so the metadata store on 5432 is fine. The consumer VPC is capped at 50 interface and Gateway Load Balancer endpoints combined, each endpoint does 10 Gbps per zone scaling to 100, and the service sees the load balancer node addresses rather than the caller’s unless proxy protocol v2 is turned on.

VPC Lattice sits above both, connecting named services rather than networks. A service network is associated with VPCs, up to 500 of them, and up to 500 services; a VPC associates directly with exactly one service network, with service-network VPC endpoints available for more. Callers reach a service by name with IAM auth policies in front of it, without routes and regardless of overlapping addresses. A service’s listeners speak HTTP, HTTPS and TLS, which covers the identity API. The metadata store’s 5432 leg needs the other half of Lattice: a resource configuration, which is TCP only, points at an IP address, a domain name or an RDS database by ARN, and is reached through a resource gateway in the provider VPC. Both halves charge by the hour plus data processing, and neither removes a VPC.

RAM-shared subnets are the option that deletes the problem rather than routing around it. The owner account keeps a VPC and shares individual subnets, through AWS Resource Access Manager, with other accounts in the same Organization. Participants launch and manage their own EC2 instances, RDS databases, Lambda functions, network interfaces and security groups inside those subnets. They cannot change the VPC, the subnets, the route tables, the NACLs, the internet gateway or the NAT gateways, and they cannot see each other’s resources. A VPC can be shared with 100 participant accounts by default, adjustable, and an account can have 100 subnets shared with it. The prerequisites are AWS Organizations, resource sharing enabled from the management account, and a resource share. Default VPCs cannot be shared.

Evaluation

Side by side

Option	Survives 400+ accounts	Few owned route tables	Cost by bytes, not accounts	Carries plain TCP	One-way shape	Saves address space
Full peering mesh	✗	✗	✓	✓	✗	✗
Transit gateway	✓	✓	✗	✓	✗	✗
PrivateLink endpoint service	✓	✓	✗	✓	✓	✗
VPC Lattice	✓	✓	✗	✓	✓	✗
RAM-shared subnets	✓	✓	✓	✓	✗	✓

No row is all ticks, and the shape of the crosses says how to combine them. Shared subnets are the only row that shrinks the address plan and the only one whose running cost does not multiply by account count. Everything inside one VPC can reach everything else, though, so it is the wrong container for workloads that need to be kept apart. A transit gateway is the only row that gives segmented any-to-any at this size. PrivateLink and Lattice tick the same boxes, since a Lattice resource configuration carries plain TCP as well. Lattice loses on what it adds: a service network, auth policies and per-service hourly charges the estate uses nowhere else, for two services that already sit behind load balancers and need one direction and one port each.

Choosing a shape per workload

The solution

Four shared VPCs, owned by a network account, take the 304 single-application accounts as participants: 96 in production, 96 in non-production, 72 in sandbox, 40 in the regulated tier. Every one of those sits under the default quota of 100 participant accounts per VPC, so the quota increase stays available as headroom rather than being the plan. Each VPC has three subnets per tier across three Availability Zones. The subnets are shared through a resource share scoped to the organizational unit, so a new account inherits the share on the day it joins the OU instead of on the day someone remembers.

The split between owner and participant is worth stating exactly, because half the objections to shared VPCs come from assuming it is more, or less, than it is. A participant creates and manages its own network interfaces, security groups, EC2 instances, RDS databases and Lambda functions in the shared subnets, and those resources count against the participant’s own quotas, not the owner’s. A participant can write a security group rule that references a group belonging to another participant or to the owner, in the account-number/security-group-id form, so identity-based rules survive the move off per-account VPCs. A participant cannot create, modify or delete the VPC, its subnets, its route tables or its NACLs; cannot create a NAT gateway or attach an internet gateway; and cannot see, attach or modify another participant’s network interfaces. It also cannot launch into the VPC’s default security group, because that one belongs to the owner. Where a common ruleset is wanted, the owner shares a named security group through RAM and manages it centrally. One quota subtlety comes with that: an interface carrying any shared security group is limited to the lower of the owner’s and the participant’s security-groups-per-interface quota.

Placement has to be discussed in Availability Zone IDs. AWS maps physical zones to zone names randomly per account, so ap-southeast-2a in the network account and ap-southeast-2a in a participant account are often different physical zones. Every shared subnet carries an AZ ID such as apse2-az1, and that identifier means the same physical zone in every account in the Organization. Any conversation about zonal spread, any Terraform variable, and any cross-zone cost estimate uses the ID. Getting it right has a direct consequence: data transfer between resources in the same zone, identified by AZ ID, is free regardless of which accounts own them, and cross-zone transfer is not.

Billing follows what each account runs. Participants pay for their own instances, databases and functions, and for their own cross-zone data transfer, transfer over peering connections, transfer through internet gateways and transfer across Direct Connect gateways. The owner pays the hourly and processing charges for the NAT gateways, transit gateway, PrivateLink endpoints and any public IPv4 addresses in the VPC. That is a real transfer of cost onto the network account, and it needs a showback split, or the network account ends up carrying the organisation’s egress bill with no way to attribute it.

Observability lands on the same split and is better for it. The owner takes one flow log at the subnet level, which captures every participant interface in that subnet, and the platform gets a single place to answer reachability questions instead of 304. Participants can still create flow logs for the interfaces they own; the owner cannot read those, and cannot delete them.

The ninety-six platform VPCs keep their own VPC and take a transit gateway attachment, and so do the four shared VPCs, for a hundred attachments in total. Three transit gateway route tables carry the segmentation the peering mesh never expressed: production, non-production, and shared services, with propagation on so an attachment installs its own prefix and nobody edits a route by hand. The VPC route tables shrink to a handful of entries each, since everything non-local now has one next hop, and the per-VPC prefixes live in the transit gateway’s route tables against a 10,000-route budget rather than a 500-route one. Resize the on-premises prefix list to its actual contents plus headroom while you are in there, because its maximum, not its length, is what counts against each referencing route table. One consequence to plan for rather than discover: only the VPC owner can attach a transit gateway to a shared subnet, so the four attachments belong to the network account, which is where they should be anyway. Keeping propagation matters whenever the topology changes, the same test that separates the cross-Region options that still propagate routes from the ones that hand back a static table. The mesh gave that up years ago.

The two shared services become PrivateLink endpoint services. Each sits behind a Network Load Balancer in the shared-services VPC, with acceptance required and the allowed-principals list (account, role or user ARNs) maintained by the same pipeline that vends accounts, so a new account is permitted on the day it is created. The owner then creates one interface endpoint per service in each of the four shared VPCs and turns on private DNS so the existing hostnames keep resolving. Every participant in that VPC uses them without creating anything. That is eight endpoints across the estate instead of the 608 that per-account VPCs would have needed, and about USD$175 a month in endpoint hours instead of USD$13,300. The direction is fixed by the mechanism, since only the consumer can open a connection, which is the property the security review had been asking for through firewall rules. It also carries the metadata store’s PostgreSQL traffic, because an endpoint service is TCP behind a load balancer and does not inspect the protocol on top. VPC Lattice could serve both legs too, a service for the identity API and a resource configuration for 5432, but that means standing up a service network and auth policies for two services that are already behind load balancers.

Two gotchas travel with the PrivateLink leg. The service sees the load balancer node addresses as the source, not the caller’s, so per-caller attribution needs proxy protocol v2 on the target group or, more usefully at 304 callers, an application-layer token that the identity service is already issuing. And a consumer only sees the Availability Zones it has in common with the provider, which is another reason the whole estate should be reasoning in AZ IDs before the first endpoint is created.

Not everything moves. An account that needs its own NACLs, its own DHCP option set, its own internet gateway, its own VPC-level flow log configuration or a VPC attribute a participant cannot set keeps its own VPC and takes an attachment. So does anything whose isolation requirement is network-level rather than IAM-level, because every workload in a shared VPC can reach every other one subject only to security groups. That test belongs in the account-vending pipeline, asked once at intake, rather than in a migration ticket two years later.

Worked example

Three accounts arrive in one week and the intake question is the same each time.

A reporting service for the finance product line: one Fargate service, calls identity and the metadata store, nothing else. No NACL requirement, no NAT of its own, no inbound from outside the estate. It becomes a participant in the non-production shared VPC and then the production one, launching tasks into shared subnets with its own security group, referencing the shared-services group by account-number/security-group-id. It creates no route table, no endpoint and no attachment, and it consumes no address space beyond the interfaces its tasks hold.

A payments service under a card-scheme obligation to keep its own subnet-level controls and produce its own network evidence: it needs NACLs it can change and a VPC flow log configuration in its own account. That is a VPC-attribute answer, so it keeps a VPC, gets a /22 rather than a /20 from a freshly summarised block, and takes a transit gateway attachment into the production route table. Its calls to identity go over the attachment it already has.

A third-party analytics vendor’s collector, running in an account the vendor administers: one-way, HTTPS, and no reason for anything of ours to be routable to it. It gets neither a shared subnet nor an attachment. The collector publishes an endpoint service on its side; the platform creates one interface endpoint per shared VPC, and the vendor never appears in a route table anywhere.

What’s worth remembering

1. Active VPC peering connections per VPC defaults to 50 and stops at 125, the hard ceiling that rules out a full mesh at this size, while routes per route table (500, raisable to 1,000) and attachments per transit gateway (5,000, adjustable) both move on request.
2. A managed prefix list referenced by a route table or a security group consumes its maximum entry count, not its current one, so an oversized list takes slots in every table that references it without that showing in the entry count.
3. VPC sharing splits along a fixed line: the owner keeps the VPC, subnets, route tables, NACLs, gateways, endpoints and transit gateway attachments, while participants create and manage their own interfaces, security groups and workloads against their own quotas, referencing other accounts’ groups as account-number/security-group-id.
4. Shared subnets require AWS Organizations and RAM sharing enabled from the management account, allow 100 participant accounts per VPC and 100 shared subnets per account by default, and cannot be used with a default VPC.
5. Zone names map to different physical zones in different accounts, so anything shared across accounts has to be placed and costed by AZ ID, where same-zone data transfer is free regardless of account ownership.
6. A PrivateLink endpoint service carries any TCP protocol in one direction with no routes and no CIDR coordination, and one endpoint in a shared VPC serves every participant in it, which is what turns 608 endpoints into eight.