A Subnet, a Gateway and a Route Nobody Wrote

The situation

A charity has moved its donation platform into a new VPC addressed 10.20.0.0/16, with three tiers: web, application, and a database.

Four things are wrong on the first day of testing.

The web servers cannot be reached from the internet. They have public IP addresses and a security group allowing inbound 443 from anywhere, and the connection times out.

The application servers cannot reach the internet at all. They sit in a subnet with no route to an internet gateway, which was the intention, but they still need to download operating system patches and call a third-party payment API.

The database sits in the web servers’ subnet. It was launched there because that was the subnet that already existed, and it holds fifteen years of donor records. Nothing can reach it at the moment, for exactly the reason nothing can reach the web servers, and that is the problem: the fix for the first symptom is a route to an internet gateway on that subnet’s route table, and the moment anyone adds it the database is on a public path too.

The nightly backup cannot reach S3. It uploads 200 GB to a bucket in the same Region and has failed every night since the move, for the same reason the application servers cannot fetch patches. The obvious remedy is to give the private subnets a NAT gateway, and the trustees have asked what that would mean: 200 GB a month leaving the VPC through a NAT gateway and an internet gateway to reach the bucket’s public endpoint, charged by the gigabyte on the way through.

A fifth question arrives later in the week: the charity’s small office needs a private path to the platform for staff administration, and the finance trustee has asked whether that means a leased line.

What actually matters

Start with what makes a subnet public, because three of the four symptoms come back to it. A subnet is not public because of a checkbox or because the instances in it have public addresses. It is public because its route table has a route to an internet gateway. An instance with a public IP in a subnet whose route table has no such route has an address nobody can reach it on, which is the first symptom exactly. Reachability in a VPC is a chain, and a break anywhere in it looks the same from outside: route table, gateway, network ACL, security group, and the operating system’s own firewall.

Then the difference between reaching out and being reached. Those are separate capabilities and they are provided by different components. A gateway that allows traffic in both directions does not fit a tier that must initiate connections but never receive them. That asymmetry is the second symptom, and it is what a network address translation gateway exists for.

Third, subnet boundaries are a security control rather than an addressing convenience. Putting the database in the same subnet as the web servers means it inherits their route table, whatever that route table turns out to say. Today it says nothing beyond the local route, which is why the database looks safe. Add the route the web tier needs and the database is on a public path, without anybody having decided that it should be. No security group tightening changes where the resource sits, so the fix is to move it, and to move it before the route is written.

Fourth, traffic to an AWS service does not have to leave the VPC at all. An instance calling S3 by its public endpoint routes out through a NAT gateway and then the internet gateway. That traffic stays on the AWS network rather than crossing the public internet, but it is charged per gigabyte processed by the NAT gateway, and the trustees are right to ask which path it takes. One component carries the call on a private path instead, with no internet gateway or NAT device involved, and for S3 and DynamoDB it carries no charge. Reaching for a NAT gateway because the backup needs egress would work and would cost more.

Finally, a VPC has two firewalls and they behave differently. One attaches to an instance’s network interface, allows only, and is stateful, so a reply to an allowed outbound request is permitted automatically. The other attaches to a subnet, supports deny as well as allow, is evaluated in rule order, and is stateless, so return traffic needs its own explicit rule. Forgetting the second property is a classic way to produce a connection that works in one direction.

What we’ll filter on

1. Allows inbound connections from the internet, or does not.
2. Allows outbound connections without allowing inbound ones.
3. Keeps traffic to AWS services inside the AWS network.
4. Operates at the subnet level or at the instance level.
5. Connects the VPC to something outside it: another VPC, an office, or a data centre.

The landscape

An internet gateway attaches to the VPC and allows traffic between it and the internet, in both directions. A subnet becomes public when its route table sends 0.0.0.0/0 to the internet gateway. Instances also need a public IP address or an Elastic IP to be reachable.

A NAT gateway lets instances in private subnets make outbound connections while remaining unreachable from outside. It lives in a public subnet, has an Elastic IP, and the private subnet’s route table sends 0.0.0.0/0 to it. It is created in one Availability Zone and is redundant inside that zone, charged per hour and per gigabyte processed. A newer regional availability mode expands a single gateway across zones automatically; the zonal one remains the default.

Route tables determine where traffic for a destination is sent. Every subnet is associated with exactly one, and the local route covering the VPC’s own range is always there and cannot be removed. Everything else about reachability in a VPC follows from which route table a subnet is associated with.

Security groups attach to an elastic network interface. They are stateful: a response to an allowed outbound request is allowed back in automatically. They support allow rules only, and all rules are evaluated together. A security group can reference another security group as a source, which is how a database tier allows only the application tier.

Network ACLs attach to a subnet. They are stateless: return traffic needs its own rule, which usually means allowing the ephemeral port range. They support allow and deny, and rules are evaluated in number order until one matches. The default ACL allows everything in both directions.

VPC endpoints provide private connectivity to AWS services without an internet gateway or NAT gateway. Gateway endpoints serve S3 and DynamoDB. You select route tables, and AWS adds a route to each one whose destination is the managed prefix list for the service. They carry no charge. Interface endpoints, which use AWS PrivateLink, place an elastic network interface with a private IP in the subnet for most other services, and are charged per hour in each Availability Zone plus per gigabyte processed.

VPC peering connects two VPCs privately. It is not transitive: three VPCs need three peerings, and a peered VPC cannot use the other’s internet gateway or VPN.

AWS Transit Gateway is a hub that connects many VPCs and on-premises networks, replacing a mesh of peerings once there are more than a handful.

AWS Site-to-Site VPN builds an IPsec tunnel over the internet between the VPC and an on-premises network. Each connection has two tunnels, a standard tunnel carries up to 1.25 Gbps, and the charge is per connection hour plus data transfer out. Latency and throughput vary with the internet connection underneath.

AWS Direct Connect provides a dedicated private circuit between a data centre and AWS, at port speeds of 1, 10, 100 or 400 Gbps, with consistent latency. AWS provisions the port within a few business days, and a cross connect then has to be ordered at a Direct Connect location through a network provider or partner, which is where the lead time goes. It is private but not encrypted by itself; a VPN over the top adds encryption where that is required.

Amazon Route 53 is DNS: domain registration, hosted zones, health checks, and eight routing policies, namely simple, weighted, latency-based, failover, geolocation, geoproximity, IP-based and multivalue answer.

Evaluation

Side by side

Component	Allows inbound from internet	Allows outbound only	Keeps AWS traffic private	Scope	Stateful
Internet gateway	✓	✗	✗	VPC	n/a
NAT gateway	✗	✓	✗	Subnet route	n/a
Gateway endpoint (S3, DynamoDB)	✗	✓	✓	Route table	n/a
Interface endpoint (PrivateLink)	✗	✓	✓	Subnet ENI	n/a
Security group	Per rule	Per rule	n/a	Network interface	✓
Network ACL	Per rule	Per rule	n/a	Subnet	✗
VPC peering	✗	✗	✓ (VPC to VPC)	VPC pair	n/a
Site-to-Site VPN	✗	✗	✓ (to on-premises)	VPC	n/a
Direct Connect	✗	✗	✓ (to on-premises)	VPC or Region	n/a

The four symptoms and their causes

Symptom	Cause	Fix
Web servers unreachable despite public IPs	The subnet’s route table has no route to an internet gateway	Add 0.0.0.0/0 to the internet gateway on the web subnets’ route table
Application servers cannot reach the internet	No outbound path from a private subnet	NAT gateway in a public subnet, 0.0.0.0/0 from the private route table to it
Database about to be exposed	It shares a subnet, and therefore a route table, with the tier that needs an internet gateway route	Move it to a private subnet before that route is added; security group allowing only the application tier
Nightly backup to S3 failing	No egress path at all, and S3’s public endpoint would need one	Gateway endpoint for S3 on the private route tables, rather than sending 200 GB through a NAT gateway

The solution

Fix the route tables first, because three of the four symptoms live there. Create at least two subnets per tier, one per Availability Zone, so the design is resilient as well as correct. The web subnets get a route table with 0.0.0.0/0 pointing at the internet gateway, which is what makes them public and what makes the web servers’ existing public IP addresses mean something. The application and database subnets get route tables with no internet gateway route at all, which is what makes them private.

Put a NAT gateway in a public subnet and point the application subnets’ default route at it. The application servers can then download patches and call the payment API, and nothing on the internet can open a connection to them, because translation only works for connections initiated from inside. Put one NAT gateway per Availability Zone if the platform has to survive the loss of a zone, since a zonal gateway lives in one zone and a route through a failed zone’s gateway goes nowhere. The regional availability mode is the alternative, spreading one gateway across zones without a route table per zone.

Move the database into a private subnet before adding that internet gateway route, not after, and treat the ordering as part of the fix. Done in the other order there is a window, however short, in which a database of donor records sits on a public path. Its security group should allow the database port from the application tier’s security group rather than from an address range, which keeps the rule correct when instances are replaced. Because security groups are stateful, no outbound rule is needed for the reply. If the trustees want a second layer, a network ACL on the database subnets can deny everything except the database port and the ephemeral range, remembering that an ACL is stateless and the return traffic needs its own rule.

Add a gateway endpoint for S3 and associate it with the private subnets’ route tables. The nightly 200 GB then goes to S3 from the private subnet directly, instead of out through the NAT gateway and the internet gateway, which is what it would have done had the NAT gateway been the whole answer to the backup failing. That keeps the per-gigabyte NAT processing charge off the backup traffic, and it answers the trustees’ question with a network path rather than an assurance. A gateway endpoint carries no charge of its own. Where the platform later needs private access to other AWS services, those use interface endpoints, which do have an hourly and per-gigabyte charge.

For the office, the answer to the finance trustee is that a leased line is not required. A Site-to-Site VPN builds an IPsec tunnel over the office’s existing internet connection, so there is no circuit to order, and the charge is a small hourly rate plus data transfer out. Direct Connect is the dedicated circuit, and it fits where consistent latency or sustained high bandwidth justifies ordering a cross connect at a Direct Connect location. A small office doing staff administration is comfortably a VPN case. Worth knowing for later: Direct Connect is private but not encrypted on its own, so a VPN runs over it where encryption is a requirement.

What’s worth remembering

1. A subnet is public because its route table has a route to an internet gateway, not because of an instance’s public IP address; reachability is a chain of route table, gateway, network ACL, security group and host firewall.
2. A NAT gateway allows outbound connections from private subnets without allowing inbound ones, lives in a public subnet, and is charged per hour and per gigabyte processed.
3. Security groups are stateful, allow-only, attach to a network interface, and can reference another security group as a source; network ACLs are stateless, support deny, attach to a subnet, and are evaluated in rule-number order.
4. Gateway endpoints give private access to S3 and DynamoDB through a route-table entry at no charge; interface endpoints use PrivateLink for most other services and are charged per hour and per gigabyte.
5. VPC peering is not transitive and does not share the other VPC’s internet gateway, NAT device, VPN or gateway endpoint; Transit Gateway is the hub once there are more than a handful of VPCs.
6. Site-to-Site VPN is an IPsec tunnel over an existing internet connection, two tunnels per connection; Direct Connect is a dedicated private circuit with consistent latency, needs a cross connect ordered at a Direct Connect location, and is not encrypted without a VPN over it.