Four Protocols and One Front Door

The situation

One platform team, one Region, three Availability Zones, one VPC on 10.60.0.0/16 with three /24 edge subnets and three /20 workload subnets. Four things need to be reachable. The team has been asked to put them behind “the load balancer” as though there were only one kind.

The HTTP API is fourteen services on Fargate behind a single hostname, routed by path prefix, with one header rule that sends x-fleet-channel: beta traffic to a canary target group. The MQTT brokers are twenty-two EC2 instances listening on TCP 8883. They terminate mutual TLS themselves, because the device certificates are the customer’s identity system, and a hundred and eighty thousand field devices hold connections open for days. The telemetry collector is six containers taking roughly forty thousand UDP datagrams a second on port 5140. And six third-party inspection appliances, bought by the security team, take traffic encapsulated in GENEVE on port 6081 and nothing else.

The constraint that makes this interesting belongs to the MQTT tier. Sixty customers run the devices behind corporate firewalls with egress rules written against IP addresses, not names. Changing those addresses means sixty change tickets. Whatever fronts the brokers has to present stable addresses that can be published once, and the same customers reach the management API on the same allow-list.

What actually matters

The first thing to settle is what layer the routing decision is made at, because that determines what the device in the path is allowed to do to the traffic. Reading a path prefix or a header means terminating the connection, parsing HTTP, and forwarding on a second connection; anything that does that has already rewritten the packet. A flow decision at layer 4 picks a target once per connection and leaves the payload alone. That is the only option when the payload is mutual TLS the load balancer holds no key for. Packet inspection needs something stranger again: the appliances have to see the original packet, source address and all, so the device in front of them cannot be a proxy. Three different jobs, and no amount of standardising makes one device good at all three.

The second is what contract the client side is holding. A DNS name is the loose contract, because the addresses behind it can change and nobody notices. An IP allow-list at sixty customers is a tight contract that gets revised at the speed of their change boards. That flips the usual reasoning. For the broker fleet the front-door address is the fixed requirement and the protocol is the easy part, so the choice comes down to which load balancer type has network interfaces you can pin addresses to. Everything reachable from those same allow-listed addresses inherits the constraint, including the parts that would otherwise have sat behind a name.

Third is what a bad day looks like, and the default that surprises people is zonal distribution. A load balancer node exists in each enabled zone, DNS hands clients one node address per zone, and whether that node may reach targets outside its own zone is a setting with different defaults per type. Get it wrong with an uneven fleet and a third of new connections land on one instance while its neighbours idle. The fix is a single attribute. On a Network Load Balancer that attribute also moves the bill, because the traffic it enables crosses a zone boundary and AWS applies EC2 data transfer charges to it. Long-lived connections make it worse: a device that resolves once, connects, and stays connected for four days is pinned to one zone, and a rebalance changes nothing until it reconnects.

Last is what each device consumes in address space and quota headroom. Every load balancer needs free addresses in each subnet it occupies, every appliance endpoint consumes another, and the per-Region counts are quotas. That arithmetic is worth doing before the design is agreed, not after the fourth front door fails to provision.

What we’ll filter on

1. Routing layer: does the decision read the request (layer 7), hash the flow (layer 4), or forward the packet untouched (layer 3)?
2. Listener protocols carried: HTTP and HTTPS, TCP 8883, UDP 5140, GENEVE 6081.
3. Pinnable front-door address: a stable IP per Availability Zone that can be published to a customer firewall and never move.
4. Target-group types available: instance, ip, alb, GENEVE, and what each one lets us register.
5. Zonal distribution behaviour: the cross-zone default, what an uneven fleet does under it, and what turning it on changes about charges and target limits.
6. Footprint: subnets, free addresses and per-Region quota consumed.

The landscape

Elastic Load Balancing offers three current types. They differ more in what you can register behind them than in what they accept in front.

Application Load Balancer. Layer 7, HTTP and HTTPS listeners only. Listener rules match on host, path, header, method, query string and source IP, and forward to target groups whose protocol is HTTP or HTTPS. Target types are instance, ip and lambda. An ip target group accepts addresses from the VPC’s subnets or from RFC 1918 and RFC 6598 space, including on-premises targets over Direct Connect or Site-to-Site VPN, but never a publicly routable address. It needs subnets in at least two Availability Zones, each with a CIDR of at least a /27 and at least eight free addresses so the load balancer can scale out. Run short of that and a scaling attempt stalls: the old nodes keep serving, and new connections can get 5xx errors. Its addresses are managed by the service, so there is no way to attach an Elastic IP or fix the address it uses in a given zone. Cross-zone load balancing is on at the load balancer level and cannot be turned off there, though a target group can opt out. It is the only type that takes an AWS WAF web ACL directly.

Network Load Balancer. Layer 4. Listener protocols are TCP, TLS, UDP, TCP_UDP, QUIC and TCP_QUIC. Elastic Load Balancing creates one network interface per enabled zone and gives it a static address: an Elastic IP for an internet-facing load balancer, or a private address you can specify from the subnet range for an internal one. Target types are instance, ip and alb. Client IP preservation is on by default for instance target groups, and on for ip target groups on UDP, TCP_UDP, QUIC and TCP_QUIC, where it cannot be turned off. It is off by default for ip target groups on TCP and TLS, where proxy protocol v2 is the other way to hand the target the original address. Stickiness is source-IP based and is not supported on TLS or QUIC listeners. Cross-zone is off by default. Security groups can be attached only at creation, so a load balancer created without one can never have one added.

Gateway Load Balancer. Layer 3. It listens for all IP packets on all ports and forwards them to its targets encapsulated in GENEVE on port 6081, using a five-tuple flow hash so every packet of a flow reaches the same appliance. Its target group protocol is GENEVE and its target types are instance and ip. The traffic it inspects is not addressed to it. A Gateway Load Balancer endpoint is created in each zone and route tables point at that endpoint, which is why the service works across VPC and account boundaries the way a PrivateLink endpoint does. Subnets need eight free addresses each. Flow stickiness defaults to five-tuple and can be relaxed to source and destination address, or those plus protocol, for appliances that need both directions of a conversation on one box. The interface takes packets up to 8,500 bytes and GENEVE adds 68 bytes, so appliances need an MTU of at least 8,568. There is no fragmentation and no path MTU discovery, so oversized packets are dropped without an ICMP message. Cross-zone is off by default here too.

Two more things sit adjacent to the decision. AWS Global Accelerator gives two anycast IPv4 addresses at the edge and can front Application Load Balancers, Network Load Balancers, EC2 instances or Elastic IPs over TCP and UDP. That turns an allow-list of per-zone addresses into an allow-list of two, with a second service in the path and client-IP-preservation conditions of its own. AWS PrivateLink endpoint services are fronted by a Network Load Balancer or a Gateway Load Balancer, which is worth knowing before the first customer asks for private connectivity instead of an allow-list. The Classic Load Balancer is the previous generation, so it is not a candidate for a new build and does not appear in the table below.

Evaluation

Side by side

Option	Layer 7 rules	Carries TCP, UDP and GENEVE	Pinnable address per zone	Target types on offer	Cross-zone default	Footprint
Application Load Balancer	✓	✗	✗	instance, ip, lambda	on, not settable	2+ zones, /27, 8 free IPs
Network Load Balancer	✗	✓ (not GENEVE)	✓	instance, ip, alb	off	1 interface per zone
Gateway Load Balancer	✗	✓ (GENEVE only)	✗	instance, ip	off	8 free IPs plus an endpoint per zone
NLB with an ALB target	✓ via the ALB	✓ (not GENEVE)	✓	alb, alongside instance and ip	off, inherited by the alb group	both of the above
Global Accelerator in front	via its endpoint	✓ (not GENEVE)	✓ (two anycast IPs)	ALB, NLB, EC2, EIP	n/a	edge, not VPC

No single row carries all four workloads, and the crosses fall in a pattern that says how to combine them. The Gateway Load Balancer is alone in the GENEVE column and useless for anything else, so the appliance fleet is decided before the discussion starts. The Application Load Balancer is alone in the layer 7 column and cannot be given a fixed address. The Network Load Balancer carries three of the four protocols and is the only type whose interfaces you can pin an Elastic IP to. That makes it the front door for anything the customer firewalls have to name. The fourth row closes the gap: register the Application Load Balancer as a target of the Network Load Balancer and the HTTP rules run behind an address the allow-list already contains.

Global Accelerator would collapse the allow-list to two addresses, so it is a reasonable later addition. It still needs a load balancer underneath it, and adding it now means the customers write two allow-lists in the same year.

What cross-zone does to a lopsided fleet

The broker fleet already demonstrated this. During a scale-in event the Auto Scaling group ended up with eight instances in the first zone, eight in the second and one in the third. With cross-zone load balancing off, the Network Load Balancer default, each zonal node distributes only to targets in its own zone, and DNS hands out the three node addresses roughly evenly. The single instance in the third zone took about a third of new connections while each of its sixteen peers took about a twenty-fourth. Nothing was unhealthy; the distribution was working exactly as configured.

Turning the attribute on levels that out, and it changes two other things. Every connection served across a zone boundary attracts EC2 data transfer charges. And on a Network Load Balancer with cross-zone enabled the ceiling changes from 500 targets per zone to 500 targets in total. The answer for a fleet this size is to keep it off and make each zone able to serve its own share. Capacity per zone then becomes a number the Auto Scaling group has to hold rather than an accident of scale-in order.

Sorting the four workloads

The solution

Three load balancers: an internal Application Load Balancer, an internet-facing Network Load Balancer carrying three listeners, and a Gateway Load Balancer with an endpoint in each zone. The Application Load Balancer is registered as a target of the Network Load Balancer, which is how fourteen sets of path and header rules end up reachable at an address a customer firewall can name.

The Network Load Balancer is the front door

Create it internet-facing across all three zones on day one with an Elastic IP chosen for each. Those three addresses are what sixty customers write down, and adding a fourth later means sixty change tickets. Attach a security group at creation whether or not there is a rule worth writing yet; a Network Load Balancer created without one can never be given one, and that is a rebuild rather than an edit.

Three listeners hang off it, and each one uses a different target type. TCP 8883 forwards to an instance target group of brokers, where client IP preservation is on by default, so the broker sees the device address and terminates the mutual TLS handshake itself. UDP 5140 forwards to an ip target group of collector tasks, where preservation is on and cannot be turned off, which is what makes per-device attribution possible in the collector. There is no UDP health check, so the target group health checks over TCP, HTTP or HTTPS against a side port the collector has to expose. A collector that answers on that port while its UDP path is broken will be marked healthy. TCP 443 forwards to an alb target group holding the Application Load Balancer.

Cross-zone stays off, with the Auto Scaling group holding enough broker capacity in each zone to serve a third of the devices. Connection termination on an unhealthy target is enabled by default, which is right for the HTTP path and worth reconsidering for MQTT sessions that should drain instead. The TCP idle timeout defaults to 350 seconds and can be raised to 6,000, so the MQTT keepalive interval has to sit under whatever is set or the load balancer drops flows the broker still holds open.

The alb target group and its edges

A target group of type alb holds exactly one Application Load Balancer, in the same VPC and the same account, reached by a TCP listener only. Its health check is HTTP or HTTPS against a listener on the load balancer itself. Its attributes are fixed at their defaults and cannot be edited, so there is no proxy protocol, no deregistration tuning and no per-target-group cross-zone override; it inherits the load balancer’s setting instead. Client IP is preserved through the hop, and the Application Load Balancer appends the device address to X-Forwarded-For before the request reaches the service. The fourteen services behind it see what they saw before the Network Load Balancer existed.

A registered Application Load Balancer counts as 50 targets per zone against the Network Load Balancer’s default of 500, or 100 if cross-zone is enabled, which is a second reason to leave cross-zone off. One Application Load Balancer can be a target of at most two Network Load Balancers, and it cannot be deleted until it is deregistered from both.

The Application Load Balancer stays internal

Give it the three workload subnets, ip target groups so the Fargate tasks are registered directly, the ACM certificate on an HTTPS listener, and the WAF web ACL that the Network Load Balancer cannot hold. The rule set is ordinary: path prefixes to thirteen target groups, and one header condition on x-fleet-channel in front of the canary group. Because it is internal, its nodes carry private addresses only and nothing reaches it except through the Network Load Balancer, which is the boundary the security review asked for. Check the eight-free-addresses rule in each of those /20 subnets before applying.

The Gateway Load Balancer sits outside the request path

It never appears in a URL or an allow-list. Create it with a GENEVE target group holding the six appliances, then create a Gateway Load Balancer endpoint in each zone and point route tables at the endpoint so the traffic being inspected is redirected through it. Two things need checking on the appliance side. The first is flow stickiness, which defaults to a five-tuple hash; an appliance that needs both directions of a conversation needs source_ip_dest_ip or source_ip_dest_ip_proto instead. The second is the MTU. The interface takes 8,500 bytes, GENEVE adds 68, appliances need at least 8,568 configured, and packets that are too large are dropped without an ICMP message because path MTU discovery is not supported. Keep cross-zone off so a flow stays in the zone it entered and the appliances in each zone carry their own share.

One interaction is worth knowing before it is discovered in production. Client IP preservation on the Network Load Balancer is not supported with a Gateway Load Balancer endpoint placed between the load balancer and its targets, even when the target is in the same VPC. Inspect at the edge and between VPCs, not between the load balancer and the fleet it balances.

Worked example

Follow one device through a working day.

Its firewall permits outbound to three addresses. It opens TCP 8883 to the second of them, reaching the Network Load Balancer node in the second zone. The node hashes the flow and picks a broker in that zone, forwards the connection with the device’s own source address intact, and the broker completes the mutual TLS handshake against the customer’s certificate authority. The connection stays open for four days. The flow hash is computed once and never revisited, so a broker added to the zone that afternoon takes new connections only.

Every thirty seconds the device sends a telemetry datagram to the same address on UDP 5140. The listener forwards to a collector task by flow hash, and because the target group is UDP the collector reads the device’s real source address off the packet with nothing to configure. UDP flow state expires after 120 seconds of silence, so a device that goes quiet for a minute and comes back may be hashed onto a different collector, which the collector’s design has to tolerate.

At noon a field engineer calls the management API on the same three addresses over TCP 443. The Network Load Balancer forwards to the alb target group. The Application Load Balancer evaluates the WAF web ACL, matches the path prefix, appends the engineer’s address to X-Forwarded-For, and forwards to a Fargate task registered by IP.

Meanwhile the platform’s own egress leaves through a route table that points at the Gateway Load Balancer endpoint in its zone. The endpoint hands packets to the Gateway Load Balancer, which encapsulates them in GENEVE and sends them to one of the six appliances, chosen by flow hash and held there for the life of the flow. The appliance decapsulates, inspects, re-encapsulates and returns them. The three workloads on the front door are not touched by any of it.

What’s worth remembering

1. The listener protocol says which load balancer types are possible; the target-group type (instance, ip, alb, or GENEVE) sets what you can register, and it is where most designs are settled.
2. Only the Network Load Balancer gives an address per Availability Zone that you can pin, which makes it the front door for anything a customer firewall has to allow-list, whatever protocol that thing uses.
3. Registering an Application Load Balancer in an alb target group puts layer 7 rules behind a static address, with a TCP-only listener, one load balancer per target group, fixed target-group attributes, and 50 target slots per zone (100 with cross-zone on).
4. Cross-zone load balancing is off by default on Network and Gateway Load Balancers, and turning it on for a Network Load Balancer adds EC2 data transfer charges; an 8/8/1 fleet with it off sends about a third of new connections to the single target.
5. Client IP preservation is on by default for instance target groups and for UDP-family ip target groups, where it cannot be turned off, and off for ip target groups on TCP or TLS, where proxy protocol v2 recovers the address.
6. The Gateway Load Balancer is reached through an endpoint in a route table rather than by name, and its 8,500-byte interface, 68 bytes of GENEVE overhead and missing path MTU discovery are the constraint that appliance vendors’ defaults tend to miss.