Forty Ingresses and One Account Quota

The situation

One EKS cluster in us-east-1 across three Availability Zones, shared by forty product teams. Each team owns a namespace and ships its own Ingress, and the AWS Load Balancer Controller has created forty Application Load Balancers. Seven more ALBs in the same account serve workloads outside the cluster. That is forty-seven against a default quota of fifty Application Load Balancers per Region.

The VPC has a single CIDR, 10.20.8.0/22, allocated by the network team from a corporate address plan that has no spare 10. space left in it. Its subnets are shared into the platform account from a networking account using AWS RAM. A thousand and twenty-four addresses cover three node subnets and three load balancer subnets. Around nine hundred pods run at steady state, and every one of them holds a VPC address because the Amazon VPC CNI assigns pods real addresses from the subnet. Free addresses in the node subnets sit in the low dozens on a quiet afternoon, and a node replacement during a busy deploy window has twice failed to place pods.

The forty teams run about a hundred and twenty backend Services between them. Target groups are in instance mode, so each ALB registers the cluster’s nodes, sends traffic to a NodePort, and kube-proxy forwards it to a pod somewhere in the cluster. Two teams have a compliance-isolated workload they want to run on Fargate. And roughly once a week a rolling deploy produces thirty to ninety seconds of 502s.

What actually matters

Two of these constraints look like the same problem and are not. The account quota fails early and visibly: the forty-eighth ALB does not get created, the controller logs an error, someone raises a Service Quotas increase and the number goes up. Address exhaustion fails late, with no error until the moment it matters, at three in the morning, when an Auto Scaling group replaces a node and the CNI cannot attach a secondary interface with addresses on it. A design that only raises the quota has moved the failure from the one that pages a human during working hours to the one that does not. Any answer has to name which consumer of the /22 it shrinks, because the load balancers and the pods draw from the same pool. An Application Load Balancer needs a subnet of at least a /27 with eight free addresses in it, in each zone, so that it can scale out. Forty of them across three subnets take those addresses from the same thousand the pods need.

Ownership is the second axis. One load balancer per Ingress means the entry point lives inside each team’s blast radius. Every cross-cutting control the platform team applies (a WAF web ACL, an access-log destination, a TLS security policy, deletion protection) has to be applied forty times, and re-applied whenever a team recreates its Ingress. Consolidating moves the device into platform ownership and turns each team’s Ingress into a set of rules on somebody else’s load balancer. That trade-off is about governance, not cost, and it cuts both ways. The platform team gains a place to hang policy, and inherits a shared failure domain where one team’s bad rule, one team’s certificate mistake, and one team’s traffic spike now reach everybody.

The 502s are a third problem, and capacity has nothing to do with them. In instance mode the ALB health-checks the node, not the pod. A node passes as long as anything on it answers the NodePort, so a Deployment can roll pods out and terminate the old ones faster than the target group registers the new ones. The rollout needs a way to block until the target group reports the new pod healthy, and that property only exists when the thing registered in the target group is the pod.

The last consideration is what a self-managed answer demands of the platform team. Putting a proxy tier inside the cluster solves the quota arithmetic in one move, and it converts an AWS-managed failure domain into one the platform team patches, upgrades, capacity-plans and gets paged for. That trade-off works for teams who need routing behaviour AWS does not offer. It is a bad one when the requirement is “fewer load balancers”.

What we’ll filter on

1. Load balancers consumed against the default of fifty Application Load Balancers and fifty Network Load Balancers per Region.
2. Addresses consumed, counting both the load balancer’s own scaling interfaces and whatever it registers as targets.
3. Deploy safety: can the rollout be made to wait for the new pod to register and pass health checks before the old one goes away?
4. Fargate targets, since two teams need pods with no node under them.
5. Tenancy control: when forty teams share a device, is there a mechanism that stops one of them overriding another’s routing?
6. Rule and certificate headroom for forty hostnames and a hundred and twenty Services on a shared device.

The landscape

Four shapes are available for getting HTTP traffic into this cluster, and the AWS Load Balancer Controller sits underneath three of them.

One ALB per Ingress. The controller’s default behaviour. Create an Ingress and it creates an Application Load Balancer, listeners, target groups and rules to match. Every team gets an independent device with its own DNS name, its own WAF association, its own certificate set and its own quota consumption. Clean isolation, and it is what has put the account at forty-seven.

IngressGroup. The same controller, with alb.ingress.kubernetes.io/group.name on each Ingress. Every Ingress carrying the same group name is merged onto a single ALB, and the controller composes their rules into one listener rule set. Rule evaluation order comes from alb.ingress.kubernetes.io/group.order, an integer with a default of 0 where lower numbers evaluate first, with ties broken lexicographically by namespace and name. Forty Ingresses can become one load balancer without any team changing its manifest structure.

An in-cluster proxy behind a Network Load Balancer. Run the NGINX ingress controller (or any similar in-cluster proxy) as a Deployment and expose it with a Kubernetes Service of type LoadBalancer. The AWS Load Balancer Controller then provisions a Network Load Balancer in front of it. Since controller version 2.5 that is the default for type: LoadBalancer, via a mutating webhook that sets spec.loadBalancerClass to service.k8s.aws/nlb. One NLB, one target group holding the proxy replicas, and all forty teams’ routing decisions made by NGINX inside the cluster from its own Ingress objects.

Gateway API. The controller added Kubernetes Gateway API support in its 2.13 line: L4 routes over NLBs from 2.13.3, and L7 routes over ALBs from 2.14.0. There are two GatewayClasses, one backed by an NLB for TCPRoute, UDPRoute and TLSRoute, and one backed by an ALB for HTTPRoute and GRPCRoute; mixing protocol layers on one Gateway is not supported. The model splits ownership the way this scenario needs it split: the platform team owns the Gateway, and each team owns an HTTPRoute attached to it. It also moves configuration that Ingress could only express as vendor annotations into typed API fields.

Two things in this space are not selectable and should not be reached for. The legacy in-tree Kubernetes cloud provider still provisions Classic Load Balancers for type: LoadBalancer Services, when the AWS Load Balancer Controller is absent or its service mutator webhook is disabled, and it now receives only critical bug fixes. The standalone ALB Ingress Controller (v1) that preceded the current controller has been superseded by it. Neither belongs in a new design.

Evaluation

Side by side

The status quo fails on the two constraints that started this. It also fails deploy safety, though not because of the load balancer count: nothing about one-ALB-per-Ingress forces instance targets, but that is where the cluster is, and the fix for the 502s is orthogonal to the fix for the quota.

The NGINX option collapses the quota problem hardest, down to one NLB with a handful of proxy replicas as targets. It also saves the most addresses of the four, because neither the load balancer nor its targets scale with the number of application pods. That has three consequences. AWS WAF cannot be associated with a Network Load Balancer, so the web ACL that currently protects public traffic would have to move into the cluster or in front of the NLB behind CloudFront. Client addresses do not survive by default. On an NLB IP target group speaking TCP or TLS, client IP preservation is off, so NGINX sees the load balancer’s address unless proxy protocol v2 is enabled on the target group and NGINX is configured to parse it. And the L7 routing decisions, the access logs and the TLS termination all become the platform team’s software to run.

Gateway API scores well on tenancy because route attachment is a first-class, permissioned relationship rather than a shared annotation, and it is the direction the ecosystem is moving. Against it: it needs controller 2.14.0 or later plus the Gateway API CRDs installed, and forty teams have to rewrite forty Ingress objects as HTTPRoutes. The underlying ALB carries exactly the same rule and target-group quotas as the IngressGroup answer, so that constraint stays.

IngressGroup is the answer that gets there without a migration, and its weaknesses are the ones with known mitigations.

The solution

Merge the forty Ingresses into three IngressGroups, switch every target group to ip mode, turn on pod readiness gates, and give the pods their own address space with a secondary VPC CIDR.

Why three groups and not one

The controller will merge all forty onto one ALB if they share a group name, and two quotas rule that out here. Target groups per Application Load Balancer is 100 and is not adjustable. Each backing Service gets its own target group in either mode. A hundred and twenty Services on one ALB therefore do not fit, and no support case will change that. Rules per ALB is 100 excluding defaults and is adjustable, but forty teams averaging two or three host-and-path rules apiece will sit against it from day one. Splitting on a boundary the platform team already recognises, say public-edge, internal-tools and regulated, puts roughly forty target groups on each ALB, gives every group room to grow, and shrinks each blast radius to a third. The account goes from forty-seven Application Load Balancers to ten.

Certificates matter here too. An ALB carries 25 certificates beyond the default one, so forty distinct hostnames on one device would not fit either without a quota increase. A single wildcard covering *.platform.example.com in ACM covers all forty. The controller will discover it from the hosts in each Ingress’s spec.tls or from the host field on its rules, so long as an HTTPS listener is declared with alb.ingress.kubernetes.io/listen-ports. Route 53 alias records per hostname point at the group’s ALB.

Why ip targets, and what comes with them

Switching alb.ingress.kubernetes.io/target-type from instance to ip registers pod addresses in the target group instead of node IDs. Four consequences follow, and only the first is about performance.

The NodePort hop disappears. Traffic goes from the ALB to the pod, not to a node port and then through kube-proxy to a pod that may be on a different node in a different Availability Zone. That removes a hop, removes the cross-zone data-transfer charge on that hop, and removes a layer that can balance a connection onto a node the ALB never selected.

Fargate becomes possible. IP targets are required when the target pods run on Fargate or on EKS Hybrid Nodes, because there is no instance to register. The two compliance-isolated workloads stop being a special case.

Health checks start testing the thing that serves the request, so a target going unhealthy means that pod, not that node.

And pod readiness gates become available, which is what stops the weekly 502s. Label each team’s namespace with elbv2.k8s.aws/pod-readiness-gate-inject=enabled and the controller injects a readiness condition into new pods. The pod does not report Ready until it is registered in the target group and healthy. The Deployment controller therefore does not proceed with the rollout, and maxUnavailable constrains something real. The feature only works with ip targets, because in instance mode the registered object is the node and the node’s health does not describe any particular pod. Pair it with a deregistration_delay.timeout_seconds chosen deliberately rather than left at the 300-second default, and a preStop sleep long enough for the deregistration to propagate before the container stops accepting connections.

What ip mode adds is target count. Nine hundred pods registered individually run against Targets per Application Load Balancer, default 1,000 and Targets per Target Group per Region, default 1,000, both adjustable. Three groups keep this comfortable. One group would have sat within a hundred targets of the default, which is the second reason for the split.

Giving the pods their own addresses

A /22 cannot hold this cluster no matter how the load balancers are arranged, so the pods move out of it. Associate a secondary CIDR block from the RFC 6598 shared address space, 100.64.0.0/10, with the VPC. A VPC association tops out at a /16, so take 100.64.0.0/16. That association is permitted alongside a 10.0.0.0/8 primary, where a second RFC 1918 range from 172.16.0.0/12 or 192.168.0.0/16 would be rejected, and it is address space outside the corporate plan. Carve three /19 pod subnets, one per Availability Zone, and enable VPC CNI custom networking with an ENIConfig per zone so secondary interfaces get their addresses from the new subnets.

Three things to be deliberate about. With custom networking enabled, no addresses from a node’s primary network interface are given to pods. The pods-per-node ceiling therefore drops by roughly one interface’s worth, the node group needs its max-pods recalculated, and only nodes launched after the change pick the feature up, so the existing ones have to be recycled. Second, the ALB’s ip target groups reach 100.64 addresses over the VPC’s local route with no extra work, and NLB IP target groups accept 100.64.0.0/10 too, so the load balancing layer needs no change. Third, pod traffic to an address outside the VPC’s own CIDR blocks is source-NATed to the node’s primary address by default, so the shared address space never appears on the corporate network. If something on-premises needs to reach a pod address directly, that assumption breaks and the design needs revisiting before the CIDR is chosen.

The original /22 keeps the nodes and the load balancers, and with three ALBs instead of forty it has room again.

The gotchas that come with a shared ALB

IngressGroup carries an explicit security warning, and it is the design’s main weakness. Any Kubernetes user with RBAC permission to create or modify an Ingress can add group.name to their own object and join the group. Once inside, they can write a rule with a lower group.order that shadows another team’s path, or claim / for themselves. The controller does no arbitration. Use it only where every user with that permission is inside one trust boundary. Back that with an admission policy in the cluster that pins each namespace to a permitted group name, forbids a catch-all path, and enforces a group.order band per namespace so no team can jump the queue. Give every namespace a distinct order value rather than letting ties fall to lexicographic namespace ordering, which nobody will predict correctly at review time.

Lifecycle is the other one. The ALB belongs to the group, not to any Ingress in it. When the last Ingress leaves the group the controller deletes the load balancer, and deletion protection on it is ignored. The next one created gets a new DNS name, and the Route 53 aliases pointing at the old one break. Keep a platform-owned anchor Ingress in each group so the group can never empty.

Finally, the subnets are shared through RAM, and a participant account cannot modify a shared subnet or its attributes, tags included. Subnet auto-discovery relies on kubernetes.io/role/elb and kubernetes.io/role/internal-elb tags, so either the networking account applies those tags, or each group’s Ingress pins its subnets explicitly with alb.ingress.kubernetes.io/subnets. Pinning is the more predictable of the two here, because it also stops the controller from picking a subnet lexicographically when a zone has more than one candidate.

Worked example

The public-edge group’s Ingress in a team namespace, once the design is in place:

metadata:
  annotations:
    alb.ingress.kubernetes.io/group.name: public-edge
    alb.ingress.kubernetes.io/group.order: "220"
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/subnets: subnet-0a1, subnet-0b2, subnet-0c3

No certificate ARN, because the wildcard is discovered from the host on the rule. No load balancer attributes, because access logging, the WAF association, the TLS policy and deletion protection are set once on the group by the platform team’s own Ingress and apply to the shared device. group.order 220 sits in the band reserved for this namespace by the admission policy.

The namespace carries elbv2.k8s.aws/pod-readiness-gate-inject=enabled, so the next rollout in it registers pods, waits for them to go healthy in the target group, and only then terminates the old ones.

The arithmetic afterwards: three Application Load Balancers plus seven unrelated ones against a quota of fifty. Around forty target groups on each ALB against a hard ceiling of a hundred. Roughly three hundred pod targets per ALB against an adjustable thousand. Nine hundred pod addresses out of a /19 per zone instead of a shared /22, and the original subnets carry three load balancers’ interfaces instead of forty, with the eight free addresses each of them needs to scale.

What’s worth remembering

1. The AWS Load Balancer Controller creates an ALB for an Ingress, an NLB for a Service of type LoadBalancer (the default since controller 2.5, via a mutating webhook that sets loadBalancerClass), and, from 2.14.0, an ALB for a Gateway.
2. target-type: ip registers pods and target-type: instance registers nodes behind a NodePort; IP targets are required for Fargate, and pod readiness gates only work with them.
3. Pod readiness gates, enabled by labelling the namespace elbv2.k8s.aws/pod-readiness-gate-inject=enabled, hold a pod out of Ready until it is healthy in the target group, which is what stops a rolling deploy from draining faster than registration.
4. group.name merges Ingresses onto one ALB and group.order sets rule precedence, but any user who can create an Ingress can join the group, so it needs a trust boundary and an admission policy rather than good manners.
5. Target groups per ALB is 100 and cannot be raised; ALBs per Region (50), rules per ALB (100), targets per ALB (1,000) and certificates per ALB (25 beyond the default) all can be.
6. When a /22 runs out, a secondary CIDR from 100.64.0.0/10 plus VPC CNI custom networking gives pods their own space without touching the corporate address plan; the pods-per-node ceiling drops and every node has to be recycled.