The situation
A pathology network is putting a patient results API behind a public endpoint. The request path as designed is CloudFront, then an Application Load Balancer, then containers on ECS, with a Network Load Balancer in front of a legacy component that some requests reach.
Three requirements collide. The regulator requires that patient data is encrypted in transit at every hop, including inside the VPC. The security team requires WAF inspection of request bodies, because the known attacks against this API are injection attempts in the payload. And one class of clients, hospital integration servers, authenticates with client certificates issued by the network’s own certificate authority.
The architecture review has stalled on where certificates go. One proposal terminates at CloudFront and runs plaintext behind it, which the regulator rejects. Another terminates only at the container, which the security team rejects because WAF then has nothing to inspect. The mutual TLS clients complicate both.
What actually matters
The first thing that matters is that terminating TLS and inspecting content are the same capability. A component that can apply a WAF rule to a request body is a component that has decrypted it. Any requirement to inspect payloads is a requirement to terminate somewhere, and the design question is where, not whether.
The second is that “encrypted in transit at every hop” does not mean one unbroken connection. It means no hop carries plaintext, which is satisfied by terminating and re-encrypting: CloudFront decrypts, inspects, and opens a new TLS connection to the origin. Reading the requirement as end-to-end when it says hop-by-hop is what produces the deadlock this review is in.
The third is that mutual TLS terminates wherever the certificate is validated, and that hop must be one that can see the client certificate. Any component in front of it that terminates TLS will present its own identity to the next hop, and the client certificate stops being visible unless it is forwarded as a header, which changes it from a cryptographic proof into an assertion the next hop has to trust.
The fourth is that the certificate’s issuer differs by hop. Public-facing hops need a publicly trusted certificate, and ACM issues those free for use with CloudFront, ALB and API Gateway. Internal hops need a certificate the client trusts, which for VPC-internal traffic is a private CA rather than a public one, and ACM Private CA exists for that.
Underneath it, several of these components cannot terminate TLS at all, and knowing which is what removes options from the table quickly.
What we’ll filter on
- Does this hop need to inspect the payload?
- Can this component terminate TLS, and if so with what certificate?
- Does terminating here destroy information the next hop needs, such as a client certificate?
- Is the traffic on this hop inside a trust boundary or crossing one?
- Who can read the plaintext if this hop is compromised?
- What issues and rotates the certificate?
The landscape
CloudFront. Terminates TLS from the client and opens a separate connection to the origin, with its own protocol and cipher policy on each side. It is where AWS WAF attaches for edge inspection, and it supports viewer certificates from ACM in us-east-1 specifically, which is the regional detail that catches people out. Origin connections can and should be HTTPS, giving hop-by-hop encryption without end-to-end.
Application Load Balancer. Terminates TLS, applies WAF, routes on content, and re-encrypts to targets if the target group protocol is HTTPS. It supports mutual TLS: it can verify a client certificate against a trust store, either passing through the certificate details as headers or verifying and rejecting at the load balancer. This is the component that resolves the mutual TLS requirement without giving up inspection.
Network Load Balancer. Operates at layer 4. It can terminate TLS, and it cannot inspect HTTP, so WAF does not attach to it. Alternatively it can pass TCP straight through, in which case the connection terminates at the target and the NLB sees only encrypted bytes. Reach for it when the requirement is to preserve an end-to-end connection or when the protocol is not HTTP; rule it out whenever payload inspection is needed.
AWS Certificate Manager. Issues and renews publicly trusted certificates at no charge for use with integrated services, and handles rotation automatically. Certificates cannot be exported, which is the constraint: a component that needs the private key on disk, such as a web server terminating TLS itself, cannot use an ACM public certificate.
ACM Private CA. A managed private certificate authority for internal certificates. It issues certificates that internal clients trust because they trust the CA, it supports exportable private keys, and it is the mechanism for encrypting VPC-internal hops and for issuing the client certificates a mutual TLS trust store validates against. It carries a monthly charge per CA plus a per-certificate charge, which is the reason people try to avoid it and usually should not.
AWS WAF. Attaches to CloudFront, ALB, API Gateway, AppSync and a few others. It inspects HTTP requests, including bodies up to a size limit, which means it must sit at a hop that has plaintext. It cannot attach to an NLB.
Service-to-service TLS inside the VPC. Encrypting traffic between the load balancer and the containers, and between containers. ACM Private CA issues the certificates; a service mesh or the application handles the handshake. This is the hop most often left plaintext on the argument that the VPC is trusted, which is precisely the argument the regulator’s requirement rejects.
TLS passthrough. An NLB or a Gateway Load Balancer forwarding encrypted bytes without terminating. It preserves the client’s connection to the backend, including client certificates, and it forfeits every inspection capability along the way.
Evaluation
Side by side
| Hop | Can terminate | WAF attaches | Sees client certificate | Certificate source |
|---|---|---|---|---|
| CloudFront | ✓ | ✓ | Only if it terminates mTLS | ACM in us-east-1 |
| ALB | ✓ | ✓ | ✓ with a trust store | ACM |
| NLB (TLS listener) | ✓ | ✗ | ✗ | ACM |
| NLB (TCP passthrough) | ✗ | ✗ | Passed to target | Target’s own |
| ECS task | ✓ | ✗ | ✓ if it terminates | ACM Private CA |
| API Gateway | ✓ | ✓ | ✓ with mTLS enabled | ACM |
The table makes the deadlock resolvable. Payload inspection requires CloudFront or ALB, both of which terminate. Mutual TLS verification requires ALB or API Gateway. Neither of those is compatible with a single unbroken connection from client to container, which means the regulator’s requirement has to be read as hop-by-hop, and reading it correctly is what unlocks the design.
The solution
Terminate at CloudFront, re-encrypt to the ALB, verify client certificates at the ALB, and re-encrypt again to the tasks with private certificates. Three terminations, no plaintext hop, and inspection where it is needed.
CloudFront terminates the public connection with an ACM certificate issued in us-east-1, and AWS WAF attaches there for the first inspection pass. Configure the origin protocol policy to HTTPS only, so CloudFront opens a TLS connection to the ALB rather than sending plaintext across the internet segment between edge and origin.
The ALB terminates that connection with its own ACM certificate and runs the second WAF pass, which is where request-body inspection belongs because the body is available and the rules can be tuned to this application rather than to edge-generic traffic. Configure the target group protocol as HTTPS so the ALB re-encrypts to the tasks.
Mutual TLS is handled at the ALB, which is the only component in the path that can verify client certificates and also inspect payloads. Configure a trust store containing the network’s CA, so hospital integration servers present their certificates and the ALB validates them, passing the certificate details to the application as headers for authorisation decisions. This is the choice that resolves the third requirement, and it means CloudFront is bypassed for that client class, which needs a separate listener or a separate hostname routed directly to the ALB.
The tasks terminate the final hop with certificates from ACM Private CA, which is what makes the VPC-internal segment encrypted rather than trusted. This is the hop most designs skip, and the regulator’s wording does not allow skipping it.
The legacy component behind the NLB is the exception worth being explicit about. If it needs the client’s original connection preserved, the NLB passes TCP through and nothing inspects those requests; if it can accept a terminated and re-encrypted connection, the NLB terminates with an ACM certificate and re-encrypts. Decide which, write down which requests reach it, and confirm whether the inspection requirement applies to that path. An unexamined passthrough is how a component ends up outside the WAF without anyone deciding it should be.
Then make the certificates somebody’s job. ACM handles renewal for the public certificates automatically. Private CA certificates on tasks need a rotation mechanism, and an expired internal certificate takes the service down as thoroughly as a bad deployment. Monitor expiry and alarm well before it.
Why not terminate once at the container and pass through everywhere. It satisfies the encryption requirement most literally and forfeits WAF entirely, which is the requirement the security team will not give up. There is no design that inspects payloads without decrypting them.
Why not terminate at CloudFront and run plaintext behind it. It is the common shape and the regulator’s wording excludes it. It is also the design that assumes the VPC is a trust boundary, which is the assumption most likely to be wrong.
Worked example
A request from a browser hits CloudFront, which terminates with the public certificate and runs the edge WAF rules. The managed rule groups catch two probes an hour before anything reaches the origin. CloudFront opens a TLS connection to the ALB with the origin protocol policy set to HTTPS.
The ALB terminates, runs the application-specific WAF rules including a body inspection rule tuned to the results schema, and opens a TLS connection to the task with a private certificate. The task terminates that and processes the request. Three connections, three certificates, no hop carrying plaintext.
A request from a hospital integration server takes a different path. It resolves a hostname routed directly to the ALB rather than through CloudFront, because the client certificate has to reach the ALB’s trust store and a CloudFront termination in front would have hidden it. The ALB validates the certificate against the private CA, passes the subject details as headers, and the application authorises on them. The WAF rules still apply, because the ALB still terminates.
The design review closes on the reading of the requirement. The regulator’s text says “encrypted in transit at every hop”, the team had been reading it as “end to end”, and a paragraph in the design document recording that distinction is what resolves an argument that had run for three weeks.
One thing is found late. The NLB in front of the legacy component was configured as TCP passthrough by default, so a subset of requests had been reaching a component with no WAF inspection and terminating TLS on software two versions behind. It is switched to terminate at the NLB with re-encryption, and the legacy component moves behind the ALB in the following quarter.
What’s worth remembering
- Terminating TLS and inspecting content are the same capability, so any requirement to inspect payloads is a requirement to terminate somewhere; the design question is where, not whether.
- “Encrypted in transit at every hop” is satisfied by terminating and re-encrypting at each hop, which is different from a single end-to-end connection, and confusing the two deadlocks the design.
- Mutual TLS terminates where the client certificate is validated, so anything in front of that hop which terminates TLS hides the certificate; ALB with a trust store is the component that verifies it and still inspects.
- An NLB cannot carry WAF, whether it terminates TLS or passes it through, so a payload inspection requirement removes it from that position in the path.
- ACM issues free public certificates for integrated services and will not let you export the private key; internal hops and client certificates come from ACM Private CA.
- The VPC-internal hop is the one most designs leave plaintext on the argument that the VPC is trusted, and it is exactly the assumption a hop-by-hop encryption requirement rejects.