Inspecting Traffic Between Accounts

The situation

A financial services group runs 34 accounts, each with one VPC, connected through a Transit Gateway in a shared network account. Routing is flat: any VPC can reach any other, subject to security groups. The security groups are permissive, because they were written when there were six accounts and everyone knew each other.

An internal red team exercise ended the argument. A foothold in a low-value marketing workload reached the payments VPC in two hops, and nothing recorded it beyond flow logs nobody was reading. The finding is that lateral movement between accounts is neither restricted nor visible.

The requirement that came out of it has three parts. Traffic between accounts must be inspectable, certain flows must be blocked outright, and the result has to be centrally managed rather than depending on 34 teams configuring their own security groups correctly. Latency between the trading VPCs is sensitive, and the platform team has been told not to add more than a couple of milliseconds to that path.

What actually matters

The first thing that matters is that inspection requires the traffic to pass through the inspecting thing, and on a Transit Gateway with a flat routing table it does not. Whatever is chosen, most of the work is routing: getting VPC-to-VPC traffic to detour via an inspection VPC and come back, which is a route table design rather than a firewall configuration.

The second is that not all east-west traffic deserves the same treatment. Inspecting everything costs latency and money on flows that carry no risk, and the trading path has a latency budget that inspection may not fit inside. Segmenting first, so that most traffic is simply not allowed rather than allowed-and-inspected, is cheaper and stronger than inspecting a flat network.

The third is that the thing in the middle becomes a dependency of everything. A centralised inspection VPC that goes down takes cross-account connectivity with it, and a design that fails closed on an appliance failure has converted a security control into an availability risk. Deciding fail-open or fail-closed, per flow class, is part of the design rather than an operational detail.

The fourth is that “inspectable” and “blocked” are different requirements that different products satisfy. Recording flows so an investigation can reconstruct what happened is achievable with logging alone. Blocking a flow needs something in the path. Conflating them leads to putting an expensive appliance in front of traffic where a flow log would have met the requirement.

Underneath all of it, the security groups are the actual finding. A permissive group set is why two hops worked, and no amount of central inspection fixes a network where everything is allowed to talk to everything.

What we’ll filter on

1. Does this restrict the flow, record it, or inspect its contents?
2. How is traffic made to pass through it?
3. What is the latency cost, and on which paths?
4. What happens to connectivity when it fails?
5. Is it managed centrally, or by each account team?
6. Does it scale to 34 accounts without 34 deployments?

The landscape

Transit Gateway route tables. The segmentation primitive, and the cheapest control on the list. Multiple route tables with selective associations and propagations mean a VPC can reach only what its route table knows about. Putting the marketing VPC in a route table that has no route to payments makes the red team’s second hop impossible with no appliance, no latency and no cost. This is segmentation rather than inspection, and it is the foundation the rest sits on.

A centralised inspection VPC. An inspection VPC attached to the Transit Gateway, with route tables arranged so that traffic between spoke VPCs is sent to the inspection VPC, inspected, and returned. This is the standard shape for centralised east-west inspection, and its cost is a Transit Gateway hop in each direction plus whatever the inspection itself adds.

AWS Network Firewall. Managed stateful and stateless inspection, deployable in the inspection VPC. It handles all IP traffic, supports Suricata rules, and logs alerts and flows. Managed centrally, scaled by AWS, and configured through firewall policies that can be shared across accounts with Resource Access Manager.

Gateway Load Balancer. The insertion mechanism for third-party appliances. It preserves the original packet using GENEVE encapsulation, distributes flows across a fleet of appliances with flow stickiness, and health-checks them. Reach for it when the required inspection is something a vendor product does and AWS does not, such as a specific IPS signature set or a DLP engine the organisation has standardised on.

VPC Lattice. Service-to-service connectivity with authorisation policies at the service level rather than the network level. It changes the question from “can this subnet reach that subnet” to “may this service call that service”, with IAM-based auth policies and per-request logging. It suits application-layer east-west control and does not inspect arbitrary IP traffic.

PrivateLink. Exposes a single service endpoint into a consumer VPC without connecting the networks at all. Where an account needs to reach one API in another account, PrivateLink gives exactly that and nothing else, and it removes the flow from the Transit Gateway entirely. It is the strongest available answer for narrow, well-defined cross-account dependencies.

Security groups referencing security groups. Within an account, and across accounts in a shared VPC or through a peering relationship, a group can reference another group as its source. It expresses tiers without addresses and costs nothing, and it is the control the organisation should have been using.

VPC Flow Logs and Traffic Mirroring. Flow logs record metadata about accepted and rejected flows. Traffic Mirroring copies actual packets to an out-of-band analysis target. Flow logs answer “who talked to whom”; mirroring answers “what did they say”, without sitting in the path, which means it inspects without being able to block.

Evaluation

Side by side

Control	Restricts	Inspects	In the data path	Latency cost	Central
TGW route table segmentation	✓	✗	✗	None	✓
PrivateLink for narrow deps	✓ strongly	✗	✗	Minimal	Per service
Security group references	✓	✗	✗	None	✗ per account
Network Firewall in an inspection VPC	✓	✓	✓	Two TGW hops plus inspection	✓
Gateway Load Balancer + appliance	✓	✓ deeply	✓	Higher	✓
VPC Lattice	✓ per service call	✗ packets	✓ at layer 7	Low	✓
Traffic Mirroring	✗	✓	✗ out of band	None	Per ENI
Flow Logs	✗	Metadata only	✗	None	✓

The table separates the three requirements cleanly. Blocking is satisfied most cheaply by route table segmentation, which costs nothing and adds no latency. Inspection needs something in the path, or Traffic Mirroring out of it. And the latency-sensitive trading path suits the rows with no data-path cost, which is a strong argument for segmenting that traffic rather than inspecting it.

The solution

Segment first with route tables, use PrivateLink for the narrow dependencies, inspect the traffic that remains, and mirror rather than intercept where latency does not allow a hop.

Segmentation comes first because it is free, it adds no latency, and it addresses the actual finding. Replace the single flat Transit Gateway route table with a set: production, non-production, shared services, and a restricted table for the payments and trading VPCs. Associate each VPC attachment with the table matching its zone, and propagate only the routes that zone is allowed to reach. The marketing VPC’s second hop stops being blocked by an appliance and starts being impossible, because there is no route.

Then take the narrow cross-zone dependencies out of the network entirely. Where a workload in one zone needs one API in another, expose it through PrivateLink rather than opening routing between the zones. The consumer gets an endpoint to that service and no path to anything else, and the dependency stops being a hole in the segmentation.

What remains is the traffic that genuinely crosses zones and cannot be reduced to a single service. Route that through a centralised inspection VPC running Network Firewall, with the Transit Gateway route tables arranged to send it there and back. Manage the firewall policy centrally in the network account and share it with Resource Access Manager, so 34 teams inherit the rules rather than implementing them.

The trading path gets the exception, made deliberately. Two Transit Gateway hops plus inspection does not fit a couple of milliseconds, so those VPCs stay in a restricted route table that permits only each other, with Traffic Mirroring to an analysis target so the traffic is inspectable after the fact without being in the path. That satisfies “inspectable” without satisfying “blockable” for that path, which is a trade to write down and have accepted rather than to make quietly.

Fix the security groups alongside, because central inspection does not repair a network where everything is permitted. Use IAM Access Analyzer and flow log analysis to find which flows are actually used, then rewrite the groups to reference each other rather than carrying wide CIDR ranges. This is slower than the routing work and it is the part that prevents the next foothold spreading within an account.

Decide the failure posture explicitly. The inspection path should fail closed for flows crossing security zones, because a compromised inspection layer permitting everything is worse than an outage in a design where those flows are meant to be rare. Flows within a zone do not traverse it and are unaffected either way, which is another argument for segmenting hard so the inspected set is small.

Why not put everything through the appliance. It is the design that gets drawn first and it means every flow pays two Transit Gateway hops and an inspection, including flows between two workloads in the same trust zone that carry no risk. Segmentation makes the inspected set small enough to afford.

Why not VPC Lattice for all of it. It is a strong answer for service-to-service authorisation and it works at layer 7 on services registered with it. A requirement to inspect arbitrary IP traffic between accounts is not what it does, and the estate has plenty of traffic that is not an HTTP service call.

Worked example

The route table redesign takes three weeks, most of it spent discovering dependencies nobody had documented. The discovery method is flow logs queried in Athena: for each VPC, which other VPCs does it actually talk to, over what ports, at what volume. Nineteen cross-zone dependencies turn up. Eleven are a single API call and become PrivateLink endpoints. Five are legitimate cross-zone flows that go through inspection. Three turn out to be nothing: leftover configuration pointing at services that were decommissioned, still permitted because the route existed.

Cutting over is done zone by zone, with the previous flat route table kept until each zone has run a week without an unexpected denial. Two surprises appear: a backup job reaching across zones on a schedule nobody had captured in a fortnight of flow logs, and a monitoring agent phoning a collector in the shared services VPC that had not been classified.

Network Firewall in the inspection VPC handles about 6% of the total east-west traffic once segmentation is done, against the 100% that a naive design would have sent through it. The cost is proportionate.

The red team runs again eight weeks later. The initial foothold is achieved in the same way, and it goes nowhere: the marketing VPC’s route table has no path to payments, and the attempt appears in the flow log as a rejected connection with no route rather than as a security group denial. The second finding is more interesting: lateral movement within the marketing account is still trivial, because the security group work is not finished. That becomes the next quarter, and it is the finding the central inspection design would never have surfaced.

What’s worth remembering

1. Inspection requires traffic to pass through the inspecting thing, and a flat Transit Gateway route table sends it directly; most of the work in east-west inspection is routing rather than firewall configuration.
2. Segment before inspecting. Route table segmentation costs nothing, adds no latency, and makes a flow impossible rather than allowed-and-examined, which is both cheaper and stronger.
3. PrivateLink removes narrow cross-account dependencies from the network entirely: the consumer reaches one service and has no path to anything else.
4. Restricting, recording and inspecting are three different requirements. Flow logs record, Traffic Mirroring inspects out of band without blocking, and only something in the data path can block.
5. A centralised inspection VPC becomes a dependency of all cross-zone connectivity, so the fail-open or fail-closed decision is a design choice per flow class, not an operational detail.
6. Central inspection does not fix permissive security groups; lateral movement inside an account is a separate problem with a separate fix, and it is usually the one the red team finds next.