ANS Lab 07 - Catch a packet with Traffic Mirroring

Scaffold: 3/5. The VPC, the source instance, the capture instance, the internal Network Load Balancer with its UDP listener on 4789, and the traffic mirror target are all built. The traffic mirror filter is built too, and it is empty. What is missing is the two filter rules and the mirror session. You add them, redeploy, and a tcpdump on the capture instance goes from silence to VXLAN.

The scenario

Flow logs tell you a conversation happened. They do not tell you what was in it: no sequence numbers, no flags, no window sizes, no payload. When the question is “what did the packets actually look like”, Traffic Mirroring is the only AWS service that answers it, because it is the only one that copies packets.

Copying packets is not free of consequence. Mirrored traffic is encapsulated in VXLAN and counts against the source instance’s own bandwidth and packet-per-second allowances, so mirroring an interface carrying 1 Gbps in and 1 Gbps out asks that instance to handle 4 Gbps. When the allowance runs out, mirrored packets are dropped first so production traffic keeps moving. That is the design constraint the filter exists to manage: mirror the flow you are investigating, keep the first hundred bytes or so, and leave everything else on the wire.

This lab builds the whole path and leaves you the two pieces that decide what crosses it.

What’s provided

• src/template.yaml - the VPC and two subnets, a source instance whose interface is declared separately (SourceEni) because Traffic Mirroring binds to an interface, not an instance; a capture instance with tcpdump installed and an HTTP health endpoint on 8081; an internal Network Load Balancer with a UDP listener on port 4789 and a UDP target group holding the capture instance; a traffic mirror target pointing at that load balancer; and an empty traffic mirror filter, with a clearly marked TODO where the rules and the session go.
• scripts/ - deploy, test, and teardown.
• solution/template.yaml - the complete, correct template.

The source instance opens one TCP connection a second to each of three ports on the capture instance: 443, 8080 and 9999. Nothing listens on any of them, so every probe is a SYN out and a RST back, which gives you traffic in both directions on three ports with no software to install.

Deploying src/template.yaml as shipped succeeds. Everything comes up, and nothing is mirrored, because a filter with no rules mirrors nothing.

Your task

Open src/template.yaml and fill in the two TODOs at the bottom.

The rules. Two AWS::EC2::TrafficMirrorFilterRule resources on MirrorFilter, both accepting TCP (Protocol: 6) between 10.20.0.0/16 and 10.20.0.0/16, and both naming port 443 only:

  MirrorRuleEgress:
    Type: AWS::EC2::TrafficMirrorFilterRule
    Properties:
      TrafficMirrorFilterId: !Ref MirrorFilter
      TrafficDirection: egress
      RuleNumber: 100
      RuleAction: accept
      Protocol: 6
      SourceCidrBlock: 10.20.0.0/16
      DestinationCidrBlock: 10.20.0.0/16
      DestinationPortRange:
        FromPort: 443
        ToPort: 443

The ingress rule is the same shape with TrafficDirection: ingress and a SourcePortRange of 443 to 443, because the reply comes back from 443 to an ephemeral port. Rules are directional and rule numbers are unique per direction, so both can be 100.

The session. One AWS::EC2::TrafficMirrorSession joining the three pieces:

  MirrorSession:
    Type: AWS::EC2::TrafficMirrorSession
    Properties:
      NetworkInterfaceId: !Ref SourceEni
      TrafficMirrorTargetId: !Ref MirrorTarget
      TrafficMirrorFilterId: !Ref MirrorFilter
      SessionNumber: 1
      PacketLength: 96
      VirtualNetworkId: 4242

PacketLength is bytes of the original packet, after the VXLAN header. 96 keeps the Ethernet, IP and TCP headers, including sequence numbers, flags and window size, and leaves the payload behind.

Run it

# Defaults: stack ans-lab-07, region ap-southeast-2.
./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # tcpdumps the capture instance and reads the result
./scripts/teardown.sh        # deletes everything

test.sh checks for a session on the source interface, checks the filter has rules, waits for the capture instance to pass its health check, then runs tcpdump -i any -n 'udp port 4789' on it through Systems Manager and counts what came back.

What success looks like

./scripts/test.sh prints the first few decoded packets, the counts, and then:

PASS: mirrored packets arrived VXLAN-encapsulated on UDP 4789, port 443 is
in the capture in both directions, and the two ports the filter did not
name are absent. The filter chose what crossed; the session carried it.

Widen it

Once it passes, do the experiment the whole feature is built around. Delete the DestinationPortRange and SourcePortRange from the two rules, remove PacketLength from the session, redeploy and run test.sh again. It now fails, because 8080 and 9999 are in the capture, and that failure is the finding: the capture also contains every byte of payload, and the mirrored copies are being carried by the source instance’s own network allowance. At this instance size and this traffic rate nothing gets dropped. At production rates the arithmetic bites, and the packets it drops are the mirrored ones.

If it fails

• No traffic mirror session on the interface. The gap is still open. Add the rules and the session to src/template.yaml and redeploy.
• VXLAN arrived but no port 443 inside it. Check which port range is on which rule. The egress rule names 443 as the destination; the ingress rule names it as the source.
• Nothing arrived on UDP 4789 at all. Three things break this quietly. The load balancer must have a UDP listener on 4789: without one, Traffic Mirroring fails with no error and no packets. The capture instance’s security group must admit UDP 4789 from the source’s private address, because client IP preservation cannot be turned off on a UDP target group, so the outer header carries the source instance’s address rather than the load balancer’s. And the target must be healthy, since an unhealthy target gets nothing.
• Only one direction shows up. If you add ingress rules and no egress rules, no outbound traffic is mirrored, and the other way round. Neither direction is implied by the other.

Reveal the solution

Deploy the complete reference template without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• A mirror session binds three separate resources: a source interface, a filter, and a target. The target is an interface, a Network Load Balancer with a UDP listener on 4789, or a Gateway Load Balancer endpoint. Miss the listener and mirroring fails silently.
• A filter with no rules mirrors nothing. Rules are evaluated lowest number first and the first match decides; rules are directional, with their own numbering per direction, and an ingress rule says nothing about egress.
• Mirrored traffic is VXLAN on UDP 4789 and it counts against the source instance’s bandwidth and PPS allowances. When those run out, the mirrored packets are the ones dropped, so a capture goes quietly incomplete exactly when the instance is busiest.
• PacketLength truncates, which is how you get headers without payload, and how you stay under the target’s MTU. Traffic Mirroring adds 54 bytes of headers for IPv4 and 74 for IPv6, so the largest source MTU that mirrors without truncation is 8947 bytes.
• The meter is per interface per hour, and it runs until the session is deleted, even if you stop the instance or detach the interface.

Next

The rest of the Advanced Networking lab track is listed alongside the written posts at The Exam Room.