ANS Lab 08 - Fail an inspection path and prove it holds

Scaffold: 3/5. Two spoke VPCs, a transit gateway with a spoke route table and an inspection route table, an inspection VPC with an AWS Network Firewall endpoint in each of two Availability Zones, a target instance, a probe function and a Network Insights path are all built. Two things are wrong on purpose: the inspection VPC’s two transit gateway subnets share one route table with nothing in it, and appliance mode is disabled on the inspection attachment. You write the route table, break one zone, watch the probe, and rebuild until the number holds through the break.

The scenario

Everything between spoke A (10.30.0.0/16) and spoke B (10.40.0.0/16) goes through Network Firewall in an inspection VPC. The firewall has an endpoint in each of two zones, and the design document says that losing one zone’s inspection path must not take spoke-to-spoke traffic down. Nobody has tested that sentence. Until somebody does, it is a claim.

The probe is a Lambda function in spoke A, in the first zone, that opens a fresh TCP connection to a target in spoke B, in the second zone, once a second, and reports how many completed. Fresh connections are deliberate. Each one is a new five-tuple, so the transit gateway hashes each one on its own, and a build that is only half working comes out near 50% rather than as a clean pass or fail. The target is in the second zone on purpose: the request always originates in zone A and the reply always originates in zone B, so the asymmetry the lab is about happens every time rather than sometimes.

The probe also labels how each failure failed. A connect timeout means the packets went somewhere and nothing came back. That is the failure that lands on the application team first, because from the application’s side it looks like a slow dependency.

What’s provided

Deploying src/template.yaml as shipped succeeds. Everything comes up. The probe gets 0%, all timeouts, because traffic reaches the inspection VPC and stops at a route table with nothing in it.

Costs

This is the most expensive lab in the track to leave running. Two firewall endpoints at $0.395 an hour each and three transit gateway attachments at $0.05 each (US East rates) come to close to a dollar an hour before the two t3.micro instances, and the meter runs whether or not the probe is doing anything. Finish it in one sitting and tear it down. Each Reachability Analyzer run is $0.10; a full pass through this lab is four or five of them. deploy.sh tags the stack for the lab reaper, which deletes it after 24 hours if you forget.

Step 1: deploy and watch it fail

# Defaults: stack ans-lab-08, region ap-southeast-2.
./scripts/deploy.sh
./scripts/test.sh

The first deploy takes 10 to 15 minutes; the firewall endpoints and the three attachments are the slow parts. The probe comes back at 0% with every failure labelled timeout, and the verdict says FAIL. Run the analysis too, for the “before” artefact:

ANALYSE=1 ./scripts/test.sh 10

It reports found: false with an explanation at the inspection VPC’s route table: the packet arrived from the transit gateway and the route table it was handed to has no route for 10.40.0.0/16.

Step 2: the fix that passes, and why it should not

Open src/template.yaml and find the TODO. The quickest thing that makes the probe pass is one default route, in the shared table, to the zone B endpoint:

ZoneBEndpointRoute:
  Type: AWS::EC2::Route
  Properties:
    RouteTableId: !Ref InspectionTgwRouteTable
    DestinationCidrBlock: 0.0.0.0/0
    VpcEndpointId: !GetAtt FirewallEndpoints.EndpointIdB

Deploy and test. 100%. Run the analysis and it says reachable, with the zone B firewall endpoint in the path. This build would be signed off.

Now fail it. The stack outputs the route table that serves zone B; delete its default route:

RTB=$(aws cloudformation describe-stacks --stack-name ans-lab-08 \
  --query "Stacks[0].Outputs[?OutputKey=='ZoneBRouteTableId'].OutputValue" --output text)
aws ec2 delete-route --route-table-id "$RTB" --destination-cidr-block 0.0.0.0/0
./scripts/test.sh

0%. Every failure is a timeout. Nothing refused the connection, nothing sent a reset, and the probe’s own kernel had a route the whole time. Losing one zone’s route took out both zones, because both zones’ transit gateway subnets read the same table, and every flow in the estate went through one endpoint.

Put the route back by hand. CloudFormation still believes the route exists, so a redeploy will not recreate it:

EP_B=$(aws cloudformation describe-stacks --stack-name ans-lab-08 \
  --query "Stacks[0].Outputs[?OutputKey=='EndpointIdB'].OutputValue" --output text)
aws ec2 create-route --route-table-id "$RTB" \
  --destination-cidr-block 0.0.0.0/0 --vpc-endpoint-id "$EP_B"

Step 3: one route table per zone

Replace the shared table with two: each associated with its own zone’s transit gateway subnet, each with a default route to the firewall endpoint in the same zone.

InspectionTgwRouteTableA:
  Type: AWS::EC2::RouteTable
  Properties:
    VpcId: !Ref InspectionVpc

InspectionTgwSubnetAAssociation:
  Type: AWS::EC2::SubnetRouteTableAssociation
  Properties:
    SubnetId: !Ref InspectionTgwSubnetA
    RouteTableId: !Ref InspectionTgwRouteTableA

ZoneAEndpointRoute:
  Type: AWS::EC2::Route
  Properties:
    RouteTableId: !Ref InspectionTgwRouteTableA
    DestinationCidrBlock: 0.0.0.0/0
    VpcEndpointId: !GetAtt FirewallEndpoints.EndpointIdA

# ...and the same three for zone B, pointing at EndpointIdB.

Update the two outputs at the bottom of the template so ZoneARouteTableId and ZoneBRouteTableId name the new tables. Leave ApplianceModeSupport: disable for now, deploy, and test.

0% again, and this time every route is present. The request leaves zone A, so the transit gateway keeps it in zone A and it meets endpoint A. The reply leaves zone B, the gateway keeps that in zone B, and it meets endpoint B, which holds no state for the flow. The stream exception policy is at its default of drop, so the reply dies there.

Now set ApplianceModeSupport: enable on InspectionAttachment and deploy. It is an in-place update. Test: 100%. The gateway now hashes each flow to one interface in the inspection VPC and uses that interface in both directions for the life of the flow.

Step 4: fail a zone, and make it hold

Delete zone B’s default route again (ZoneBRouteTableId now names the zone B table) and test:

aws ec2 delete-route --route-table-id "$RTB_B" --destination-cidr-block 0.0.0.0/0
./scripts/test.sh

PARTIAL, somewhere around half. The flows the hash sends to zone A complete; the ones it sends to zone B time out. The transit gateway does not read your route tables. It moves traffic out of a zone only when the zone fails or when the attachment has no subnet there, so a broken route in one zone’s inspection path is invisible to it and half the estate stays broken until somebody acts.

The action is to take zone B out of the attachment:

ATT=$(aws cloudformation describe-stacks --stack-name ans-lab-08 \
  --query "Stacks[0].Outputs[?OutputKey=='InspectionAttachmentId'].OutputValue" --output text)
SUBNET_B=$(aws cloudformation describe-stacks --stack-name ans-lab-08 \
  --query "Stacks[0].Outputs[?OutputKey=='InspectionTgwSubnetBId'].OutputValue" --output text)
aws ec2 modify-transit-gateway-vpc-attachment \
  --transit-gateway-attachment-id "$ATT" --remove-subnet-ids "$SUBNET_B"
aws ec2 describe-transit-gateway-vpc-attachments \
  --transit-gateway-attachment-ids "$ATT" \
  --query 'TransitGatewayVpcAttachments[0].State'   # wait for "available"
ANALYSE=1 ./scripts/test.sh

100%, with zone B’s route still missing, and the analysis reports reachable through the zone A endpoint. That pair, the analysis from step 1 and this one, with the probe rate beside them, is what an auditor asking “did you test failover” gets handed.

Read the analysis with care. It reports the shortest reachable path, so with the route deleted and both subnets still attached it also came back reachable, through zone A, and said nothing about the half of the flows the hash was sending to zone B. The probe rate is the evidence; the analysis is the document.

To restore zone B, add the subnet back and recreate the route. Or leave it: teardown.sh puts the subnet back before it deletes anything.

Run it

./scripts/deploy.sh              # deploys src/template.yaml, then uploads the probe
./scripts/test.sh                # 30 connections, one a second, and a verdict
./scripts/test.sh 60             # more connections, for a steadier rate
ANALYSE=1 ./scripts/test.sh      # also run the Reachability Analyzer path check
./scripts/teardown.sh            # deletes everything with a meter on it

test.sh prints the probe’s JSON, then a verdict: PASS at 95% or above, FAIL at 5% or below with the two builds that produce it, and PARTIAL in between, which is what one broken zone looks like.

If it fails

Reveal the solution

Deploy the complete reference template without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

Next

The rest of the Advanced Networking Specialist material is in The Exam Room.