ANS Lab 06 - Catch a broken path before anyone notices

Scaffold: 3/5. Two VPCs on a transit gateway, an instance in each, the Network Insights path holding the intent, a CloudTrail trail, an EventBridge rule, a checker function and an alarm are all built. Two gaps are left: one route in the app VPC’s route table, and the analysis call in the handler. You run the analysis by hand first, read the explanation code, fix the route, then write the code that turns the same check into standing intent.

The scenario

The app tier in 10.30.0.0/16 has to reach a shared service in 10.40.0.0/16 on TCP 443, through a transit gateway. That sentence is a requirement, and today it is enforced by nobody. Somebody deletes a route at 4pm on a Thursday, and the first thing that notices is a customer.

VPC Reachability Analyzer turns the sentence into something a machine can check. You define the path once, source to destination with a protocol and a port, and every analysis against it answers one question: given the configuration right now, can a packet get from here to there. When the answer is no, it names the component that blocks it.

The catch, and the reason one step of this lab is spent on it: the analysis reads configuration. It sends no packets. A path can be reachable and still carry nothing, so a check that is trusted too far is worse than no check at all.

What’s provided

• src/template.yaml - two VPCs with a subnet, route table and security group each; a transit gateway with default association and propagation enabled and a VPC attachment for each side; a t3.micro in each subnet; the AWS::EC2::NetworkInsightsPath holding the intent; a CloudTrail trail and its bucket; the checker function and its role; the EventBridge rule; and the alarm on the verdict. There is a clearly-marked TODO where the app VPC’s forward route goes.
• src/handler.py - the checker. Everything is written except run_analysis(), which raises NotImplementedError.
• scripts/ - deploy, test, and teardown.
• solution/ - the complete template and handler.

Deploying src/template.yaml as shipped succeeds. Everything comes up; the path is simply unreachable, because the app VPC’s route table has only its local route.

Costs

Two transit gateway attachments and two t3.micro instances are metered by the hour, so this is a lab to finish and tear down, not to leave running overnight. Reachability Analyzer is charged per analysis run and Network Access Analyzer per network interface analysed; a full pass through this lab is a handful of runs. The trail’s first copy of management events is free. deploy.sh tags the stack for the lab reaper, which deletes it after 24 hours if you forget.

Step 1: deploy and ask the question by hand

# Defaults: stack ans-lab-06, region ap-southeast-2.
./scripts/deploy.sh

The transit gateway attachments take a few minutes. When the deploy finishes it prints the path id. Run one analysis against it from the CLI, before you write any code:

PATH_ID=$(aws cloudformation describe-stacks --stack-name ans-lab-06 \
  --query "Stacks[0].Outputs[?OutputKey=='PathId'].OutputValue" --output text)

ANALYSIS=$(aws ec2 start-network-insights-analysis \
  --network-insights-path-id "$PATH_ID" \
  --query 'NetworkInsightsAnalysis.NetworkInsightsAnalysisId' --output text)

aws ec2 describe-network-insights-analyses \
  --network-insights-analysis-ids "$ANALYSIS" \
  --query 'NetworkInsightsAnalyses[0].{status:Status,found:NetworkPathFound,codes:Explanations[].ExplanationCode}'

Repeat the describe until status is succeeded. You should see found: false and one code:

{ "status": "succeeded", "found": false, "codes": ["NO_ROUTE_TO_DESTINATION"] }

NO_ROUTE_TO_DESTINATION means the route table consulted at that hop has no applicable route to the destination. Look at the full Explanations array rather than just the code, because the explanation carries the route table it is complaining about:

aws ec2 describe-network-insights-analyses \
  --network-insights-analysis-ids "$ANALYSIS" \
  --query 'NetworkInsightsAnalyses[0].Explanations'

Step 2: fix the route

Open src/template.yaml, find the TODO, and add the forward route:

AppForwardRoute:
  Type: AWS::EC2::Route
  DependsOn: [AppAttachment, SharedAttachment]
  Properties:
    RouteTableId: !Ref AppRouteTable
    DestinationCidrBlock: 10.40.0.0/16
    TransitGatewayId: !Ref TransitGateway

The DependsOn is load-bearing: a route pointing at a transit gateway fails to create until the VPC has an attachment. Redeploy and run the analysis again; it should come back found: true.

Step 3: write the checker

Fill in run_analysis() in src/handler.py. Three moves:

1. ec2.start_network_insights_analysis(NetworkInsightsPathId=path_id) and keep the returned NetworkInsightsAnalysisId. The call returns immediately with status running; there is no waiter.
2. Poll ec2.describe_network_insights_analyses(NetworkInsightsAnalysisIds=[...]) until Status leaves running. Valid values are running, succeeded and failed.
3. Return the shape _verdict() defines: NetworkPathFound as reachable, the set of ExplanationCode values from Explanations, and the length of ForwardPathComponents as the hop count.

Then deploy and test:

./scripts/deploy.sh
./scripts/test.sh

Step 4: watch it catch a real change

The EventBridge rule fires the checker on routing and security-group changes. Break the path the way an on-call engineer would:

RTB=$(aws cloudformation describe-stacks --stack-name ans-lab-06 \
  --query "Stacks[0].Outputs[?OutputKey=='AppRouteTableId'].OutputValue" --output text)

aws ec2 delete-route --route-table-id "$RTB" --destination-cidr-block 10.40.0.0/16

The DeleteRoute call goes to CloudTrail, CloudTrail delivers it to EventBridge, the rule invokes the checker, the analysis comes back unreachable, and the function publishes PathReachable=0. Delivery is not instant: CloudTrail management events typically take a few minutes to arrive, and the analysis itself takes tens of seconds on top of that. Watch it land:

aws logs tail /aws/lambda/$(aws cloudformation describe-stacks \
  --stack-name ans-lab-06 \
  --query "Stacks[0].Outputs[?OutputKey=='FunctionName'].OutputValue" \
  --output text) --follow

Then put the route back (./scripts/deploy.sh) and confirm the next run publishes a 1.

Worth trying while you are here: revoke the ingress rule on the shared security group instead of deleting the route, and read the code you get. It is a different one, and it names a different component.

SG=$(aws ec2 describe-security-groups \
  --filters "Name=tag:Name,Values=ans-lab-06-shared" \
  --query 'SecurityGroups[0].GroupId' --output text)

aws ec2 revoke-security-group-ingress --group-id "$SG" \
  --protocol tcp --port 443 --cidr 10.30.0.0/16

Step 5: ask the opposite question once

Reachability Analyzer answers “can this path work”. Network Access Analyzer answers “does a path exist that should not”. Different tool, different question, and the second one is how you check segmentation rather than connectivity. Run one scope by hand:

cat > /tmp/ans-lab-06-scope.json <<'JSON'
[
  {
    "Source": { "ResourceStatement": { "ResourceTypes": ["AWS::EC2::InternetGateway"] } },
    "Destination": { "ResourceStatement": { "ResourceTypes": ["AWS::EC2::NetworkInterface"] } }
  }
]
JSON

SCOPE=$(aws ec2 create-network-insights-access-scope \
  --match-paths file:///tmp/ans-lab-06-scope.json \
  --query 'NetworkInsightsAccessScope.NetworkInsightsAccessScopeId' --output text)

SCOPE_ANALYSIS=$(aws ec2 start-network-insights-access-scope-analysis \
  --network-insights-access-scope-id "$SCOPE" \
  --query 'NetworkInsightsAccessScopeAnalysis.NetworkInsightsAccessScopeAnalysisId' \
  --output text)

aws ec2 get-network-insights-access-scope-analysis-findings \
  --network-insights-access-scope-analysis-id "$SCOPE_ANALYSIS"

This lab’s VPCs have no internet gateway, so in an otherwise empty account the findings come back empty, which is the answer you wanted. In an account with other work in it, every finding is a network interface something on the internet can reach. Delete the scope when you are done:

aws ec2 delete-network-insights-access-scope --network-insights-access-scope-id "$SCOPE"

Where the analysis stops

The tool models configuration. It does not send a packet, so it cannot tell you:

• whether anything is listening on 443 at the far end. A path to an instance with no process bound to the port reads as reachable.
• whether a middlebox drops the traffic at runtime. Reachability Analyzer reads Network Firewall’s stateful and stateless 5-tuple rules, but not domain lists, Suricata rules or rule options, and it does not consider the health of load balancer targets.
• anything on the far side of a Direct Connect or a VPN. The analysis stops at the AWS edge: the virtual private gateway or the transit gateway attachment is where the model ends, so a path can be clean all the way to the edge while the customer router drops it.
• whether the return path works. With TCP, when a path traverses a transit gateway route table, only forward traffic is analysed, so run the reverse path as a second analysis if asymmetry is possible.

So the standing check is one instrument among three. Pair it with a flow log on both attachment-adjacent interfaces, which records what actually moved, and a real connection test from a host, which is the only thing that proves a process answered.

Reveal the solution

Deploy the complete reference build without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

Tear it down

./scripts/teardown.sh

The script deletes the analyses the checker ran and empties the trail bucket before deleting the stack, because CloudFormation owns neither. The transit gateway attachments are the slow part; let the wait finish.

What you just learned

• A Network Insights path is connectivity intent written down. Creating it is free; you are charged per analysis run against it, so a standing check costs what you decide to spend on triggers.
• NO_ROUTE_TO_DESTINATION, ENI_SG_RULES_MISMATCH and SUBNET_ACL_RESTRICTION are the three codes that cover most broken paths, and each explanation names the component it is complaining about.
• Reachability Analyzer needs more than the ec2:*NetworkInsights* actions: it reads dozens of describe APIs and calls the internal Tiros service, which is why AmazonVPCReachabilityAnalyzerFullAccessPolicy exists.
• EventBridge only sees AWS API Call via CloudTrail events when a trail in the account is actively logging management events. Without one, the rule matches nothing and never says so.
• Configuration analysis and data-plane evidence answer different questions. Reachability Analyzer says the path is allowed; flow logs and a real connection say whether it works.

Next

The rest of the Advanced Networking Specialist material is in The Exam Room.