ANS Lab 09 - Take down a BGP session on purpose

Scaffold: 4/5. A cloud VPC behind a virtual private gateway, a stand-in on-premises network with a router running strongSwan and FRR, a Site-to-Site VPN connection between them with dynamic routing, and a script that does what the Direct Connect Resiliency Toolkit failover test does are all built. Both IPsec tunnels come up. The router runs BGP over one of them. You run the failover test and watch it fail, add the second peering, and run it again.

The scenario

The resiliency test everyone skips is the hybrid one. The second virtual interface is on the diagram, and proving it carries traffic appears to need someone in a colocation cage unplugging a cable, so it stays unproven until the day the primary drops and the whole company finds out at once.

The Direct Connect Resiliency Toolkit failover test exists so nobody has to unplug anything. You choose a virtual interface, its BGP peering session and a duration; AWS puts that peering session in the down state for that long, and restores it when the time is up or when you stop the test. If the redundant virtual interface is really carrying the same prefixes, traffic moves and the application never notices. If it is not, the test fails and traffic stops, which is the finding the test exists to produce. Only the account that owns the virtual interface can start one, CloudTrail records who did, and the test history is kept for 365 days with one of four states: In progress, Completed, Cancelled, Failed.

None of that can be built in a lab account. A virtual interface needs a connection and a connection needs a cross-connect in a colocation facility. What can be built is the thing the test acts on. A Site-to-Site VPN connection with dynamic routing gives you two tunnels and two BGP peering sessions to the same virtual private gateway for about five cents an hour, and the mechanics are identical: one session goes down on purpose, its prefix is withdrawn, and either the other session is still advertising it or the route is gone.

What’s provided

• src/template.yaml - the cloud VPC (10.60.0.0/16) with a virtual private gateway, route propagation into its route table, and a t3.micro serving a page on port 80; the on-premises stand-in (192.168.10.0/24) with an Ubuntu router carrying an Elastic IP; a customer gateway (ASN 65010) and one Site-to-Site VPN connection with StaticRoutesOnly: false. The router’s user data reads the VPN connection’s configuration, brings up IPsec on both tunnels, and writes an FRR neighbour for every tunnel named in the BgpPeers SSM parameter. As shipped the parameter names tunnel1 only, with a clearly-marked TODO on the line below it.
• scripts/deploy.sh - deploys the stack, then has the router rebuild its BGP configuration from the parameter, so a redeploy after you edit it reaches FRR.
• scripts/failover-test.sh - the stand-in for the Toolkit. start takes a named session down for a number of minutes, stop restores it early and records Cancelled, history prints every test run on the router.
• scripts/test.sh - the whole run: confirms the path, starts a two-minute test on tunnel1, watches from the router and from the AWS side, waits for the restore, prints the CloudWatch metric and the history, and gives a verdict.
• scripts/teardown.sh - deletes everything.
• solution/template.yaml - the complete build, with both tunnels peered.

Deploying src/template.yaml as shipped succeeds. Both tunnels are up at IPsec, one BGP session is established, 192.168.10.0/24 is propagated into the cloud route table, and the router can fetch the page. It looks redundant. It is not.

Costs

The VPN connection is metered by the hour from the moment it exists, about five cents, whether or not a tunnel is up. The two t3.micro instances and the Elastic IP add a few cents more. Under ten cents an hour in total, and nothing here needs to run overnight: deploy.sh tags the stack for the lab reaper, which deletes it after 24 hours if you forget.

Step 1: deploy, and run the test as shipped

# Defaults: stack ans-lab-09, region ap-southeast-2.
./scripts/deploy.sh
./scripts/test.sh

The deploy takes about ten minutes. The VPN connection has to become available before the router can read its tunnel addresses and pre-shared keys, and the router installs strongSwan and FRR before it can use them.

test.sh first shows you where things stand: the tunnel telemetry from the AWS side, show bgp summary from the router, the propagated route, and an HTTP probe from the router to the cloud instance. Then it starts the failover test on tunnel1 and watches. With the unedited template the timeline reads:

  03:14:07  sessions=1  route-to-cloud=yes  http=200
  03:14:12  sessions=0  route-to-cloud=no   http=000
  03:14:17  sessions=0  route-to-cloud=no   http=000
  ...
  03:14:12  192.168.10.0/24 propagated: no

The session went down, the router withdrew its prefix, the virtual private gateway dropped the propagated route, and the router lost the route it had learned to the cloud VPC. Two minutes of nothing. test.sh says FAIL and names the gap.

Step 2: add the second peering

Open src/template.yaml, find the BgpPeers parameter, and uncomment tunnel2:

      Value: |
        tunnel1
        tunnel2

Nothing else changes. IPsec was already up on both tunnels; what was missing was a BGP session inside the second one, and the bootstrap writes the FRR neighbour for any tunnel the parameter names. Then:

./scripts/deploy.sh
./scripts/test.sh

What success looks like

Before the test, show bgp summary lists two neighbours in Established, and the AWS-side telemetry shows an accepted route on both tunnels. During the window the timeline reads:

  03:31:40  sessions=2  route-to-cloud=yes  http=200
  03:31:45  sessions=1  route-to-cloud=yes  http=200
  03:31:50  sessions=1  route-to-cloud=yes  http=200

and every route-table sample from the AWS side says propagated: yes. Then:

PASS: tunnel1's BGP session was down for 2 minute(s) and traffic
kept flowing. The prefix stayed in the cloud route table because the
second session was still advertising it, and the router still held a
route to 10.60.0.0/16 learned over tunnel2. The redundant path is real.

After the verdict, two more things are worth reading. The test history shows the run as Completed with its start and end times. The CloudWatch TunnelState metric for the tunnel under test dips to 0 for the window while the other stays at 1, because for a BGP VPN the metric is 1 only while the session is established, and the tunnel’s IPsec never went down. Metric data can lag a few minutes; if the dip is not there yet, query it again later.

Cancel one early

The Toolkit lets you stop a test before the duration is up, which matters when the application starts misbehaving and two hours of degraded service is not what you signed up for. The stand-in does the same:

./scripts/failover-test.sh start tunnel2 30    # half an hour, on the other session
./scripts/failover-test.sh stop                # restore it now
./scripts/failover-test.sh history             # recorded as Cancelled

Each start and stop is a Systems Manager command, and the CloudTrail SendCommand event carries who ran it and when. On Direct Connect the events are StartBgpFailoverTest and StopBgpFailoverTest, and that record is what an auditor asking for evidence of a failover test is looking for.

If it fails

• No BGP session is established yet. The router is still bootstrapping. Wait a few minutes and run test.sh again. If it stays at zero, read /var/log/lab-bootstrap.log on the router through Session Manager.
• The session is up but the prefix is not in the cloud route table. Route propagation is what turns an accepted prefix into a route the VPC forwards on. The template enables it; if you removed the VPNGatewayRoutePropagation resource while experimenting, put it back.
• NOT_A_PEER when starting a test on tunnel2. The parameter still names only tunnel1, or you edited it and did not redeploy. deploy.sh is what pushes the change to FRR.
• Traffic still fails with both tunnels peered. Check show bgp summary shows both neighbours Established before the test starts. A neighbour that is configured and stuck in Active or Connect is not carrying anything, and the test is right to fail.
• The history says Failed. The neighbour did not go down when asked. The Toolkit uses the same word for the same reason: the test did not run for the time specified, usually because of the router.

Reveal the solution

Deploy the complete reference template without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• The Resiliency Toolkit failover test puts a chosen virtual interface’s BGP peering session in the down state for a duration you set (180 minutes by default, up to 4,320) and restores it when the time is up or when you stop it. Only the account that owns the virtual interface can run it.
• A tunnel that is up at IPsec and has no BGP session inside it is not a redundant path. The prefix is advertised over sessions, not tunnels, and when the only session drops the prefix goes with it.
• The evidence lives in four places: the BGP session state, the route table the prefix propagates into, the CloudWatch metric (TunnelState here, VirtualInterfaceBgpStatus on a virtual interface), and the test history, which the Toolkit keeps for 365 days as In progress, Completed, Cancelled or Failed. CloudTrail is the record of who started it.
• Do not run the Toolkit test during a Direct Connect maintenance period. The session may be restored early, during or after the maintenance, and a test that ended before its time proves nothing.

Next

The rest of the Advanced Networking lab track is listed alongside the written posts at The Exam Room.