ANS Lab 05 - Resolve a name that lives somewhere else

Scaffold: 3/5. Two VPCs on a transit gateway, a private hosted zone on each side, an inbound endpoint standing in for the datacentre’s DNS servers, and a Lambda function in each VPC that speaks DNS and reports which server answered. Two gaps are left, both in src/template.yaml. The first is the outbound endpoint, forwarding rule and association that let the AWS VPC resolve the datacentre’s zone. The second is the inbound endpoint that lets the datacentre resolve the AWS zone. Then you move every endpoint address into one Availability Zone, take that zone away, and watch what each direction does.

The scenario

The datacentre runs a zone called corp.lab.internal, served by its own DNS servers, and db.corp.lab.internal lives in it. The AWS VPC runs a private hosted zone called aws.lab.internal, and orders.aws.lab.internal lives in that. Each side can resolve its own names and neither can resolve the other’s.

A VPC’s Resolver, the address at the base of the VPC range plus two, is only reachable from inside that VPC. Route 53 Resolver endpoints exist to carry queries across that boundary. An outbound endpoint gives the Resolver a pair of network interfaces to forward queries from, and a forwarding rule says which domain goes that way and to which addresses. An inbound endpoint gives the outside world a pair of addresses to send queries to, which the endpoint hands to the VPC’s Resolver. Nothing on either side is exposed beyond those addresses, and that is what makes hybrid DNS a routing problem rather than a trust problem.

In this lab the datacentre is another VPC. Its “DNS servers” are an inbound endpoint at 10.20.1.53 and 10.20.2.53, already built. Its conditional forwarder, the thing a Windows or BIND server would run to send aws.lab.internal to AWS, is a Lambda function that queries a server you name.

What’s provided

• src/template.yaml - both VPCs, each with a workload subnet per AZ and a /28 attachment subnet per AZ; a transit gateway attached to both; the two private hosted zones with one A record each; the datacentre inbound endpoint and its security group; a security group for the AWS-side endpoints, already open on port 53 both ways; a Lambda function in each VPC; and two TODO blocks where the AWS-side endpoints go.
• src/handler.py - the lookup function. It is complete. It sends one A query over UDP to a server you choose, or to the VPC’s own Resolver by default, and returns the outcome (NOERROR, NXDOMAIN, SERVFAIL, TIMEOUT), the answers and the round-trip time. The same code is deployed in both VPCs.
• scripts/ - deploy, test, and teardown. deploy.sh runs a lab-specific preflight first: it counts the Resolver endpoints already in the Region against the quota, and on an update checks the workload subnets still have addresses free.
• solution/template.yaml - the complete, correct template.

Deploying src/template.yaml as shipped succeeds. Both zones answer inside their own VPC, and nothing crosses.

Your first task: the AWS VPC resolves the datacentre

Deploy and run the test before changing anything:

# Defaults: stack ans-lab-05, region ap-southeast-2. Allow 8 to 12 minutes.
./scripts/deploy.sh
./scripts/test.sh

The first lookup, db.corp.lab.internal asked of 10.10.0.2, comes back NXDOMAIN. Read that carefully: the Resolver did answer, and the answer was “no such name”. No private zone matched and no rule matched, so the query went to the public internet, which has never heard of lab.internal. A missing forwarding rule does not look like an error. It looks like the name does not exist.

Fill the first TODO in src/template.yaml with three resources: an OUTBOUND endpoint with its two addresses pinned (the template comment gives the exact block), a FORWARD rule for corp.lab.internal targeting 10.20.1.53 and 10.20.2.53 on port 53 through that endpoint, and a rule association binding the rule to AwsVpc. Redeploy and test. The lookup now returns 10.20.2.10, and the ms column shows the round trip through two endpoints and a transit gateway.

Your second task: the datacentre resolves the AWS VPC

The second half of the test asks orders.aws.lab.internal of each address in AwsInboundIps, from the datacentre-side function. Both time out, because nothing is listening at 10.10.1.53 or 10.10.2.53 yet.

Fill the second TODO with an INBOUND endpoint carrying exactly those two addresses. There is no rule and no association for an inbound endpoint: it hands every query it receives to the AWS VPC’s Resolver, and that Resolver already holds aws.lab.internal. Redeploy and test. Both addresses answer 10.10.2.10.

Your third task: take an Availability Zone away

Now the part the lab is for. Redeploy with every AWS-side endpoint address in subnet A:

LAYOUT=one-az ./scripts/deploy.sh
./scripts/test.sh

It passes. Two addresses in one subnet is a legal endpoint, and every console and every test says it is fine. Now lose subnet A in both VPCs:

LAYOUT=one-az FAIL_AZ=a ./scripts/deploy.sh
./scripts/test.sh

FAIL_AZ=a associates a network ACL with no allow rules with subnet A on both sides, which drops everything in and out of that subnet. Both directions now fail. The AWS-side lookup returns SERVFAIL (or TIMEOUT, if the Resolver was still retrying when the function stopped waiting): the rule matched, the Resolver forwarded, and no reply came back through interfaces in a subnet that drops everything. The datacentre-side lookups both TIMEOUT. Then put the layout back while leaving the zone down:

LAYOUT=two-az FAIL_AZ=a ./scripts/deploy.sh
./scripts/test.sh

The AWS-side lookup answers again. The datacentre-side shows one address answering and one dark, with a WARN rather than a FAIL, because a conditional forwarder with two servers configured retries the second when the first goes quiet. That is the state two Availability Zones are for.

Finish by restoring the zone: FAIL_AZ=none ./scripts/deploy.sh.

What success looks like

./scripts/test.sh prints one line per lookup, then:

PASS: names resolve in both directions. The AWS VPC reaches the datacentre
zone through the outbound endpoint and its rule; the datacentre reaches the
AWS zone through the inbound endpoint.

If it fails

• NXDOMAIN for db.corp.lab.internal. The query went to the public internet. Either the rule does not exist or it is not associated with the AWS VPC. A rule with no association is a rule that applies nowhere.
• SERVFAIL for db.corp.lab.internal. The rule matched and forwarding happened, but no target answered. Check the endpoint’s interfaces are OPERATIONAL, the security group allows port 53 out, the route to 10.20.0.0/16 points at the transit gateway, and the targets are the datacentre addresses, not the datacentre’s .2.
• TIMEOUT against an inbound address. Nothing is listening there, or a network ACL is dropping the packets. Check FailedAz in the stack outputs before you check anything else.
• The deploy fails with LimitExceededException. The Region already holds enough endpoints to hit the quota of 4. The preflight should have caught it; if it did not, aws route53resolver list-resolver-endpoints shows what is there.
• Teardown sits in DELETE_FAILED. A subnet or security group still has a network interface attached, from an endpoint or a Lambda function that has not released it yet. teardown.sh waits and retries; run it again if it gave up.

Reveal the solution

Deploy the complete reference template without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• A VPC’s Resolver is reachable only from inside the VPC. Everything that crosses a VPC boundary in either direction goes through an endpoint: outbound for queries leaving, inbound for queries arriving. Each endpoint is two or more network interfaces, billed by the hour.
• Forwarding is three things, and all three are needed: an outbound endpoint to forward from, a rule naming the domain and the targets, and an association binding the rule to a VPC. A missing rule shows up as NXDOMAIN, not as an error, because the query went to the internet instead.
• An inbound endpoint needs no rule. It hands queries to the VPC’s Resolver, so it can answer for any private hosted zone associated with that VPC. The on-premises side points a conditional forwarder at its addresses.
• Two addresses is the minimum an endpoint accepts, and two addresses in one Availability Zone passes every check while sharing one failure. Spread them, and the Resolver’s redundant queries and the forwarder’s retry carry the loss of one zone.
• Endpoints and VPC-attached functions release their network interfaces after the resource is gone, not with it. A teardown that deletes subnets straight after has to wait for the interfaces, or it fails.

Next

The rest of the Advanced Networking material is in The Exam Room.