Steering Traffic With BGP Attributes

The situation

The network:

• Two Direct Connect connections, both 10 Gbps, terminating in two different AWS Direct Connect locations. Each carries a Transit VIF to a Direct Connect Gateway, which is associated with a Transit Gateway in eu-west-1.
• BGP sessions on each VIF. AWS ASN is 64512 (Amazon-side defaults for Direct Connect). Our on-prem ASN is 65001.
• On-prem advertises a handful of summary prefixes (10.1.0.0/16, 10.2.0.0/16) over both sessions.
• AWS advertises the VPC prefixes attached to the TGW (10.10.0.0/16, 10.20.0.0/16) over both sessions.
• Today’s behaviour: traffic is roughly split across both links because the BGP attributes come out identical on both paths, and both ends do equal-cost multipath by default.

Compliance wants deterministic primary-and-backup behaviour. The SRE lead wants the failover to be automatic and based on BGP state rather than manual route swaps. The network engineer wants to do this without AS-path prepending becoming the only hammer.

The question: which BGP attribute on which side, and what’s the correct configuration?

What actually matters

Before reaching for any specific attribute, it’s worth remembering what BGP’s best-path selection looks like and where the pressure points are.

BGP’s best-path algorithm is a cascade. Simplified (the full list has 13+ tie-breakers; the ones that matter in practice):

1. Highest weight. Cisco-proprietary, local to the router. Rarely used across a whole network because it’s not a standard attribute.
2. Highest local preference, the attribute for influencing outbound traffic from your own AS. Local pref is set when a route enters your AS and propagates inside the AS via iBGP. Higher wins.
3. Locally originated preferred over learned.
4. Shortest AS-path, shorter wins. The lever for influencing other ASes’ view of your routes, because AS-path is one of the few attributes that travels across AS boundaries.
5. Lowest origin type (IGP < EGP < incomplete).
6. Lowest MED. Multi-Exit Discriminator. Meant for hinting to a neighbour AS “if you have multiple links to me, prefer this one to get to these prefixes.” Only valid between direct neighbours; doesn’t propagate further. Lower wins.
7. eBGP over iBGP.
8. Lowest IGP metric to BGP next-hop.
9. Oldest route.
10. Lowest router ID.

The immediately useful pair for our problem:

• Outbound from on-prem to AWS: local preference, set on our routers when we receive AWS’s advertisements. Higher local-pref on the primary VIF’s routes means our iBGP-connected on-prem routers all agree to prefer that path outbound. This is our side’s lever; AWS has no influence here.
• Inbound from AWS to on-prem: the lever we can pull on AWS’s side. Local-pref on AWS isn’t exposed to us directly, but Direct Connect publishes a set of BGP community tags that map to AWS-side local-pref values. Tagging our advertised prefixes with the right community tells AWS “when choosing between the two VIFs for these prefixes, prefer this one.”

Those two levers together, our local-pref on inbound routes, our community tags on outbound routes, give symmetric primary/backup control.

AS-path prepending is the third lever worth naming. Prepending lengthens our AS-path in advertisements, making the prepended route look worse to the receiver. Community tags are better than prepending when the receiver supports them (AWS does) because they’re explicit; prepending is the fallback when community tags aren’t honoured.

MED is the fourth. On Direct Connect, AWS does advertise MED values that reflect AWS-side distance, but we don’t really manipulate MED ourselves, we manipulate communities, and AWS does the MED bookkeeping internally.

What we’ll filter on

1. Controls outbound path from on-prem, do our on-prem routers agree which VIF to send traffic out?
2. Controls inbound path from AWS, does AWS prefer the primary VIF for return traffic?
3. Honoured across AS boundary, does the attribute actually propagate and influence the neighbour?
4. Clean failover on link loss, when the primary BGP session drops, does the backup take over without manual intervention?
5. Auditable / explicit, can we look at a route and know why it’s preferred?

The BGP attribute landscape

1. Weight (Cisco). Local to the router. Highest wins. Useful on a single Cisco box; not useful as a network-wide policy because it’s proprietary and not carried in BGP updates. Fails on network-wide reach.

2. Local preference. Attribute that only lives inside one AS, it’s carried in iBGP updates but stripped at eBGP egress. Set on ingress from the neighbour; propagates inside our AS. The lever for outbound traffic from our AS. Higher wins.

3. AS-path prepending. Lengthen the AS-path of our advertisements by repeating our ASN multiple times. Shorter AS-path wins at the neighbour. Works across AS boundaries but is blunt: we’re polluting the AS-path, and some downstream ASes may have their own best-path logic that ignores or post-processes AS-path (reducing the effect).

4. MED (Multi-Exit Discriminator). Advertised to the neighbour, tells them which of multiple links to prefer for return traffic to the advertised prefixes. Only compared between routes from the same neighbour AS. Direct Connect does advertise MED internally; we rarely set it ourselves.

5. BGP communities (AWS Direct Connect flavour). AWS publishes three outbound communities we can tag our advertisements with:

• 7224:7100 → AWS local-pref 100 (low, “backup”)
• 7224:7200 → AWS local-pref 200 (medium, “default”)
• 7224:7300 → AWS local-pref 300 (high, “primary”)

When we tag our advertised prefixes with these communities, AWS sets local-pref on its side accordingly. Higher wins at AWS; so 7224:7300 on primary and 7224:7100 on backup gives us the symmetric control we want.

AWS also publishes inbound communities (scope tags) that tell us the AWS region of the learned route: 7224:8100 for same-region, 7224:8200 for same-continent, 7224:8300 for global. These are informational, not preference-setting.

Side by side

The pair that wins: local preference on our side for outbound, BGP communities on our side (AWS honours) for inbound.

BGP’s best-path cascade, visualised

The pick(s) in depth

On-prem local-pref on received routes from AWS; community tags on advertised routes to AWS; keep AS-path prepending in reserve as a backup lever.

The configuration, in pseudo-Cisco-ish BGP:

! On-prem side (AS 65001) -- inbound policy from AWS

neighbor 169.254.100.1 remote-as 64512          ! primary VIF
neighbor 169.254.100.1 route-map FROM-AWS-PRI in
neighbor 169.254.100.1 route-map TO-AWS-PRI out

neighbor 169.254.200.1 remote-as 64512          ! backup VIF
neighbor 169.254.200.1 route-map FROM-AWS-BAK in
neighbor 169.254.200.1 route-map TO-AWS-BAK out

route-map FROM-AWS-PRI permit 10
  set local-preference 300                       ! prefer primary outbound

route-map FROM-AWS-BAK permit 10
  set local-preference 100                       ! backup path

route-map TO-AWS-PRI permit 10
  set community 7224:7300                        ! AWS prefers this inbound

route-map TO-AWS-BAK permit 10
  set community 7224:7100                        ! AWS deprecates this inbound

Two levers, symmetric:

Local-preference on FROM-AWS-* route maps. Our on-prem ASBRs, when learning AWS’s VPC prefixes, stamp a higher local-pref on routes learned via the primary VIF. iBGP carries this local-pref to all on-prem routers. Every on-prem router agrees that outbound traffic to AWS prefixes goes via the primary router’s next-hop.

Community 7224:7300 / 7224:7100 on TO-AWS-* route maps. Our outbound advertisements to AWS carry the community. AWS’s side of Direct Connect uses the community to set its own local-pref on the learned route. AWS’s BGP best-path across both VIFs picks local-pref 300 > local-pref 100. Return traffic from AWS takes the primary VIF.

Symmetric: outbound goes via the primary because of our local-pref; inbound comes back via the primary because of AWS’s local-pref, which we set via community.

The backup lever, held in reserve: AS-path prepending on TO-AWS-BAK. If for any reason AWS’s honouring of the 7224 communities stops working as expected, we can also prepend our ASN two or three times on the backup to make the AS-path look worse. Communities win the tie before AS-path even gets compared; prepending is belt-and-braces.

What about ECMP? If the business actually wants both links active (load-shared) and failover when one dies, don’t manipulate local-pref or communities, leave both paths with identical attributes and let BGP multipath do its job. Most networks run multipath on the default config. The primary/backup shape is a deliberate choice, usually because of asymmetric bandwidth or cost, or because one path is a higher-cost resilience path (think: one DX and one VPN).

A worked failover

Monday, 11:47 AM. The primary DX’s cross-connect fails; the primary BGP session goes down within seconds (BGP hold-timer expiry, typically 90-180 seconds, or faster if BFD is configured, which it should be).

Before failure:

• AWS has two routes for 10.1.0.0/16: via primary (local-pref 300) and backup (local-pref 100). Best path = primary.
• On-prem has two routes for 10.10.0.0/16: via primary (local-pref 300) and backup (local-pref 100). Best path = primary.

The primary session dies:

• AWS withdraws the primary route for 10.1.0.0/16. Only the backup remains. Best path = backup.
• On-prem withdraws the primary route for 10.10.0.0/16. Only the backup remains. Best path = backup.

Both sides flip simultaneously (within BGP’s convergence time). Traffic now goes primary → backup on both directions. No manual intervention.

When the primary comes back:

• AWS relearns the primary route with local-pref 300 (from the community). Best path flips back to primary.
• On-prem relearns the primary route with local-pref 300 (from the FROM-AWS-PRI route-map). Best path flips back to primary.

Again, simultaneous. This is the “symmetric primary/backup” behaviour the business asked for.

What’s worth remembering

1. Local-pref controls outbound from your AS. Set on ingress via a route-map; propagates inside the AS via iBGP. Not visible outside the AS. Higher wins.
2. Communities are the lever for influencing AWS’s local-pref. 7224:7300 for primary (local-pref 300), 7224:7200 for default (200), 7224:7100 for backup (100). Applied to your advertisements to AWS via an outbound route-map.
3. AS-path prepending is blunter but universal. Use it when community tags aren’t honoured, or as belt-and-braces on top of communities. Longer AS-path = less preferred; watch out for downstream ASes that deduplicate or ignore prepended paths.
4. MED is for multi-link same-neighbour hinting; Direct Connect handles it internally, so rarely a lever you set.
5. Symmetric routing = symmetric attributes. Set both sides of the preference (your local-pref + your communities) so outbound and inbound return use the same path. Asymmetric return breaks stateful firewalls.
6. BGP’s best-path is a cascade. Know the order; in practice local-pref and AS-path are the two that decide most real-world contests, with communities the polite way to set AS local-pref.
7. BFD is a must for fast failover. BGP default hold-timer is 90s-ish; BFD drops detection to seconds. On Direct Connect, enable BFD on the BGP session.
8. ECMP vs primary/backup is a design choice. Identical attributes on both paths = ECMP (load share + failover). Different attributes = deterministic primary + backup. Decide which one the business actually wants before configuring.

BGP is not magic, and on Direct Connect only a small subset of attributes are levers. Local-pref on our side of the border decides which link we send traffic out. Community 7224:7300 / 7224:7100 on advertisements to AWS decides which link AWS sends traffic back on. Two knobs, one per direction, both explicit. The work isn’t memorising the full 13-step best-path algorithm, it’s picking the two attributes that match the two directions and setting them consistently.