Lab: Fan Out Events With SNS and SQS

This is the first lab in the SA Pro hands-on track. The reference posts argue the decisions; this stands one up. It pairs with the event-driven architecture post, which walks the whole decoupling landscape. Here we build the smallest honest slice of it: fan-out with a filter. The full lab is in lab-sa-pro-01-event-fan-out.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every SA Pro lab tags its stack for the reaper on deploy.

The scenario

An order service publishes an event for every order. Two teams consume them. The fulfilment team wants every order. The fraud team only wants the large ones, the orders worth 1000 or more, and they do not want to pay to receive and then discard the small ones.

The lazy design is one queue that both teams read, with the fraud team filtering in its own code. That couples the two teams to the same queue and makes the fraud team pay to pull and drop every small order. The design that scores on a Professional paper is one topic and two queues, with a filter policy on the subscription that matters, so Amazon SNS drops the events the fraud team does not want before they are ever delivered.

What you’re given

CloudFormation builds the topic, both queues, a shared dead-letter queue, the queue policy that lets SNS deliver to SQS, and both subscriptions. The catch-all queue’s subscription has no filter, so it receives every order. The high-value queue’s subscription has a FilterPolicy, and that policy is a template parameter fed from a file you control:

HighValueSubscription:
  Type: AWS::SNS::Subscription
  Properties:
    TopicArn: !Ref OrderEvents
    Protocol: sqs
    Endpoint: !GetAtt HighValueQueue.Arn
    RawMessageDelivery: true
    FilterPolicyScope: MessageAttributes
    FilterPolicy: !Ref HighValueFilterPolicy

The gap is src/filter-policy.json. It ships as {}, an empty policy, which matches every message. So out of the box the fan-out works but nothing is filtered: both queues receive all four orders.

Your task

Write a filter policy that matches only orders whose order_value message attribute is 1000 or more. SNS filter policies are a map from attribute name to a list of match rules, and they support numeric comparisons:

{
  "order_value": [{ "numeric": [">=", 1000] }]
}

Two details decide whether this works. First, the filter matches on the message attributes, not the body, which is why the publisher sends order_value as a Number attribute and not only inside the JSON payload. If the value lived only in the body, you would set FilterPolicyScope: MessageBody and write the policy against the body’s shape instead. Second, the attribute is compared as a number, so the rule is {"numeric": [">=", 1000]}, not a string match; a string match on "1000" would miss 1500 because lexical order is not numeric order.

Run it

./scripts/deploy.sh          # deploys with your filter-policy.json
./scripts/test.sh            # publishes four orders and counts each queue
./scripts/teardown.sh        # deletes everything

test.sh purges both queues, publishes four orders worth 300, 1500, 750, and 2000, waits for delivery, and reads the depth of each queue. With the starter {} policy the high-value queue holds all four. With the right policy it holds two:

Standard (catch-all) queue: 4   (expected 4)
High-value queue:           2   (expected 2)
PASS: fan-out delivered every order, and the filter kept only the high-value ones.

If it fails

• Both queues show 4. Your filter policy is still matching everything, either because it is {} or because the attribute name does not match what the publisher sends. The attribute is order_value.
• High-value queue shows 0. The comparison is probably a string match, or the scope is wrong. Numeric attributes need a numeric rule, and the scope has to be MessageAttributes to match the attribute rather than the body.
• Nothing arrives anywhere. Check the queue policy allows sns.amazonaws.com to SendMessage; without it, SNS cannot deliver and the messages are silently dropped at the subscription.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• Fan-out is one publisher and many independent subscribers. Each queue is decoupled from the others and from the publisher’s rate, so a slow consumer on one queue cannot back up the other.
• A subscription filter policy moves the routing decision into SNS, before delivery. The fraud team never receives, pays for, or discards the small orders.
• Filter scope matters: attributes by default, body if you set it, and numeric rules for numeric values so 1500 is not compared as a string.
• • The dead-letter queue is attached to each SQS queue rather than to the subscription. After five failed receives a poison message moves to the DLQ, so one bad order does not wedge the queue. A redrive policy on the subscription is a different backstop: it catches what SNS could not deliver in the first place.

Next

The rest of the SA Pro lab arc is in the track’s README: a Transit Gateway to route between VPCs, guarded cross-region backups, weighted blue/green traffic shifting, tag-driven cost visibility, and a DMS change-data-capture migration.