Lab: Fan Out Events With SNS and SQS

The reference posts argue the decisions; a lab stands one up. This one pairs with the event-driven architecture post, which walks the whole decoupling landscape. Here we build the smallest honest slice of it: fan-out with a filter. The full lab is in lab-sa-pro-01-event-fan-out.zip.

If this is your first lab in the track, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every SA Pro lab tags its stack for the reaper on deploy.

Nothing in this stack bills by the hour. SNS and SQS are priced per request, SNS charges nothing for notification delivery to an SQS queue beyond the data transferred, and every account gets a million SQS requests a month free, so the few dozen requests this lab makes do not register. Forgetting this stack costs less than forgetting the one in the Transit Gateway lab, where two attachments bill for every hour they stand. Tear it down anyway.

The scenario

An order service publishes an event for every order. Two teams consume them. The fulfilment team wants every order. The fraud team wants only the large ones, the orders worth AUD$1,000 or more, and it does not want to pay to receive and then discard the small ones.

The tempting shortcut is one queue that both teams read, with the fraud team filtering in its own code. That couples the two teams to one queue and charges the fraud team for pulling and deleting every small order. The decoupled design is one topic and two queues, with a filter policy on the subscription that matters, so Amazon SNS drops the events that do not match before they are ever delivered.

Both queues sit in one account here so the whole thing deploys and tears down in one command. In an organisation the fraud team’s queue lives in the fraud team’s account. One part of this carries over unchanged: the queue policy still grants sqs:SendMessage to the sns.amazonaws.com service principal with an aws:SourceArn condition naming the topic. Two parts are new. The topic policy has to allow the queue owner’s account to call sns:Subscribe, and the queue owner should be the one to create the subscription, because a subscription created by anyone who does not own the queue stays at Pending Confirmation and receives nothing until it is confirmed, which takes two parties: someone with read access to the queue retrieves the SubscribeURL from the confirmation message, and the subscription’s owner is the one who confirms with it. A subscription’s dead-letter queue has to sit in the same account and Region as the subscription, so undeliverable messages land on the subscribing team’s side rather than yours.

What you’re given

CloudFormation builds the topic, both queues, a shared dead-letter queue, the queue policy that lets SNS deliver to SQS, and both subscriptions. The catch-all queue’s subscription has no filter, so it receives every order. The high-value queue’s subscription has a FilterPolicy, and that policy is a template parameter fed from a file you control:

HighValueSubscription:
  Type: AWS::SNS::Subscription
  Properties:
    TopicArn: !Ref OrderEvents
    Protocol: sqs
    Endpoint: !GetAtt HighValueQueue.Arn
    RawMessageDelivery: true
    FilterPolicyScope: MessageAttributes
    FilterPolicy: !Ref HighValueFilterPolicy

The gap is src/filter-policy.json. It ships as {}, an empty policy, which matches every message. So out of the box the fan-out works but nothing is filtered: both queues receive all four orders.

Your task

Write a filter policy that matches only orders whose order_value message attribute is 1000 or more. SNS filter policies are a map from attribute name to a list of match rules, and they support numeric comparisons:

{
  "order_value": [{ "numeric": [">=", 1000] }]
}

Two details decide whether this works. First, the filter matches on the message attributes, not the body, which is why the publisher sends order_value as a Number attribute and not only inside the JSON payload. If the value lived only in the body, you would set FilterPolicyScope: MessageBody and write the policy against the body’s shape instead, and the payload would have to be well-formed JSON for SNS to read it.

Second, the comparison has to be numeric. Attribute-based filtering compares policy values against String and String.Array attributes, with Number supported for numeric values, and >= is a numeric operator only. A quoted "1000" is an exact-match rule. String matching in SNS carries no ordering: it gives you equality, prefix, suffix, wildcard, anything-but and equals-ignore-case, and none of those can say “or more”. Numeric matching runs from minus a billion to a billion with five digits after the decimal point, so 1000 sits well inside the range.

One timing detail explains most confusing runs. A change to a subscription filter policy takes up to 15 minutes to take full effect, and test.sh waits eight seconds for delivery. If you redeploy and the counts still look like the old policy’s, run the test again before you go hunting for a mistake in the JSON.

Run it

./scripts/deploy.sh          # deploys with your filter-policy.json
./scripts/test.sh            # publishes four orders and counts each queue
./scripts/teardown.sh        # deletes everything

All three default to the stack sa-pro-lab-01 in ap-southeast-2; set STACK or AWS_REGION to move them.

test.sh purges both queues, publishes four orders worth AUD$300, AUD$1,500, AUD$750 and AUD$2,000, waits for delivery, and reads the depth of each queue. With the starter {} policy the high-value queue holds all four. With the right policy it holds two:

Standard (catch-all) queue: 4   (expected 4)
High-value queue:           2   (expected 2)
PASS: fan-out delivered every order, and the filter kept only the high-value ones.

If it fails

• Both queues show 4. Either the policy is still the empty {}, which matches every message, or the new policy has not propagated yet. A policy change can take up to 15 minutes.
• High-value queue shows 0. The comparison is probably a string match, the attribute name in the policy is not the one the publisher sends, or the scope is wrong. A policy key with no matching attribute on the message fails the match and nothing is delivered, so a misspelling here empties the queue rather than filling it; the attribute is order_value. A numeric attribute needs a numeric rule, and the scope has to be MessageAttributes to match the attribute rather than the body.
• Counts that miss 4 and 2 in either direction. test.sh purges both queues before it publishes, and a purge takes up to 60 seconds. SQS answers a second purge of the same queue inside that minute with PurgeQueueInProgress, which leaves the previous run’s orders in place and reads high. Messages published while a purge is still running can be deleted along with the old ones, which reads low. Leave a minute between runs.
• Nothing arrives anywhere. Check the queue policy allows sns.amazonaws.com to sqs:SendMessage on the queue, with an aws:SourceArn condition naming the topic you are publishing to. A policy that blocks delivery is a client-side error, and SNS does not retry client-side errors: it discards the message, and with no redrive policy on the subscription there is nowhere to go looking for it. Server-side failures are the ones it retries, up to 100,015 times over 23 days for an SQS endpoint.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• Fan-out is one publisher and many independent subscribers. Each queue is decoupled from the others and from the publisher’s rate, so a slow consumer on one queue cannot back up the other.
• A subscription filter policy moves the routing decision into SNS, before delivery. The fraud team’s queue never receives the small orders, so nobody pays to pull them and delete them.
• Scope is attributes by default and the body when you set it, and a numeric attribute needs a numeric rule. String matching has no ordering, so “1000 or more” cannot be written as a string comparison.
• The dead-letter queue here is attached to each SQS queue rather than to the subscription. After five receives without a delete a poison message moves to the DLQ, so one bad order does not stall the queue.
• A redrive policy on the subscription is a different backstop, and this stack does not set one. It catches what SNS could not deliver in the first place, which matters because a client-side failure is discarded on the first attempt and leaves only a failed-delivery metric behind.

Next

The rest of the SA Pro lab arc is in the track’s README: a DMS change-data-capture migration, weighted blue/green traffic shifting, immutable replicated backups with S3 Object Lock, and tag-driven cost visibility.