Designing a Decoupled, Event-Driven Architecture

The situation

We run an online retailer out of ap-southeast-2. An ordinary day is about 40,000 orders. A sale day runs to roughly 150,000, and the first few minutes after a drop peak near 30 orders a second. The order pipeline is one synchronous chain: checkout takes the payment, calls the warehouse service to reserve stock, calls invoicing, calls notifications, and only then returns 200 to the customer. Four services, one HTTP call chain, one shared fate.

The warehouse service is the weak link. It fronts a legacy warehouse-management system that slows during picking waves, and when its p99 climbs past ten seconds, checkout’s own timeout fires. Customers see an error and press the button again, so we have duplicate orders to reconcile on top of the abandoned carts. The last sale event turned a twenty-minute warehouse slowdown into an hour of checkout errors and a day of support tickets.

The pipeline also crosses account lines. Checkout and the warehouse shim live in the retail production account, analytics owns a separate account and budget, and fraud screening runs in a third under a team that will not take a dependency on our release train. Everyone agrees checkout should accept the order and get out of the customer’s way. The disagreement starts at “put a queue in”, because AWS has at least five services that could be that queue.

What actually matters

Decoupling is three separate promises, and naming them first narrows the field. Availability decoupling means a slow consumer no longer stalls the producer, because something durable absorbs the backlog. Knowledge decoupling means the producer stops knowing who consumes its output, so a fourth consumer becomes a subscription change rather than a code change in checkout. Time decoupling means the messages outlive their delivery, so a consumer built next quarter can read this quarter’s events. A plain queue gives the first. A topic or a bus adds the second. Only a retained, replayable store gives the third.

Delivery semantics decide what the consumers have to survive. At-least-once delivery means duplicates arrive and every consumer has to be idempotent about them, which is the default across this whole space. Ordering is the constraining promise: guaranteeing that a cancellation never overtakes the order it cancels forces serialisation, and serialisation caps throughput. So the ordering scope should be the narrowest one that is still correct, per customer or per order rather than global. Read any exactly-once claim for its boundary. Each is scoped to something specific, like a FIFO queue’s five-minute deduplication interval or a Standard workflow’s execution, never to the pipeline as a whole.

Traffic shape decides the rest. Point-to-point work (“reserve this stock”) needs a buffer with competing consumers, where queue depth is the scaling signal and a burst becomes backlog instead of errors. Broadcast (“something happened”) needs fan-out, and there is a real fork inside fan-out between sending a copy to everyone and routing by content. Streams are a third thing: the same retained records, read by several consumers, each at its own position. Whichever shape wins, a multi-step sequence with retries and undo steps still needs an owner. Choreography spreads it across consumers reacting to each other’s events; orchestration puts it in one place that can retry, time out and compensate. Choosing neither is the failure mode.

Account boundaries move several of these answers, and they are the ones most often skipped. Nearly every quota here is per account and per Region, so a shared bus concentrates one PutEvents ceiling across every producer in the organisation. Some plumbing does not cross an account line at all: an SQS dead-letter queue has to sit in the same account and Region as the source queue it serves. And the bill follows the direction of delivery, because EventBridge charges the sending account for a cross-account event and leaves the receiving account unbilled.

What we’ll filter on

1. Coupling removed: availability only, or also knowledge, or also time?
2. Delivery semantics: at-least-once with idempotent consumers, ordered and deduplicated per group, or a consumer-tracked position?
3. Traffic shape: point-to-point buffer, copy-to-all fan-out, content-based routing, or a shared stream?
4. Retention and replay: can a repaired consumer re-read history, how far back, and in what order?
5. Throughput ceiling, and whether it is counted per account, per Region, per queue or per shard.
6. Reach and failure handling: does it cross accounts and Regions, where do poison messages land, and who owns compensation?

The landscape

SQS, standard queues. The point-to-point buffer. Producers send, competing consumers poll, and each message goes to one of them. Delivery is at-least-once with best-effort ordering, so consumers must be idempotent and order-tolerant. Throughput is nearly unlimited per API action. Retention defaults to four days and caps at fourteen; the visibility timeout defaults to 30 seconds and caps at 12 hours. ApproximateNumberOfMessagesVisible is the natural scaling metric for the consumer fleet. A MessageGroupId on a standard queue now enables fair queues, which detect a tenant holding a disproportionate share of in-flight messages and prioritise the quiet ones; there the field is a tenant label, not an ordering promise.

SQS, FIFO queues. The same buffer with strict ordering inside a message group and deduplication across a five-minute interval, keyed on a deduplication ID or a SHA-256 hash of the body. Ordering is per group rather than per queue, so groups are the concurrency unit: different groups run in parallel, one group runs one message at a time. The base quota is 300 transactions a second per API action, or 3,000 messages a second with ten-message batches. High throughput mode lifts that to 9,000 transactions a second in ap-southeast-2, against 70,000 in us-east-1, us-west-2 and eu-west-1. The subtle cost is head-of-line blocking: a message that keeps failing stops its whole group until it succeeds or dead-letters.

SNS. Pub/sub push fan-out. A producer publishes once and SNS pushes a copy to every subscription: SQS queues, Lambda functions, HTTPS endpoints, Amazon Data Firehose streams, email, SMS. Checkout publishes “order placed” without knowing who listens. Filter policies act on message attributes or on a JSON message body, chosen per subscription through FilterPolicyScope, so body-aware routing is not an EventBridge-only capability. Standard topics keep no history: an undeliverable message is retried, then dropped or dead-lettered per subscription, and a subscriber added tomorrow sees nothing from today. FIFO topics add ordered fan-out into FIFO queues plus an archive of up to 365 days that a subscriber replays for itself with a ReplayPolicy. Quotas are per account and per Region, and lower than the reputation suggests: in ap-southeast-2, 1,500 messages a second for standard topics and 3,000 for FIFO, against 30,000 in us-east-1.

EventBridge. The event bus. Same direction as SNS, but routing lives in rules matching any field of the event body. A rule sends to up to five targets, a hard quota, drawn from more than twenty AWS target types plus API destinations for external HTTPS endpoints. Besides the default bus where AWS service events land, there are custom buses and partner buses fed by SaaS providers. An archive filters by event pattern and retains for a set number of days or indefinitely; a replay resends to the archive’s source bus only, in one-minute buckets rather than original order. A bus resource policy can grant a whole AWS Organization by org ID, and every cross-account bus target created since March 2023 needs an IAM role. Chaining does not work: a bus forwarding events received from another bus will not pass them to a third. Cross-Region targets reach any commercial Region. Defaults in ap-southeast-2 are 1,200 PutEvents a second and 2,250 invocations a second, both adjustable. Custom events cost USD$1.00 per million published, AWS service events on the default bus are free, and the sending account pays for cross-account delivery. Pipes is the point-to-point sibling: one source, optional filter, optional enrichment, one target.

Kinesis Data Streams. The retained, ordered stream. Records land in shards by partition key, are strictly ordered within a shard, and every consumer reads the same records independently at its own position. Retention starts at 24 hours and extends to 365 days, and a consumer can start from the oldest record, a timestamp or a sequence number. A provisioned shard takes 1 MB/s or 1,000 records a second in, and serves 2 MB/s out across five GetRecords calls a second shared by every polling consumer. Enhanced fan-out gives each registered consumer its own 2 MB/s per shard over HTTP/2, up to 20 consumers, or 50 in On-demand Advantage mode, which ap-southeast-2 supports. Propagation delay averages around 200 ms with one shared consumer and 1,000 ms with five, against roughly 70 ms on enhanced fan-out. A resource policy on the stream, or on one registered consumer, grants another account read access. It is the only option here giving all three decouplings.

Step Functions. An orchestrator rather than a messaging service, here because “decouple the pipeline” often means “this workflow needs an owner”. Standard workflows run up to a year with exactly-once execution, are billed per state transition, and keep execution history for 90 days. Express workflows run up to five minutes at very high volume, billed on executions, duration and memory; asynchronous Express is at-least-once and synchronous Express at-most-once. Express supports Request Response only, so .sync, .waitForTaskToken and Distributed Map are Standard-only. Across accounts, the Credentials field on a Task state names a role in the target account to assume before the call.

Amazon MQ, briefly. Managed ActiveMQ Classic and RabbitMQ, speaking JMS, NMS, OpenWire, AMQP, MQTT, STOMP and WebSocket. It rehosts a broker without rewriting producers and consumers, on sized brokers with maintenance windows. Cross-Region data replication for ActiveMQ gives an asynchronous replica broker that a failover request promotes.

Evaluation

Side by side

Matching the shape to the service

The solution

SQS is the buffer, and the visibility timeout sets the retry clock. A consumer receives a message and the clock starts. Finish and delete within the timeout, or the message reappears for another consumer. Set it too short and healthy work gets processed twice; set it too long and a crashed consumer delays the retry by the full window. Several multiples of the p99 processing time is the usual starting point, extended in flight with ChangeMessageVisibility. Delivery is at-least-once regardless, so the warehouse consumer must treat “reserve stock for order 4711” as a no-op the second time, keyed on the order ID.

Dead-letter queues carry more design weight than their one-line configuration suggests. Without a redrive policy, a payload that always crashes the consumer cycles forever, consuming receive quota while looking like ordinary traffic. With maxReceiveCount at five it parks in the DLQ, an alarm on DLQ depth reaches a human, and redrive-to-source returns it once the bug is fixed. Two constraints catch people. The DLQ must live in the same account and Region as its source queue, so a central “bad messages” account is not available. And for a standard queue expiry is measured from the original enqueue timestamp rather than the move, so a DLQ needs a longer retention period than the queue it drains.

FIFO’s ordering is per message group, and the group is the design decision. Group by customer ID and one customer’s messages serialise while thousands of customers process in parallel. Group by a single constant and the whole queue becomes one lane, which caps throughput at that lane’s processing speed and makes every poison message a roadblock for everyone else. Deduplication is the other half of the contract, and it absorbs the double-click and the retried API call at the queue rather than in consumer logic. It does not absorb a duplicate sent six minutes later, so the idempotent-consumer rule still stands.

SNS and EventBridge overlap more than they used to, and the tiebreakers have moved. Body filtering is no longer one of them, because SNS filter policies read a JSON payload when FilterPolicyScope is set to MessageBody. What is left is real enough: EventBridge routes from one rule to targets of more than twenty types, archives and replays, accepts events from SaaS partner buses, and grants a whole Organization access through one bus resource policy. SNS reaches far more subscribers per topic and pushes with lower latency, and its standard topics have no archive at all. Throughput is close to a wash rather than a point in SNS’s favour: 1,500 SNS messages a second here against 1,200 EventBridge PutEvents, both per account and both adjustable.

Two habits keep a bus healthy. Configure a dead-letter queue on every target, because delivery retries run for 24 hours and then the event is gone. And turn the archive on from day one, since a replay reaches back only as far as the archive does.

Choose Kinesis for the replay and the independent readers, then plan for shard management. The partition key decides which shard a record lands in, so it also decides both the ordering scope and the hot-spot risk. Keying by store ID during a single-store flash sale funnels the burst into one shard while its neighbours idle; customer ID spreads load and still preserves the ordering that matters. The event source mapping holds the failure policy for Lambda consumers, and its defaults are the trap: MaximumRetryAttempts and MaximumRecordAgeInSeconds both default to -1, so an unconfigured mapping retries a poison record until retention expires it and the shard behind it moves nowhere. Set both, turn on BisectBatchOnFunctionError, give the mapping an on-failure destination, and return ReportBatchItemFailures so only the failed records are retried.

Step Functions fits when the failure handling is the feature. A fulfilment flow of reserve stock, capture payment and create shipment is three calls, but the real specification is longer. Retry the WMS call three times with exponential backoff, but never retry the payment capture. Time out reservation at five minutes. If shipment creation fails after payment was captured, refund and release. In a choreographed system that logic smears across three consumers and their queues; in a state machine it is legible in one place, and every execution’s history names the step order 4711 died on, with its input and its error, for 90 days.

Standard is the default for sagas: exactly-once execution, a year of runway, and task tokens to pause mid-flow for a courier webhook or a human approval. Express suits short, hot paths where at-least-once execution is acceptable and per-transition billing would dominate the bill at volume. Check the asymmetry before picking Express for a saga, because .waitForTaskToken, .sync and Distributed Map are Standard-only. Billing cuts the other way too: a Standard state machine iterating over ten thousand items belongs in a Distributed Map or an Express child workflow. Across accounts, the Credentials field on a Task state names the role to assume for that call, which is how one state machine drives fulfilment in our account and fraud screening in someone else’s.

Amazon MQ stays out. It answers a migration question, and this is a new build.

Worked example

Checkout shrinks to its actual job. It validates the cart, authorises payment, writes the order, and publishes one event to a custom EventBridge bus in the retail account, with an idempotency key on the publish so a client retry cannot mint a second order:

{
  "Source": "retail.checkout",
  "DetailType": "OrderPlaced",
  "Detail": {
    "orderId": "ord-4711",
    "customerId": "cust-2093",
    "storeId": "syd-01",
    "total": 342.50,
    "lines": [{ "sku": "SKU-99182", "qty": 2 }]
  }
}

The customer’s confirmation now takes the payment authorisation plus one PutEvents call, tens of milliseconds of AWS on top of the payment. Whatever the warehouse is doing has left checkout’s latency histogram. At the 30-a-second peak we use about 2.5% of the Region’s default 1,200 PutEvents a second, which leaves room for the producers that land on this bus later.

Four rules route the event. The fulfilment rule matches every OrderPlaced and targets an SQS FIFO queue with the message group ID set to the customer ID, so one customer’s order and their later cancellation stay in sequence while unrelated customers flow in parallel. At 30 orders a second that queue sits at a tenth of the 300-per-action base quota, so high throughput mode is not needed. The notifications rule targets the confirmation-email Lambda. The analytics rule targets an Amazon Data Firehose stream into S3, which puts every order in the lake without the analytics team ever talking to the checkout team. The fraud rule matches on the body, $.detail.total over AUD$500, and targets the fraud team’s bus in their account through a role on the target; they are not billed for it, and we pay the published USD$1.00 per million. The archive is on with 90-day retention, every target has a DLQ, and a loyalty-points service next quarter is one new rule and no change to checkout.

The warehouse consumers poll the FIFO queue and scale on queue depth. During a sale afternoon orders arrive at about eight a second while the fleet drains five, so the backlog grows by three a second. Twenty minutes in, that is roughly 3,600 messages waiting, twelve minutes of work at the fleet’s current rate, and ApproximateAgeOfOldestMessage sitting near seven minutes. The alarm fired when it crossed five. Nothing times out and nobody sees an error page. When the WMS recovers the fleet drains the backlog, and the slowdown shows up on a dashboard instead of in the support queue.

Each dequeued order starts a Step Functions Standard execution that owns fulfilment end to end. Reserve stock, retrying three times with backoff because the WMS drops connections. Capture the authorised payment, with no automatic retry, so a capture failure goes straight to the failure branch. Create the shipment, then wait on a task token for the courier’s label webhook with a 24-hour timeout. The fraud check is a Task with a Credentials role into the fraud account, so the workflow crosses the boundary without either team sharing a deployment. The Catch on the capture-and-ship stretch runs compensation: release the reservation, refund the payment, and emit OrderFailed back onto the bus, where a rule routes it to the apology-email Lambda and a support queue.

Kinesis is absent because nothing here yet needs a replayable multi-reader stream. The WMS still speaks its 2009-era message format, so an EventBridge Pipe with an API destination enrichment translates between the queue and the state machine’s input, keeping the legacy shim in configuration instead of code.

What’s worth remembering

1. Decoupling is three promises: availability, knowledge and time; name which the workload needs before naming a service.
2. SNS filter policies now read the message body as well as attributes, so the tiebreakers against EventBridge are archive and replay, partner buses, and org-wide access through a bus policy.
3. The topic-plus-queues pattern, a topic or bus in front and an SQS queue per consumer behind, gives producers fan-out and each consumer its own buffer, retry and DLQ.
4. Almost every quota here is per account and per Region, an SQS dead-letter queue must sit in its source queue’s own account and Region, and EventBridge bills the sending account for cross-account delivery.
5. Step Functions Standard is the answer when the workflow needs an owner: exactly-once, up to a year, 90 days of history, and .waitForTaskToken and Distributed Map that Express lacks.
6. Dead-letter everything, because at-least-once delivery guarantees the poison message arrives: a redrive policy on queues, a per-target DLQ on bus rules, and explicit retry bounds on stream event source mappings.