Designing a Decoupled, Event-Driven Architecture

The situation

We run an online retailer out of ap-southeast-2: roughly 40,000 orders a day, peaking around 30 orders a second during sale events. The order pipeline is one synchronous chain. Checkout takes the payment, calls the warehouse service to reserve stock, calls invoicing, calls notifications, and only then returns 200 to the customer. Four services, one HTTP call chain, one shared fate.

The warehouse service is the weak link. It fronts a legacy warehouse-management system that slows to a crawl during picking waves, and when its p99 climbs past ten seconds, checkout’s own timeout fires. Customers see an error, hit the button again, and now we have duplicate orders to reconcile on top of the abandoned carts. During the last sale event, a twenty-minute warehouse slowdown cost us an hour of checkout errors and a day of support tickets.

Everyone agrees on the diagnosis: checkout should accept the order and get out of the customer’s way, and the downstream work should happen at whatever pace the downstream services can manage. The disagreement starts when someone says “put a queue in”, because AWS has at least five services that could be that queue, and they are not interchangeable.

What actually matters

Decoupling is really three separate promises, and it pays to name them before shopping. The first is availability decoupling: a slow or dead consumer no longer stalls the producer, because something durable sits between them and absorbs the backlog. The second is knowledge decoupling: the producer stops knowing who consumes its output, so adding a fourth consumer is a subscription change, not a code change in checkout. The third is time decoupling: the messages outlive their delivery, so a consumer built next quarter can read this quarter’s events. A plain queue gives us the first. A pub/sub topic or bus adds the second. Only a retained, replayable stream gives us the third. Deciding which promises this pipeline actually needs is most of the decision.

Delivery semantics matter just as much, because they dictate what the consumers must be able to survive. At-least-once delivery means every consumer sees occasional duplicates and must be idempotent; that is the default almost everywhere in this space, and fighting it is harder than accepting it. Ordering is the expensive promise. Guaranteeing that a customer’s “cancel” never overtakes their “place order” forces serialisation somewhere, and serialisation caps throughput, so we should buy ordering only at the granularity we need (per customer, per order) rather than globally. Anyone offering exactly-once should be read carefully: it is always scoped to a boundary, like a FIFO queue’s five-minute deduplication window or a Standard workflow’s execution, never a property of the whole pipeline.

Then there is the shape of the traffic between services. Point-to-point work (“reserve this stock”) needs a buffer with competing consumers, where queue depth is the scaling signal and a burst becomes backlog instead of errors. Broadcast (“something happened, whoever cares can react”) calls for fan-out, and there is a real fork between fanning out a copy to everyone and routing by content so each target sees only what its rule matches. Streams (“everything that happened, in order, for anyone to read at their own pace”) are a different animal again: multiple independent consumers over the same retained records, each tracking its own position.

Finally, somebody has to own the failure handling. A message that can never be processed (the poison message) needs somewhere to go, and a dead-letter queue on every consumer is the difference between one bad payload parking itself for inspection and one bad payload blocking a whole queue. And when the downstream work is a sequence with retries, timeouts, and undo steps, that sequence needs an owner. Choreography spreads the workflow across consumers reacting to each other’s events, which scales knowledge-free but makes “where is order 4711 stuck?” a distributed-tracing question. Orchestration puts the workflow in one place that can retry, time out, and compensate. Both are legitimate; pretending we have chosen neither is not.

What we’ll filter on

1. Coupling removed: availability only, or also knowledge (producer ignorant of consumers), or also time (replay)?
2. Delivery semantics: at-least-once with idempotent consumers, or ordering and deduplication per group, or a consumer-tracked stream position?
3. Traffic shape: point-to-point buffering, copy-to-all fan-out, content-based routing, or a shared stream?
4. Retention and replay: can a new or repaired consumer re-read history, and how far back?
5. Throughput and burst behaviour: what is the ceiling, and does a burst become backlog or backpressure?
6. Failure handling: retries, dead-letter routing, and who owns multi-step compensation?

The landscape

SQS, standard queues. The point-to-point buffer. Producers send, a pool of competing consumers polls, each message is processed by one consumer. Delivery is at-least-once with best-effort ordering, so consumers must be idempotent and order-tolerant. Throughput is effectively unlimited, retention is up to fourteen days, and the visibility timeout gives a consumer exclusive time to finish before the message reappears for someone else. A redrive policy moves a message to a dead-letter queue after maxReceiveCount failed attempts, which is how a poison message parks instead of looping forever. Queue depth (ApproximateNumberOfMessagesVisible) is the natural auto-scaling metric for the consumer fleet. This is availability decoupling in its purest form: checkout does still know it is sending to the warehouse queue.

SQS, FIFO queues. The same buffer with two extra promises: strict ordering within a message group, and deduplication within a five-minute window (content-based hashing or an explicit deduplication ID). Ordering is per message group ID, not per queue, so groups are the concurrency unit: messages in different groups process in parallel, messages in one group process one at a time. The price is throughput, 300 messages a second without batching and 3,000 with, though high-throughput mode raises the ceiling substantially by partitioning across groups. The subtle cost is head-of-line blocking: a message that keeps failing blocks its entire group until it succeeds or dead-letters, because delivering the next message first would break the ordering promise.

SNS. Pub/sub push fan-out. A producer publishes once to a topic; SNS pushes a copy to every subscription: SQS queues, Lambda functions, HTTP endpoints, Firehose, email, SMS. This gives us knowledge decoupling, since checkout publishes “order placed” without knowing who listens, and subscription filter policies let each subscriber trim the stream to the messages it needs. Standard topics keep no history: a message that cannot be delivered is retried and then dropped (or dead-lettered per subscription), and a subscriber added tomorrow sees nothing from today. The workhorse pattern is topic-to-queues: SNS does the fan-out, an SQS queue per consumer does the buffering and retry, and each consumer gets durability without the producer publishing more than once. FIFO topics exist for ordered fan-out into FIFO queues when the ordering promise has to survive the broadcast.

EventBridge. The event bus: same publish-and-subscribe direction as SNS, but routing lives in rules that match on any field of the event body, not just attributes. A rule matches events by pattern (source, detail-type, any nested field) and sends them to up to five targets, with over twenty target types across AWS services plus API destinations for external HTTP APIs. Buses come in three flavours: the default bus (where AWS service events land), custom buses for our own domains, and partner buses that receive events from SaaS providers directly. An archive on the bus retains matched events and can replay them onto the bus later, which is how a bug in a consumer becomes “fix, then replay Tuesday” instead of “fix, then apologise”. EventBridge Pipes is the point-to-point sibling: source, optional filter, optional enrichment, target, with no bus in the middle. The trades against SNS: richer routing and replay, lower throughput ceilings (a soft, raisable regional limit on PutEvents) and higher typical latency (around half a second rather than SNS’s milliseconds).

Kinesis Data Streams. The retained, ordered stream. Records land in shards by partition key; within a shard they are strictly ordered, and every consumer reads the same records independently, tracking its own position. Retention runs from 24 hours by default up to 365 days, and a consumer can start reading from the oldest record, a timestamp, or a sequence number, which makes replay a first-class operation rather than a recovery procedure. Provisioned capacity is per shard (1 MB/s or 1,000 records/s in; 2 MB/s out, shared across consumers, or 2 MB/s per consumer with enhanced fan-out); on-demand mode does the shard maths for us at a premium. Lambda consumes via event source mappings with per-shard batching, and the poison-record story is different from queues: a failing batch blocks its shard until the mapping’s bisectBatchOnFunctionError, retry limits, and on-failure destination deal with it. This is the only option here that gives all three decouplings at once: availability, knowledge, and time.

Step Functions. Not a messaging service at all: an orchestrator, here because “decouple the pipeline” often turns out to mean “this workflow needs an owner”. A state machine holds the sequence, the retries with backoff, the timeouts, the branching, and the compensation path, and calls services directly through SDK integrations or waits on a task token for a human or webhook callback. Standard workflows run up to a year with exactly-once execution semantics, priced per state transition, built for sagas. Express workflows run up to five minutes at very high volume, priced by duration, with at-least-once semantics for asynchronous invocations, built for short bursts of orchestration where an occasional re-run is tolerable. The state machine turns “where is order 4711 stuck?” from a tracing expedition into a console lookup.

Amazon MQ, briefly. Managed ActiveMQ and RabbitMQ, for applications that already speak JMS, AMQP, MQTT, or STOMP. Its purpose is lift-and-shift: move the broker to AWS without rewriting the producers and consumers. It is a broker on instances, with broker-shaped scaling limits, so for a new cloud-native build the answer is almost always SQS, SNS, or EventBridge instead. It is worth it when the migration backlog is long and the messaging rewrite is not this quarter’s problem.

Evaluation

Side by side

Matching the shape to the service

The solution

SQS is the buffer, and the visibility timeout is its heartbeat. A consumer receives a message and the clock starts: finish and delete within the visibility timeout, or the message reappears for another consumer. Set it too short and healthy work gets processed twice; set it too long and a crashed consumer delays retry by the full window. The rule of thumb is several multiples of the p99 processing time, extended in-flight via ChangeMessageVisibility for genuinely long work. Because delivery is at-least-once regardless, idempotency is not optional: the warehouse consumer should treat “reserve stock for order 4711” as a no-op the second time it sees it, keyed on the order ID. Dead-letter queues deserve more respect than they get. Without a redrive policy, a payload that always crashes the consumer cycles forever, burning receives and hiding behind healthy traffic; with maxReceiveCount set to something like five, it parks in the DLQ where an alarm on queue depth pages a human, and the redrive-to-source feature sends it back once the bug is fixed.

FIFO’s ordering is per message group, and the group is a design decision. Group by customer ID and one customer’s messages serialise while thousands of customers process in parallel; group by a single constant and the whole queue is one lane, which caps throughput at that lane’s processing speed and makes every poison message a roadblock for everyone. Deduplication is the other half of the FIFO contract: within a five-minute window, two sends with the same deduplication ID (or identical content, with content-based dedup enabled) become one message, which absorbs the double-click and the retried API call at the queue rather than in consumer logic. It does not absorb a duplicate sent six minutes later, so the idempotent-consumer rule still stands.

SNS and EventBridge overlap, and the tiebreakers are routing depth, replay, and latency. If subscribers can express what they need with filter policies on message attributes and everyone tolerates fire-and-forget history, SNS is simpler, faster, and has enormous fan-out headroom. The moment routing needs to inspect the event body (“orders over $500 from the NZ store to the fraud checker”), or a consumer needs events replayed after a bad deploy, or events arrive from a SaaS partner or thirty other AWS accounts, EventBridge is doing work SNS cannot. The five-targets-per-rule limit is not a real constraint, since a rule can target an SNS topic or SQS queue and fan out from there. Two habits keep a bus healthy: configure a dead-letter queue on every target (delivery retries last up to 24 hours, then the event is gone unless it has somewhere to land), and turn on the archive from day one, because replay only reaches back as far as the archive does.

Kinesis is chosen for the replay and the independent readers, and paid for in shard management. The partition key decides which shard a record lands in, so it also decides the ordering scope and the hot-spot risk: keying a retail stream by store ID during a single-store flash sale funnels the burst into one shard while its neighbours idle. Keys should spread load while preserving the ordering that matters (customer ID does both here). Consumers sharing a shard split 2 MB/s of read throughput between them; past two or three consumers, enhanced fan-out gives each registered consumer its own 2 MB/s pipe and push delivery. For Lambda consumers, the event source mapping is where failure policy lives: bisectBatchOnFunctionError splits a failing batch to isolate the bad record, maximumRetryAttempts bounds the blockage, and an on-failure destination receives the metadata of what was skipped. Without those, one poison record stalls its shard until it expires from retention.

Step Functions fits when the failure handling is the feature. A fulfilment flow of reserve stock, capture payment, and create shipment is three calls, but the real spec is longer: retry the WMS call three times with exponential backoff, but not the payment capture; time out reservation at five minutes; if shipment creation fails after payment captured, refund and release. In a choreographed system that logic smears across three consumers and their queues. In a state machine it is legible in one place, and every execution’s history shows exactly which step order 4711 died on and with what input. Standard is the default for sagas: exactly-once execution, a year of runway, task tokens to pause mid-flow for a courier webhook or a human approval. Express suits short, hot paths where at-least-once execution is acceptable and per-transition pricing would sting at volume. The cost model is worth respecting: Standard charges per state transition, so a chatty state machine iterating over ten thousand items belongs in a distributed map or an Express child workflow, not ten thousand Standard transitions.

Amazon MQ is the migration answer, not the design answer. When a Java estate speaks JMS to an on-prem ActiveMQ, rehosting the broker on Amazon MQ moves the workload to AWS this quarter and defers the rewrite. Nothing about a new build points at it: SQS out-scales it, SNS out-fans it, and neither needs instance sizing or maintenance windows.

Worked example

Checkout shrinks to its actual job. It validates the cart, authorises payment, writes the order, and publishes one event to a custom EventBridge bus, with an idempotency key on the publish so a client retry cannot mint a second order:

{
  "Source": "retail.checkout",
  "DetailType": "OrderPlaced",
  "Detail": {
    "orderId": "ord-4711",
    "customerId": "cust-2093",
    "storeId": "syd-01",
    "total": 342.50,
    "lines": [{ "sku": "SKU-99182", "qty": 2 }]
  }
}

The customer gets their confirmation in the time it takes to authorise payment and hit PutEvents, tens of milliseconds of AWS on top of the payment call. Whatever the warehouse is doing no longer appears in checkout’s latency histogram.

Three rules on the bus route the event. The fulfilment rule matches every OrderPlaced and targets an SQS FIFO queue with the message group ID set to the customer ID, so one customer’s order and their thirty-seconds-later cancellation stay in sequence while unrelated customers flow in parallel. The notifications rule targets the confirmation-email Lambda directly. The analytics rule targets a Firehose stream into S3, which is the cheap way to get every order into the lake without the analytics team ever talking to the checkout team. The bus archive is on with 90-day retention, every target has a DLQ, and when we add a loyalty-points service next quarter it will be one new rule and zero changes to checkout.

The warehouse consumers poll the FIFO queue and scale on queue depth. During a picking wave the WMS slows, consumers process fewer messages a second, the backlog grows to a few thousand, and an alarm notes that fulfilment is running twenty minutes behind. Nothing times out; nobody sees an error page. When the WMS recovers, the fleet drains the backlog. The slow consumer has become a graph instead of an incident.

Each dequeued order starts a Step Functions Standard execution that owns fulfilment end to end: reserve stock (retry three times, backing off, because the WMS drops connections), capture the authorised payment (no automatic retry; a capture failure goes straight to the failure branch), create the shipment, then wait on a task token for the courier’s label webhook, with a 24-hour timeout. The Catch on the capture-and-ship stretch runs compensation: release the reservation, void or refund the payment, and emit OrderFailed back onto the bus, where a rule routes it to the apology-email Lambda and a support queue. When order 4711 gets stuck, the execution history names the step, the input, and the error, and a support engineer reads it in the console rather than grepping four services’ logs.

Two deliberate leftovers. Kinesis is absent because nothing here needs a replayable multi-reader stream yet; the day clickstream analytics or a real-time stock position arrives, it slots in beside the bus rather than replacing it. And the WMS still speaks its 2009-era message format, so an EventBridge Pipe with an enrichment step translates between the queue and the state machine’s input, keeping the legacy shim in configuration instead of code.

What’s worth remembering

1. Decoupling is three promises, not one: availability (a slow consumer stops hurting the producer), knowledge (the producer stops knowing its consumers), and time (history can be re-read); name which ones the workload needs before naming a service.
2. SNS copies to every subscriber with milliseconds of latency and attribute filters; EventBridge routes on event content with archive, replay, cross-account and SaaS sources, at a lower throughput ceiling and half a second of latency.
3. The topic-plus-queues pattern (SNS or EventBridge in front, an SQS queue per consumer behind) gives producers fan-out and consumers their own buffer, retry, and DLQ.
4. Step Functions is the answer when the workflow itself needs an owner: retries, timeouts, and compensation live in one state machine; Standard for exactly-once sagas up to a year, Express for short high-volume runs.
5. Dead-letter everything: queues via redrive policy, bus targets via per-target DLQs, stream consumers via bisect-and-destination, because at-least-once delivery guarantees the poison message will eventually arrive.