Surviving a Dependency's Rate Limits

The situation

A logistics company’s tracking service calls a carrier’s public API to refresh consignment status. The carrier permits 100 requests per second per customer and returns HTTP 429 above it, with a Retry-After header.

The tracking service runs as a Lambda function triggered by SQS, with a reserved concurrency of 200. Each invocation makes one carrier call. When a batch of updates arrives, 200 concurrent invocations produce well over 100 requests per second, the carrier throttles, the function’s SDK retries three times with a short fixed delay, and the effective load triples. Messages fail, return to the queue, and are redelivered, which produces another wave.

Last Tuesday a routine backlog of 40,000 consignments took six hours to clear and the carrier sent an email about abuse. The queue’s redrive policy sent 3,000 messages to the dead letter queue, and nobody has a mechanism to replay them safely.

What actually matters

The first thing that matters is that the limit is not negotiable in the short term, so this is a demand-shaping problem. Everything about the design should aim at offering close to 100 requests per second and no more, rather than at getting more than 100 through.

The second is that concurrency is the control, not the retry policy. The function’s reserved concurrency of 200 is what sets the offered rate, and no amount of retry tuning fixes a system whose steady-state demand exceeds the ceiling. Retries handle the edges; concurrency handles the rate.

The third is that retries against a throttle need to respect what the throttle told you. A Retry-After header is the dependency stating when to come back, and a client that ignores it in favour of its own backoff is guessing worse than the information available.

The fourth is that failed messages returning to the queue produce a second wave that looks like new demand. A redrive with no delay converts a throttling episode into an oscillation, and the dead letter queue then accumulates messages nobody can safely replay because replaying them recreates the problem.

Underneath it, some of this work does not need doing. Refreshing every consignment on a schedule regardless of whether anything changed is what generates the volume, and a carrier that offers webhooks would remove most of it.

What we’ll filter on

1. What sets the rate at which we offer requests?
2. Does the client respect what the throttle told it?
3. Do retries add load to a dependency that is already at its limit?
4. What happens to work that cannot be done now: dropped, delayed, or requeued immediately?
5. Can the work be reduced rather than paced?
6. Is there a mechanism to replay a backlog without recreating the incident?

The landscape

Lambda reserved concurrency. The direct control on how many invocations run at once, and therefore on the offered request rate. Setting it so that concurrency multiplied by the per-invocation call rate stays under the limit is the primary fix, and it is a one-field change that most of these designs never make.

SQS as the buffer. The queue is already doing the right thing: absorbing a burst and letting the consumer set the pace. The failure is that the consumer’s pace was set higher than the dependency’s ceiling. Batch size and batching window on the event source mapping shape how work arrives.

Maximum concurrency on the event source mapping. For SQS triggers specifically, a per-mapping concurrency limit that avoids consuming the function’s whole reserved concurrency, which is the finer control when one function serves several queues.

Exponential backoff with jitter. Waiting longer after each failure, with randomness so retrying callers do not synchronise. Without jitter, a hundred clients backing off identically produce a wave at each interval, which is the same problem at a lower frequency.

Honouring Retry-After. When the dependency states when to come back, using that value beats computing one. Not every SDK does this automatically for a third-party API, so it is usually application code.

Adaptive retry mode in the AWS SDKs. Client-side rate limiting that measures throttling and reduces the request rate accordingly, alongside a retry quota that stops retries consuming capacity indefinitely. Relevant when the throttled dependency is an AWS service rather than a third party, and worth knowing as the pattern to imitate.

A token bucket in front of the caller. Explicit rate limiting on the producing side, so the system offers a controlled rate regardless of how much work is queued. Implemented with a shared counter, or approximated by concurrency limits where per-invocation call rates are predictable.

Step Functions with a paced iterator. Where the work is a batch rather than a stream, a state machine with a Wait state and a map with limited concurrency paces a backlog deterministically, with visible progress and a resumable position.

DLQ redrive. SQS supports redriving messages from a dead letter queue back to the source, with a configurable rate. That rate is what makes a replay safe: putting 3,000 messages back at once recreates the incident.

Webhooks or event subscriptions. Where the dependency offers them, replacing polling with notification removes most of the request volume. This is the change that makes the rate limit stop mattering rather than accommodating it.

Evaluation

Side by side

Control	Reduces offered rate	Respects the dependency	Handles a backlog	Effort
Lower reserved concurrency	✓ directly	Indirectly	✓ slowly	One field
Event source max concurrency	✓ per queue	Indirectly	✓	One field
Backoff with jitter	Slightly	✓	✗	SDK config
Honour Retry-After	✓ during throttling	✓ exactly	✗	App code
Token bucket	✓ precisely	✓	✓	Shared state
Step Functions paced batch	✓	✓	✓ resumable	A state machine
Rate-limited DLQ redrive	✓ during replay	✓	✓	Configuration
Webhooks instead of polling	Removes the load	✓	n/a	Depends on vendor

The top two rows are one field each and would have prevented the incident on their own, which is worth noticing before designing anything more elaborate. The bottom row is the only one that makes the problem go away rather than managing it.

The solution

Set concurrency so the offered rate sits under the limit, honour what the throttle tells you, redrive the dead letter queue at a controlled rate, and ask the carrier for webhooks.

Start with concurrency, because it is the cause. Each invocation makes one call and takes roughly 400 ms, so a concurrency of 200 offers around 500 requests per second against a limit of 100. Setting the event source mapping’s maximum concurrency to 40 puts the offered rate near the ceiling with headroom. This is one field and it is most of the fix.

Then fix the retry behaviour so it helps rather than amplifies. Read the Retry-After header and wait that long rather than applying a local backoff, add jitter so concurrent invocations do not resume together, and cap the attempts so a message that cannot succeed goes to the queue’s visibility timeout rather than being retried in-process while holding concurrency.

Let the queue do the buffering it is there for. Increase the visibility timeout so a throttled message is not redelivered while the previous attempt is still backing off, and set maxReceiveCount high enough that a transient throttling episode does not send work to the dead letter queue. The DLQ should hold messages that are genuinely undeliverable, not ones that arrived during a busy hour.

For the 3,000 messages already in the dead letter queue, use SQS’s redrive with a rate limit rather than moving them back in bulk. Replaying at 20 messages per second clears the backlog in under three minutes of queue time and never approaches the carrier’s limit.

Then reduce the work, which is the largest available improvement and the one nobody asked for. Most consignments do not change status between polls, so refreshing every one on a fixed schedule generates volume proportional to the corpus rather than to the change rate. Ask the carrier whether they offer webhooks or a delta endpoint; where they do, polling becomes a reconciliation safety net at a much lower frequency and the rate limit stops being a constraint at all.

Finally, monitor the throttle rate as a first-class signal. An alarm on 429 responses per minute is what tells you the offered rate has drifted above the ceiling as the business grows, rather than discovering it from an email about abuse.

Why not raise the retry count and push through. It is the instinctive response and it increases load against a ceiling that is not moving, which is how last Tuesday took six hours instead of forty minutes.

Why not ask the carrier for a higher limit first. Worth asking, and it takes commercial negotiation and time, and a system that only works at a limit somebody else controls will break again when the business grows. Shape the demand and ask for the increase.

Worked example

Setting the event source mapping’s maximum concurrency to 40 is deployed on a Thursday. The following Monday’s backlog of 38,000 consignments clears in 52 minutes with zero throttling responses, against six hours and an abuse email.

The Retry-After handling is written and, interestingly, almost never fires afterwards, because the concurrency change removed the condition that produced throttling. It stays in as the safety net for the case where the carrier lowers the limit without telling anyone, which they do four months later.

The DLQ redrive at 20 messages per second clears the 3,000 backlog in about two and a half minutes. The previous attempt, before the concurrency fix, had been abandoned twice because it recreated the incident.

The carrier conversation produces a delta endpoint rather than webhooks: a query for consignments whose status changed since a timestamp. Switching to it takes the daily request volume from 1.1 million to about 40,000, which is a 96% reduction and makes the rate limit irrelevant. The full poll is kept as a nightly reconciliation at a low concurrency, because a delta endpoint that misses an update is a class of bug worth catching.

The 429 alarm fires once in the following six months, when the carrier reduced the limit to 60 without notice. The system throttled, backed off on Retry-After, cleared slowly, and nobody was paged.

What’s worth remembering

1. Throttling is a healthy system declining work, so the response is to shape demand rather than push harder; retrying into a limit increases offered load and stops the dependency recovering.
2. Concurrency sets the offered rate, and no retry tuning fixes a system whose steady-state demand exceeds the ceiling. For a Lambda consumer that is one field.
3. Honour Retry-After when the dependency provides it, because it is the dependency stating when to come back and it beats any backoff you compute.
4. Add jitter to any backoff, or callers synchronise and produce a wave at each interval instead of a smooth recovery.
5. Redrive a dead letter queue at a controlled rate; replaying a backlog in bulk recreates the incident that produced it.
6. Ask whether the work needs doing at all. A delta endpoint or webhooks makes the volume track the change rate rather than the corpus size, which removes the constraint instead of managing it.