Preventing Lost Writes in DynamoDB Global Tables

The situation

The platform is a subscription-management service with tens of thousands of subscribers split roughly evenly across three continents. When they went multi-region two years ago the team chose DynamoDB Global Tables. Every subscriber record lives in every region; regional API tiers write to their local replica; replication is asynchronous, typically within a second. The design has served them well for throughput and availability. It has also, it turns out, been quietly dropping writes.

The London incident is the visible one. A subscriber changed their delivery address on the self-service portal at 14:07:12.140 UTC. At 14:07:12.189 UTC, a retention job in us-east-1 wrote a last_contacted timestamp to the same subscriber item. Both writes were UpdateItem calls. Neither used a condition expression. Replication propagated each update; when the two versions met, DynamoDB kept the one with the later internal timestamp per item and discarded the other. The retention job’s write happened to be later. The delivery address update was overwritten and lost.

The team reviewed the item and found three things: the address field had reverted to its pre-London value, the last_contacted timestamp from the us-east-1 job was present, and no metric, log, or error recorded a write rejection. They want to understand what’s happening and what they can do while keeping the “regional read/write, low latency everywhere” property that made Global Tables the correct choice.

What actually matters

Multi-region active replication is a set of tradeoffs, and the existing architecture traded something the team didn’t fully appreciate at the time. It’s worth naming what properties the fix needs to preserve and which ones have to change.

Local read and write latency is the property we can’t lose. A subscriber in London hitting an endpoint that writes to us-east-1 gets a 150ms round-trip baseline floor before any work happens. That’s the cost the team avoided by going multi-active; undoing it for every write would halve throughput and make the EU portal visibly slower. Whatever we change has to leave writes landing on the local replica for the general case.

Silent data loss is the property we have to eliminate. An overwrite on the same attribute is a conflict the application can reason about. A drop of a write to a different attribute on the same item because of a millisecond-scale timestamp race isn’t a conflict the application can even detect, let alone reason about. The fix doesn’t need to prevent all concurrent writes (that’s MRSC’s job, at different cost), it needs to make the conflict visible. Detectable loss is a bug we can fix; silent loss is a bug we don’t know we have.

Application retrofit has to be targeted, not universal. A pattern that works only if every item is redesigned isn’t realistic for a running product. A pattern that can be applied to contested items, subscriber profiles, preferences, payment methods, while leaving append-only items (audit logs, event streams) untouched is shippable in sprints.

Batch work needs rethinking, not just data-model tweaks. The retention job that wrote last_contacted from us-east-1 to a subscriber who lives in London is the visible failure. The hidden failure is that any batch job operating on items whose local writers live elsewhere is a second writer on those items, racing replication that hasn’t arrived yet. The architectural fix is moving batch jobs into the home region of the items they touch, a structural change, not a data-model one.

Strongly consistent reads won’t rescue us. The instinctive first fix is “turn on ConsistentRead=true”, but in MREC that’s a regional guarantee, not a global one. A strongly consistent read in eu-west-1 five milliseconds after a us-east-1 write returns the pre-write value, and from eu-west-1’s perspective that is the latest write the region knows about. We shouldn’t confuse a strong local read with a global read.

A synchronously-replicated alternative exists and has a different shape. DynamoDB offers a stronger-consistency mode that replicates synchronously to other regions and rejects conflicting concurrent writes outright. It’s a real answer but a different architecture, synchronous write latency on every write, a fixed regional topology, different failure modes. Not a drop-in toggle for the current setup.

What we’ll filter on

1. Multi-region active writes, local writes on each regional replica.
2. No silent conflict loss, conflicts must be detectable by the application.
3. Local low-latency reads, no mandatory cross-region hop.
4. Reasonable application retrofit, targeted changes, not a rewrite.
5. Handles append-only and contested data differently, pattern per shape.

The replication landscape

DynamoDB Global Tables (v2019.11.21, multi-active, LWW). Each region holds a full replica. Writes are accepted at any regional endpoint and replicated asynchronously. Concurrent updates to the same item resolve by latest internal timestamp per item. Resolution is per item, not per attribute. The losing write disappears without error. The current architecture.

DynamoDB Global Tables (MRSC mode). Multi-region strong consistency: synchronous replication to at least one other region, backs a globally strong read. Requires exactly three regions. Adds latency to every write. Rejects conflicting concurrent writes with ReplicatedWriteConflictException. A different architectural trade.

Aurora Global Database. Relational (MySQL or PostgreSQL). A primary in one region, up to ten secondary read-only clusters. Storage-level replication. Only the primary accepts writes; write-forwarding exists but still round-trips. Not multi-active and not a drop-in for DynamoDB.

Single-region DynamoDB with application-managed replication via Streams. The pre-Global-Tables pattern. You’re now running a custom multi-master system with your own conflict code. Backwards step.

Keyword routing: sharding writes for a key to one region. Not a storage technology but a routing decision. For any given primary key, deterministically designate one region as its home and route all writes for that key there. No cross-region concurrency on that key. Useful as a targeted pattern for contested keys.

Global Tables + version attribute + conditional updates. Application-level optimistic concurrency. Add a version attribute and a condition expression asserting the version hasn’t moved. Turns silent data loss into a detectable event. Composes with keyword routing for the narrow set of items that need linearisable behaviour.

Side by side

The row that ticks every column is Global Tables plus an application-level version attribute with conditional updates. Where an item is known to be heavily contested, layer in keyword routing so the item has a single writer region. Both compose cleanly.

Matching data shape to pattern

Optimistic concurrency in DynamoDB, in depth

The pattern is old and well-understood. The DynamoDB-specific shape is worth naming.

Add a version attribute to every item that needs protection, a monotonically increasing integer is the standard choice. On every update, include a condition expression asserting that the version the caller read is still the current version, and increment the version as part of the same update. The condition and the update are evaluated atomically inside a single UpdateItem call.

UpdateItem:
  TableName: subscribers
  Key: { subscriber_id: "sub_42" }
  UpdateExpression: "SET delivery_address = :addr, #v = :new_v"
  ConditionExpression: "#v = :expected_v"
  ExpressionAttributeNames:  { "#v": "version" }
  ExpressionAttributeValues:
    ":addr":       { "S": "12 High Street, London" }
    ":expected_v": { "N": "17" }
    ":new_v":      { "N": "18" }

On ConditionalCheckFailedException, re-read, re-apply the business logic, and retry. For new items, use attribute_not_exists(subscriber_id) and set version = 1, rejects the put if the item exists.

Now the critical detail for Global Tables: conditional writes are evaluated against the local region’s copy of the item. A conditional write in eu-west-1 checks the eu-west-1 replica’s version, not the global latest. If a us-east-1 write hasn’t yet replicated to eu-west-1 when London’s conditional write fires, London’s write succeeds locally against a version that’s stale from a global perspective. Replication will then carry both updates, and LWW per item applies again.

This sounds like the pattern doesn’t help. It does, because the version attribute changes what LWW resolves to:

• Without a version attribute, LWW drops writes whenever two updates touch the same item at roughly the same time, even across different attributes.
• With a version attribute, both updates bump the version. When replication carries two conflicting versions to a region, LWW still picks one; but now the other is visible as a missed version, an item whose version jumped from 17 to 18 but ought to have been 19, with the other update’s changes absent. The loss is detectable.

Detectable is not prevented. For contested items, combine optimistic concurrency with a routing decision: designate a home region, send every write there. Inside the home region the conditional write’s local strong consistency is a real guarantee.

A practical split:

• Subscriber profile items (occasionally updated from any region), protect with the version attribute and conditional updates; retry on ConditionalCheckFailedException.
• Heavily contested items (billing balances, quota counters, rate-limit tokens), route by key to a single home region; use the version attribute for in-region concurrency.
• Append-only items (event streams, audit logs), use a ULID or time-based sort key so writes don’t collide. No conflict, no version attribute needed.

The London item, with the fix applied

The portal in eu-west-1 reads the subscriber item; version is 17. It submits an UpdateItem setting delivery_address to the new value and version to 18, conditional on the current version being 17. Locally the write succeeds.

Simultaneously, the retention job in us-east-1 reads from its local replica, where London’s write hasn’t yet replicated, and sees version 17. It submits UpdateItem setting last_contacted and version to 18, conditional on version being 17. Locally it also succeeds.

Replication propagates both. Each region now has two candidates at version 18. LWW picks one. Suppose us-east-1 wins.

The difference: the eu-west-1 portal got an acknowledgement stamped version 18. The next time the portal updates the subscriber it will submit with expected_version = 18. If replication has carried the us-east-1 write in, the expected version matches the new current version and the write proceeds. If the item has moved further along a different path, the version check catches the divergence and the portal re-reads and reconciles. Either way, the silent-loss pattern is gone.

A reconciliation job can also scan for the missed-version signature, an item whose version advanced without a corresponding audit entry, and flag candidates for review.

The batch retention job itself wants a structural fix: run it in the home region of each subscriber. If a subscriber has eu-west-1 home affinity, the job updating their last_contacted runs in eu-west-1. The item is never written from more than one region concurrently, and LWW never fires.

Why strongly consistent reads don’t rescue you

A strongly consistent read returns the latest committed write for that region’s replica. In a Global Table running MREC, replication is asynchronous, so ConsistentRead=true in eu-west-1 five milliseconds after a us-east-1 write on the same item returns the pre-write value. The regional guarantee is still intact: from eu-west-1’s perspective that is the latest write the region knows about.

MRSC does offer a globally strong read, at the cost of three-region topology and synchronous replication on every write. A real tool; not a small configuration change to an existing Global Table.

Strongly consistent reads still earn their keep within a region: after a conditional write succeeds locally, a follow-up strongly consistent read in the same region will see the new version.

What’s worth remembering

1. Global Tables (v2019.11.21) default mode is multi-active with per-item last-writer-wins. Conflict resolution is a merge, not a rejection. The losing write is discarded silently.
2. LWW is per item, not per attribute. Two updates touching different attributes on the same item still race; one whole item state wins.
3. Strongly consistent reads in MREC are a regional guarantee, not a global one. Reading locally after a remote write can return stale data.
4. MRSC exists, rejects conflicting concurrent writes with ReplicatedWriteConflictException, and requires exactly three regions with synchronous cross-region latency, a different architectural choice, not a tweak.
5. Aurora Global Database is primary-plus-secondary. Write-forwarding exists but it’s not multi-active.
6. Application-level version attributes with conditional updates turn silent data loss into a detectable event. The version check is evaluated against the local replica.
7. Key-home routing, send all writes for a given primary key to one region, is the clean fix for heavily contested items. It composes with the version attribute.
8. Append-only data wants time-ordered sort keys, not version attributes. No conflict means nothing to resolve.
9. Batch jobs that touch cross-region items are second writers. Move the job to the home region of the data it touches.
10. Detectable loss is a bug you can fix; silent loss is a bug you don’t know you have.