How to Share KMS Key Material Across Two Regions

The situation

• Data plane in us-east-1 and eu-west-1. Each region runs a full copy of the stack: API Gateway, ECS on Fargate, Aurora Global Database, S3 buckets with bidirectional replication.
• Customer data, tenant documents and export artefacts, is written to S3 with SSE-KMS using a customer-managed symmetric encryption key (arn:aws:kms:us-east-1:111111111111:key/aaaa1111-...). The key is single-region.
• Replication: S3 Cross-Region Replication copies objects to eu-west-1. The replicated object retains its original SSE-KMS metadata, including the us-east-1 key ARN that wrapped the data key at write time.
• The symptom: when the European stack calls GetObject, S3 tries kms:Decrypt with the stored key ARN. KMS in eu-west-1 does not host that key. Status quo is the SDK shipping the decrypt back to us-east-1 (80-100ms trans-Atlantic per read), or a Lambda decrypt proxy the security team wants gone.

The constraints: symmetric decrypt must run locally in both regions; key material must be identical in both regions; rotation must be a single coordinated act; failover must be a governance change, not a data migration; operational overhead must be low.

What actually matters

Before mapping services to the ask, worth naming the deeper properties of an answer.

The first observation is that KMS’s regional model drives everything here. Every KMS key is a regional resource by design, the material lives in a regional HSM partition, the key policy is regional, aliases are regional, CloudTrail events are regional. That’s the property that makes KMS independent-failure-domain, and it’s the property that creates this problem the moment two regions share ciphertext. A ciphertext carries the id of the key that wrapped it; that id is a regional ARN; following it leads to one region. For a single-region workload this is invisible; for a multi-region workload with shared ciphertext, it’s the defining constraint.

Identical key material is the physics requirement. If the primary wraps a data key and hands the ciphertext to the replica region, the replica must have the same bytes in its HSM partition to unwrap. No amount of clever policy, routing, or token-passing changes this. Either the material is present in both regions (the answer we want) or it isn’t (and something has to bridge). The bridging options, routing, proxies, dual-wrapping envelopes, all fall into the shape of “compensate for the material being in one place.” They all work. They all offend either the latency budget, the blast-radius budget, or the operational budget for an active-active architecture.

Coordination is the other hard property. If the same material lives in two HSMs, we need one rotation ceremony. Two independent annual-rotation cadences would drift; after a few rotations the replica’s material is something the primary never produced, and a ciphertext wrapped by the replica could be unwrappable by the primary in 8 months. The coordination has to be built into the instrument, not layered on top. Two separate keys with an ops team that remembers to rotate them on the same day is an instrument that will eventually fail on a Thursday afternoon.

Governance independence between the regions matters in the opposite direction. While the material must be identical, everything about governance should be regional. The us-east-1 key policy names local application roles; the eu-west-1 key policy names its local roles; CloudTrail events for each region land in each region’s trail; aliases resolve per region. That’s how a regional KMS incident stays confined, how least-privilege reviews stay scoped to the right local principals, and how an audit in one region doesn’t pull in evidence from the other. The instrument we want is “shared material, independent policy surface.”

Finally, failover has to be boring. If us-east-1 has a long outage, eu-west-1 keeps writing and reading; if the outage is long enough that we have to promote eu-west-1 to primary, it’s a governance change, a single API call, not a data migration, not a re-encryption ceremony, not a downtime window. The instrument has to distinguish “which region currently happens to be primary” from “which regions hold the material,” and let the first change without disturbing the second.

What we’ll filter on

1. Identical key material across regions. The HSM-held bytes are the same on both sides.
2. Local symmetric decrypt. KMS in eu-west-1 resolves calls against its own regional endpoint.
3. Coordinated key rotation. New material propagates across every region carrying a related key.
4. Graceful region-failure story. Either side can stand alone.
5. Minimal ops. No hand-rolled envelope format, no custom replication code, no rotation orchestrator.

The KMS cross-region landscape

Five candidates.

1. KMS multi-region keys (MRKs). Launched June 2021. A primary key is created in one region; ReplicateKey mints a replica in another region that carries the same key material and the same key id. Each replica is a fully independent KMS resource, its own key policy, aliases, grants, and CloudTrail, but the bytes inside the HSM are shared. Symmetric MRKs encrypt and decrypt interoperably across the set; a wrap made by one member is unwrappable by any other member with no re-encryption and no cross-region call. Rotation propagates automatically across the set.

2. Single-region KMS with cross-region routing (the status quo). Keep the key in us-east-1 and route decrypts from eu-west-1 across the Atlantic, either by SDK default or via a proxy. One key, one region; every European decrypt is a cross-region round trip. A us-east-1 KMS incident takes the European decrypt path with it.

3. KMS grants per region. Grants attach permissions, not key material. A grant on the us-east-1 key permitting a European role to call Decrypt is legal, but the request still has to reach us-east-1. Grants answer who may use the key, not where the key lives.

4. Envelope encryption with a per-region KEK. Generate a data key, encrypt the payload under it, wrap the data key twice, once with a us-east-1 KEK, once with a eu-west-1 KEK, and ship both wraps with the ciphertext. Each region decrypts locally. Works; needs a custom dual-wrap envelope, a custom replication rewriter (S3 SSE-KMS only holds one wrapped key natively), and two independent rotation schedules.

5. External key store / CloudHSM. Move key material outside AWS entirely, or into a CloudHSM cluster. Correct answer when the real constraint is “AWS must not hold the material”; overkill when the constraint is “the material must be in both regions.”

Side by side

Option	Identical material	Local decrypt	Coordinated rotation	Graceful failover	Ops overhead
KMS multi-region keys	✓	✓	✓	✓	✓
Single-region KMS + routing	✗	✗	✓	✗	✓
KMS grants per region	✗	✗	,	✗	✓
Envelope with per-region KEK	✗	✓	✗	,	✗
External key store / CloudHSM	,	,	✗	,	✗

Multi-region keys are the only row with all ticks. Cross-region routing is the problem this scenario exists to replace. Grants move permissions, not material. Per-region envelope encryption meets the local-decrypt requirement only by making the application own a custom envelope format and a custom rotation ceremony. External key stores answer a different question entirely.

Multi-region key architecture

Multi-region keys, in depth

The name misleads. A multi-region key is not a single global key replicated by magic; it is a set of related KMS keys, each a fully-fledged regional resource, that happen to share key material and a common key id.

Primary and replicas. The first key in the set is the primary, created with CreateKey --multi-region true. Further members are replicas, created with ReplicateKey from an existing member. Each replica is a distinct KMS key in its own region, visible in that region’s console, with its own policy, its own aliases, its own CloudTrail events. It is not a mirror; it is a peer.

Shared id, shared material. Every member carries the same key id prefixed mrk-: something like mrk-1234567890abcdef1234567890abcdef. The ARN differs only by region. The material inside the HSM partition is byte-for-byte identical, so a ciphertext wrapped by one member is unwrappable by any other member without translation.

Cryptographic interoperability. For symmetric encryption MRKs, any member can encrypt and any member can decrypt. A data key wrapped by the us-east-1 member is unwrappable by the eu-west-1 member verbatim. For asymmetric MRKs, public keys are identical across the set, so signatures verify anywhere. For HMAC MRKs, the MAC is verifiable anywhere.

Independent policy surface. Policies, aliases, grants, and tags are per-member. The us-east-1 key policy can name local application roles; the eu-west-1 key policy names its local roles. Only the material and the key id are shared, everything governance-shaped is regional.

Coordinated rotation. Enable automatic annual rotation on any member, and the next rotation generates new material shared across the whole set. All multi-region keys in the same set share the same key material, so they automatically share the same key rotation state. No per-replica cadence, no drift risk.

Changing the primary. The primary designation is governance, not cryptography. UpdatePrimaryRegion promotes a replica to primary and demotes the old primary to a replica. Crypto behaviour is unchanged; the new primary becomes the origin for future ReplicateKey calls. This is the failover lever, if us-east-1 has a long outage, promote eu-west-1; when us-east-1 returns, it resumes as a replica with no data-plane action.

What MRKs are not. Not a replication system for ciphertext, the storage layer (S3 CRR, Aurora Global Database, DynamoDB global tables) still has to replicate data. Not a way to cross AWS partitions, commercial, GovCloud, and China each have their own MRK sets. Not a replacement for the key policy, each member’s policy still gates who can use it.

Service support. S3 SSE-KMS, EBS, RDS, DynamoDB, Secrets Manager, and Parameter Store accept MRK ARNs wherever they accept a customer-managed key. S3 CRR preserves the SSE-KMS metadata across regions; the destination KMS in the other region unwraps the data key locally. Client-side encryption SDKs recognise the mrk- prefix and pick up the local member at call time.

A worked encrypt/decrypt trace

Write in us-east-1. A US client PUTs report.pdf to the source bucket. The bucket’s default SSE-KMS configuration names the MRK by ARN. S3 calls kms:GenerateDataKey on the primary; KMS returns a 256-bit plaintext data key and its wrapped form. S3 encrypts the object under the plaintext data key (AES-256-GCM), discards the plaintext from memory, and stores the wrapped data key in the object metadata alongside the key ARN that wrapped it.

S3 Cross-Region Replication. CRR copies the object to eu-west-1 with the SSE-KMS metadata intact. It does not re-encrypt. The wrapped data key travelling with the object is still the one minted by the primary. For a single-region key this would be the dead end; for an MRK it is not, because the material needed to unwrap is present in eu-west-1 too.

Read in us-east-1. A US reader calls GetObject. S3 calls kms:Decrypt on the primary with the wrapped data key, unwraps locally, decrypts the object, streams it back. One KMS call, one region.

Read in eu-west-1. A European reader calls GetObject against the destination bucket. S3 sees the wrapped data key in the metadata and the key ARN that created it, still us-east-1:key/mrk-1234.... S3 calls kms:Decrypt on the eu-west-1 replica with the wrapped data key. Because the replica shares the primary’s material, the unwrap succeeds locally without contacting us-east-1. When the ciphertext’s embedded key id matches a local multi-region member, KMS accepts the call without requiring the caller to rewrite the ARN.

Rotation. Automatic annual rotation is enabled on the primary. When rotation fires, KMS generates new material and shares it across every related member, the replica in eu-west-1 picks it up without operator action. Rotation emits CloudTrail events in every region that holds a member. Old ciphertexts remain decryptable because KMS keeps previous backing material alongside current.

Failover drill. A us-east-1 KMS incident leaves eu-west-1 encrypting and decrypting against the replica unaffected. If the incident is prolonged, UpdatePrimaryRegion promotes eu-west-1 to primary and demotes the old primary to replica. No ciphertext changes, no re-encryption, no downtime for crypto operations. When us-east-1 returns, it resumes as a replica.

Why the three runners-up don’t fit

Cross-region routing. Three costs the scenario explicitly calls out. Latency, 80-100ms trans-Atlantic per read. Blast radius, a us-east-1 KMS incident silently takes the European decrypt path with it, while the European ECS service stays green. Policy confusion, the us-east-1 key policy becomes a bi-regional artefact naming principals from both accounts, edited whenever either region changes.

Grants. A grant on the us-east-1 key permitting European principals to call Decrypt is legal, but the request still reaches us-east-1 because the material is there. Grants are the right instrument when the question is who may use this key temporarily, and the wrong instrument when the question is where does the material live.

Per-region envelope encryption. Useful when a region must hold material that legally cannot be replicated elsewhere, a data-residency constraint where dual-wrap is a feature, not a burden. For a SaaS that genuinely wants the same key in both regions, it duplicates what MRKs provide natively and adds a custom envelope format, a custom replication rewriter, and two rotation schedules.

Subtleties worth knowing

CloudTrail is per-region. Each KMS operation logs in the region where it happened. Investigating a cross-region decrypt pattern reads each trail separately; KMS does not aggregate.

Replicas count toward per-region quotas. Each replica is a customer-managed key in its own region and counts against that region’s limit.

Alias resolution is per-region. An alias alias/customer-data has to be created in every region where code resolves it to a key id. Deployment tooling should create matching aliases at replica-time.

Key policy drift is real. Policies are independent per-member. A replica’s policy can diverge from the primary’s over time; org-wide tooling surfaces drift in external-access findings.

Partitions don’t cross. Commercial, GovCloud, and China each have their own MRK sets; cross-partition replication is not supported.

What’s worth remembering

1. KMS multi-region keys are a set of related regional keys that share key material and key id; each member is an independent KMS resource with its own policy, aliases, grants, and CloudTrail.
2. Primary and replicas: primary created with CreateKey --multi-region true; replicas with ReplicateKey. The designation is governance, all members are peers for encrypt and decrypt.
3. Ciphertexts are interoperable across the set, for symmetric MRKs, a wrap made in one region decrypts in any region where a related key exists, with no re-encryption and no cross-region call.
4. Rotation is coordinated, enabling rotation on one member rotates material for the whole set automatically.
5. Primary can be moved – UpdatePrimaryRegion promotes a replica; failover is a governance action, not a data migration.
6. MRKs don’t replicate ciphertext, the storage layer (S3 CRR, Aurora Global Database, DynamoDB global tables) still replicates data; MRKs handle the key side.
7. Cross-region routing, grants, per-region envelope encryption, and external key stores are different tools, routing fails on latency and blast-radius; grants address permissions not location; per-region envelope encryption needs a custom wrap and duplicate rotation; external key stores answer a different question.
8. Replicas stay within a partition, commercial, GovCloud, and China each have their own sets.
9. Policies are independent per region, drift is a real ops concern; detect it with org-wide tooling.
10. CloudTrail is per-region, each member logs locally; investigations read each trail on its own.