01Choose a write topology per domain
A home-region model gives a single authority and simple conflict behavior at the cost of remote write latency. Multi-leader improves local writes but forces merge semantics. Some data—preferences—merges naturally; other data—unique ownership—does not.
Assign write ownership per data domain. A single home region simplifies ordering for an account but adds remote-write latency. Active-active writes let multiple regions accept changes, so you must define conflict handling. Append-only events with globally unique IDs are easier to merge than two regions both decrementing the last available item. Geography does not remove the invariant.
Walk through a regional partition. Which region can still acknowledge writes? What do remote readers see? If two sides both accept conflicting updates, how will you reconcile them? A last-write-wins timestamp policy can discard a legitimate change and depends on clock behavior; it is a product decision with consequences, not a generic correctness repair.
Reads can often be local even when writes have a home, but read-your-writes then needs a strategy such as temporarily routing to the owner or waiting for a replication position. Record data ownership in routing metadata and design ownership changes as a protocol. A DNS change alone cannot safely transfer authority over mutable state.
- Read replicas reduce latency but expose lag
- Session routing can preserve read-your-writes
- Keep control plane failure separate from data plane
02Residency constrains architecture
Data residency can dictate storage and processing location, backup geography, and operator access. Make the policy explicit before global replication.
Treat residency as a supplied constraint on particular data classes. Customer documents, account metadata, usage logs, backups, and derived embeddings may have different placement rules. Create a table showing allowed locations for each class and every processor. A primary database in an allowed region does not help if its backup or diagnostic export leaves that boundary.
Follow the whole lifecycle: ingestion, replication, support access, processing, backup, deletion, and disaster recovery. If a region becomes unavailable and data cannot leave its allowed geography, recovery must use permitted capacity or accept downtime. The design should expose that tradeoff before an incident. Obtain the applicable requirements from the product and legal owners rather than inferring law from a region name.
Deletion also propagates across copies and derived indexes. Keep versioned deletion intent so an older replication event cannot resurrect data. Define what remains in backups, who can restore it, and how restored data is brought back into compliance with deletion records. This is a recovery workflow, not just a delete statement.
- Classify data before placing it
- Keep region-independent identifiers
- Know which dependencies are actually regional
03Evacuation is a capacity event
Failover succeeds only if the destination has capacity, credentials, data, configuration, and tested runbooks. DNS/edge routing is the smallest part. Plan write fencing, backlog replay, and failback.
Capacity must exist where traffic will land after failure. Suppose two regions normally handle 400 requests/s each and each can serve at most 600 at the required latency. Losing one leaves 800 requests/s for a 600-capacity survivor. The healthy region still needs load shedding, degraded behavior, or more reserved capacity. Steady-state utilization alone concealed the risk.
Failover has several decisions: detect a problem, establish the old writer cannot safely continue, promote a valid replica, route traffic, and validate product behavior. With asynchronous replication, identify the last durable position and any accepted writes that may be missing. Refuse unsafe writes or expose a recovery state instead of quietly pretending the new replica contains all history.
Failback needs equal care. The recovered region may contain divergent state, stale jobs, or old credentials. Reconcile data, rejoin replication, warm dependencies, and shift traffic gradually. Practice the procedure and measure RTO and RPO with actual failure injection. A diagram with two regions is only a starting hypothesis about recoverability.
- Measure replication lag
- Exercise partial and full-region loss
- Prevent two regions from believing they are sole writer