01Partial failure creates uncertainty
If a request times out, the remote node may be down, the network may be slow, the response may be lost, or the operation may have committed. APIs need stable IDs and retriable semantics because certainty is unavailable.
A process that cannot hear from another process cannot immediately distinguish crash from delay. Consider a scheduler whose worker pauses for a minute. Its lease expires and a replacement receives the job; then the old worker resumes. Both may believe they can write. A lease limits authority in time, but a paused process cannot be trusted to notice expiry before acting.
A fencing token gives each new owner a monotonically increasing generation. The storage service remembers the highest accepted generation and rejects older ones. If owner A holds token 7 and owner B obtains 8, A's later write with 7 is refused after 8 has been installed at the protected resource. Every relevant write path must enforce the rule. A token checked only in the client offers no protection from a stale client.
Use wall-clock time for human timestamps and monotonic time for local elapsed durations. Clock adjustments can make wall-clock intervals negative or misleading. Logical clocks express ordering relationships without promising exact physical time. None of these clocks turns a timeout into proof of failure; your protocol still needs an explicit response to uncertainty.
- Use monotonic clocks for elapsed time
- Leases expire; stale holders need fencing tokens
- Health checks are observations, not truth
02CAP is a partition-time statement
When communication between replicas is partitioned, a given operation cannot both always respond and remain linearizable. Systems make different choices by operation; outside partitions, latency/consistency tradeoffs still exist under PACELC.
Imagine replicas in regions A and B with a broken link. A accepts a change that B cannot observe. If B must always answer reads, it can return the old value. If every completed read must reflect the latest completed write, B may have to reject or wait. CAP describes this conflict during a partition; saying “pick any two” without specifying the failed communication and affected operation misses the mechanism.
A system can make different choices for different operations. A product catalog may return a stale description, while reserving the last item must reach the current inventory owner or fail. State whether “consistent” means linearizable access, read-your-writes, monotonic reads, or eventual convergence. These promises are different and have different implementation costs.
Quorum intersection is a useful condition, not a complete protocol. With N = 3, writes to two replicas and reads from two overlap at at least one replica. You still need rules for versions, concurrent writes, failures during a write, and which value a read chooses. Do not infer linearizability merely from R + W > N. Outside a partition, contacting distant replicas also creates latency tradeoffs.
- Replication alone does not create availability
- Quorum intersection can preserve a single history
- Eventual consistency needs conflict and convergence rules
03Coordinate only what must agree
Consensus maintains a fault-tolerant ordered log. It is useful for metadata, membership, and ownership but expensive in every data operation. Scale by partitioning ownership and making operations idempotent or commutative.
Consensus lets replicas agree on a sequence despite some nodes failing. In a leader-based replicated log, the leader proposes entries, replicas persist them, and the protocol decides when an entry is committed. A new leader must preserve committed history. A three-member majority group can continue with one member unavailable, but not with two; membership changes need protocol rules too.
Use agreement for decisions whose conflict would break an invariant: selecting an owner, changing placement metadata, or ordering noncommutative state transitions. Avoid funneling unrelated independent work through one global agreement point. Partition ownership so different keys can progress separately, while retaining a clear rule for operations spanning owners.
Sometimes change the operation instead of adding coordination. Adding independent counts can be made commutative; assigning each tenant a single writer can remove cross-tenant contention. Convergent replicated structures require carefully defined merge semantics, not an arbitrary “last writer wins” patch. Ask what conflicts the business can accept, how duplicates are handled, and whether a merged state still satisfies the invariant.
- Majority tolerates minority failure
- Leader election does not prevent stale actors by itself
- Exactly-once is usually deduplication plus transactions within a boundary