01An SLO is a product promise
99.9% allows about 43.8 minutes of downtime per 30-day month; 99.99% about 4.38 minutes. Multiplying serial dependency availability reveals why every synchronous hop spends budget.
Define the service-level indicator before choosing a target. For checkout, count valid purchase attempts that finish correctly within a deadline, not successful health checks on the API process. State exclusions and measurement location. A 99.9% success target permits one bad attempt per thousand over its window; a time-based availability target over a 30-day month permits 43.2 minutes. These are different denominators.
An error budget turns the target into a change policy. When the service has consumed most of its budget, slow risky releases and prioritize the dominant failure source. Do not spend days eliminating harmless internal errors while customers cannot complete checkout. Slice the indicator by region or critical customer class so an aggregate success rate cannot hide a concentrated outage.
For sequential required dependencies, success requires all stages. Under an illustrative independence assumption, three 99.9% available stages yield 0.999³ ≈ 99.70% availability. Real failures may be correlated, so multiplying advertised numbers is not a guarantee. Use the calculation to find dependency risk, then examine shared networks, stores, credentials, and deployment paths.
- Measure the user journey, not server uptime
- Error budgets govern release vs reliability work
- Tail latency can violate an SLO before total outage
02Redundancy must cross failure domains
Two replicas on one host do not protect against host loss; two AZs may not meet a three-AZ quorum strategy. Active-active improves recovery but adds state and conflict complexity.
Two application instances on one host survive a process crash but not host loss. Two zones can still depend on one regional control plane or database. Name the failure domain each copy crosses, and trace whether the surviving path has credentials, configuration, data, and capacity. Redundancy only helps if the replacement is both independent enough and able to serve.
RPO is the maximum accepted data-loss interval; RTO is the target recovery duration. If asynchronous replication can lag 30 seconds, an immediate failover may lose recent accepted writes even when recovery takes only a minute. Decide whether to wait, recover from another log, or explicitly reconcile missing operations. A faster failover does not by itself improve RPO.
A restore test is part of the design. Restore a representative backup, apply subsequent logs, validate business invariants, and measure time until useful traffic can resume. Include DNS or routing changes, caches, and dependent clients. A runbook that ends at “database starts” has not established that checkout works.
- Failover detection can be riskier than failover mechanics
- Exercise backups by restoring them
- Know RPO (data) and RTO (time)
03Stop failure from spreading
Bound every call with a timeout and concurrency limit. Circuit breakers stop futile work. Bulkheads reserve resources. Jitter avoids synchronized retry waves. Graceful degradation preserves the critical product path.
A slow dependency consumes callers' connections and worker slots. If those callers keep accepting new requests, a local problem spreads. Bound concurrency per dependency, set deadlines, and shed work before shared resources are exhausted. A bulkhead reserves separate capacity so an optional recommendation call cannot consume every checkout worker.
A circuit breaker stops repeated calls to a failing dependency for a period and probes for recovery. It complements timeouts and budgets; it does not decide what response is safe. An unavailable recommendation can be omitted, while an unavailable payment authorization cannot be replaced by invented success. Define degraded behavior by workflow and test it with the dependency actually unavailable.
Retries and fallbacks need capacity too. If every request fails over to a smaller pool, the fallback becomes the next outage. Reserve headroom, reduce admitted traffic, or lower the service level deliberately. During recovery, ramp traffic rather than releasing an entire backlog at once. Reliability includes returning to steady state without a second failure wave.
- Caller deadline should shrink downstream
- Shed cheap and early
- Prefer static/history over total failure when safe