01Protocol follows conversation shape
REST is excellent for public resource APIs. gRPC fits typed internal request/response and streaming. SSE is simple server-to-client streaming over HTTP. WebSockets fit bidirectional, long-lived interaction. HTTP/2 reduces connection pressure; HTTP/3 improves behavior under packet loss.
Choose a transport by drawing a conversation. A browser asks for an existing invoice and gets one response: ordinary HTTP works. A report streams progress from server to browser: SSE can carry named events over an HTTP response. A collaborative editor sends edits in both directions throughout a session: a WebSocket may fit. These choices describe communication shape; they do not establish delivery, replay, or authorization guarantees.
A stream needs an application protocol. Give events an ID, type, payload, and explicit completion or error meaning. After a disconnect, can a client resume from the last event, fetch a durable result, or only restart? Sending a final text fragment is not the same as committing the operation. If a server has already performed a side effect, reconnecting must recover that operation rather than silently create another.
Connection reuse and multiplexing can reduce setup costs, but a protocol upgrade cannot remove a slow database query or an overloaded worker. Include DNS, connection setup, proxy buffering, and downstream processing in a latency trace. Measure the path you actually deploy, because a locally smooth stream can arrive in buffered bursts behind an intermediary.
- Prefer the least stateful protocol that meets the need
- Streaming changes cancellation, backpressure, and observability
- Connection reuse matters before protocol trivia
02Retries are load amplifiers
A timeout does not prove failure; it proves the caller stopped waiting. Retrying a non-idempotent operation can duplicate side effects. Bound retries, add jitter, propagate deadlines, and use idempotency keys at the effect boundary.
A timeout means you stopped waiting; it does not tell you whether the remote effect occurred. A payment might commit before its response is lost. Retrying with a new identity then creates another payment. Define an idempotency key scoped to the caller and operation, bind it to the request parameters, and save the result at the effect boundary. Reusing a key with different parameters should be rejected.
Allocate one end-to-end deadline. With 800 ms total, 100 ms already spent, a 300 ms failed attempt, 50 ms backoff, and 100 ms reserved for response handling, only 250 ms remains for another attempt. Pass a remaining duration downstream. Independent full timeouts at each layer can make a supposedly bounded request continue long after the caller has left.
Retry only failures that might improve and only while budget remains. Use backoff with jitter to avoid synchronized retry waves, and limit concurrency as well as attempt count. If three layers each make up to three attempts, the leaf may receive 27 calls. Choose one retry owner where possible; otherwise coordinate a shared budget. Cancel work on disconnect when the operation contract permits it.
- Retry transient failures, not every 4xx
- Budget attempts inside the user deadline
- Deduplicate at the component that commits the effect
03Boundaries are organizational and operational
Split when a domain needs independent ownership, scaling, security, or release cadence. Each synchronous hop increases latency and correlated-failure surface, so keep the critical path short.
A service boundary owns a contract, data, deployment behavior, and an on-call burden. Splitting an application into five services creates five independent failure opportunities and several network calls. Start by grouping responsibilities that must change or commit together, then split when ownership, scaling, security, or fault isolation justifies the operational cost. A module boundary is often enough initially.
Define compatibility before independent deployment. Adding an optional response field is usually easier to roll out than renaming a required field. Deploy readers that tolerate the new representation before writers depend on it. For streams, version event types and specify how clients treat unknown events. An internal API also needs compatibility rules if its clients and server deploy at different times.
Load balancing distributes requests among eligible backends; it cannot make every backend healthy or every request equally expensive. Readiness should reflect whether a process can serve new work, while overload admission can refuse work before queues grow without bound. Record request IDs across boundaries so you can connect the client-visible failure to the dependency attempt that caused it.
- High cohesion inside, low coupling across
- Prefer stable contracts over shared tables
- Do not begin with microservices as a status symbol