01Measure from both directions
RED—rate, errors, duration—describes request-serving behavior. USE—utilization, saturation, errors—describes resources. Together they connect a user symptom like p99 latency to a cause like GPU queue saturation.
Start outside the system: can a user complete the critical workflow, and how long does it take? Then inspect internal signals such as queue age, database saturation, and dependency errors to explain the symptom. CPU alone does not tell you whether a feature is useful; low CPU can coexist with requests waiting on a failed dependency. Link each dashboard panel to a diagnostic question.
Use rates and distributions carefully. An average of 100 ms may combine many fast requests with a small set taking ten seconds. Track latency percentiles and errors by meaningful bounded dimensions, such as region and operation. Unbounded labels like request_id or arbitrary tenant_id create too many time series. Put per-request detail in logs or traces and use controlled cohorts or targeted analysis for tenant diagnosis.
Alerts should demand an action. “Checkout success is below target and the budget is burning rapidly” gives an operator an urgent symptom. “CPU exceeded 70%” may be useful context but can page during healthy batch work. Define the initial investigation and owner with the alert; otherwise the notification merely transfers an unexplained chart to a tired person.
- Prefer histogram percentiles over averages
- Control high-cardinality labels
- Define SLI exactly at the user boundary
02Reconstruct one request
Propagate a correlation or trace ID through services and asynchronous work. Structured logs make fields queryable; traces preserve causality. Sample intelligently and never log secrets or raw sensitive prompts by default.
A trace connects spans for one request: edge, API, database, queue publish, and later worker processing. Propagate trace context across asynchronous messages rather than stopping at the broker. Attach stable operation and deployment identities so a retry can be distinguished from a new business action. Record enough timing to separate time waiting for a worker from time spent executing.
Logs capture discrete facts: an authorization denial, state transition, or provider error. Metrics summarize populations; traces show selected paths. They answer different questions and should cross-reference each other. Avoid logging secrets or unrestricted document bodies merely to simplify debugging. A redacted error category, source/version ID, and access-controlled diagnostic record may provide the needed evidence.
Sampling creates blind spots. If you sample one percent uniformly, a rare failure may not appear in the retained traces. Consider retaining errors, unusually slow requests, or temporarily targeted traffic, while bounding cost. Absence from sampled traces does not prove an event did not happen. Preserve durable business records for effects that must be audited regardless of observability sampling.
- Carry context in message headers
- Record retry attempt and idempotency key
- Link model, prompt, tool, cost, and outcome
03Rollouts are experiments with brakes
Canaries expose a small traffic slice, feature flags separate release from deployment, and blue-green gives fast environment rollback. Define health gates and automatic halt conditions before rollout.
A canary is an experiment with limited exposure: send a small eligible population to a new version and compare task success, latency, and resource use. Define the stop rule before rollout. A canary without a meaningful metric or rollback owner only limits the initial blast radius; it does not establish that the change is safe.
Rollback can reverse code without reversing data. If a release writes a new schema that old code cannot read, deploying the old binary can worsen the incident. Use an expand-and-contract migration: add compatible structures, deploy readers and writers gradually, migrate data, then remove the old form after rollback is no longer needed. Test mixed-version behavior.
During an incident, separate mitigation from explanation. First reduce customer harm with a reversible action—shed traffic, disable an optional feature, or stop a rollout. Preserve a timeline and evidence for later diagnosis. A useful incident report connects trigger, propagation, detection, mitigation, and a concrete prevention change; “be more careful” does not repair a system mechanism.
- Alert on user symptoms and SLO burn
- Postmortems improve systems, not blame people
- Capacity plans need lead time for scarce resources