01Choose by semantics
A queue gives one worker ownership of a task. A log retains ordered partition history so multiple consumer groups can independently replay it. Kafka ordering is per partition—not global—so partition by the entity whose events must stay ordered.
A queue distributes work that should be processed, often allowing messages to disappear after acknowledgement. A durable log retains an ordered sequence that consumers track by position and can replay. Both can deliver asynchronously, but replay, retention, consumer independence, and ordering determine which fits. A notification send and a rebuildable analytics projection have different needs.
Follow one message through states: accepted, persisted, available, delivered, processing, effect committed, acknowledged. The producer must know what broker acceptance guarantees. The consumer must know when redelivery can occur. Ordering usually applies within a chosen partition or key, not across all events. If all events for an account must be ordered, map that account consistently and plan for hot accounts.
A schema is a long-lived contract because retained messages may be replayed after code changes. Include stable event identity, event time, schema version, and the data needed by the consumer. Decide whether an event records a historical fact or asks the receiver to read current state. Those meanings differ when the source has changed since the event was produced.
- Events describe facts; commands request actions
- Offsets are consumer progress, not business completion
- Retention enables replay and new consumers
02At-least-once is the practical default
A worker can finish the side effect and crash before acknowledgment, so redelivery happens. Make handlers idempotent with a stable operation key, transactional state transition, or effect ledger.
A worker can commit a database update and crash before acknowledgement. Redelivery then repeats the work. If the effect and deduplication record share one transactional database, insert the message ID and perform the update atomically; a duplicate sees the existing ID and skips the effect. Merely setting a “processed” flag after the effect leaves a crash window.
External effects are harder. A sent email cannot be rolled back with your database. A provider idempotency key or status lookup can help reconcile an ambiguous send. Without either, document whether you accept possible duplicates or possible omission. A blanket “exactly once” label hides this boundary. Broker delivery guarantees do not automatically cover a remote business effect.
The transactional outbox solves a different gap: committing application state and recording the intent to publish. Write both in one transaction; a relay later publishes outbox rows. A relay crash after publish but before marking the row sent can produce another delivery, so consumers still need duplicate handling. The outbox prevents lost intent; it does not eliminate every duplicate.
- Exactly-once claims have a system boundary
- Retries need exponential backoff and jitter
- Dead-letter queues need an owner and replay procedure
03Backpressure is part of the API
When arrival exceeds service rate, queues grow and latency becomes unbounded. Bound the queue, reject or defer low-priority work, shed optional load, and expose truthful status.
Queues absorb temporary imbalance, not unlimited overload. If producers add 1,000 jobs/s and workers finish 800, backlog grows by 200/s. After ten minutes that is 120,000 extra jobs. Even if throughput later returns to normal, you need spare capacity to drain them. Estimate waiting time from backlog and completion rate, and observe oldest-message age to connect backlog to user impact.
Set bounds at every stage: accepted work, in-flight messages per worker, retry attempts, message age, and dead-letter retention. Retry transient failures with delay; separate poison messages that will fail identically. In a strictly ordered partition, skipping a poison event may violate business order, while stopping blocks following events. Make that tradeoff explicit instead of silently moving it aside.
Backpressure communicates that the system cannot accept work at the promised pace. You can reject, defer, lower priority, or offer an asynchronous completion time. Expose truthful status so clients do not create duplicates by resubmitting. Protect high-priority jobs from a flood of bulk work through separate pools or scheduling policy, then test whether retries respect the same limits.
- Track queue age, not only depth
- Reserve capacity for high-priority tenants
- Admission control belongs before scarce work