01A cache is a prediction
You predict that the same result will be needed again before it becomes invalid. Estimate working-set size and hit rate. A 95% hit rate can reduce source load 20×; a 50% hit rate often does not justify major complexity.
Calculate misses before celebrating hits. At 10k reads/s and 95% hits, the source receives 500 reads/s. At 90%, it receives 1,000. The cache looks healthy in both cases while database demand doubles. Measure the hit rate by traffic class and key popularity; an average can hide one expensive query that misses often. Include fill cost and bytes, not only lookup count.
Cache-aside means the application reads the cache, fetches the source on a miss, then fills the cache. This creates a race at expiry: many requests can all miss before any fill finishes. If a hot key gets 3,000 requests/s and recomputation takes 200 ms, about 600 callers arrive during that window. Request coalescing lets one recomputation serve them, subject to bounded waiting and failure handling.
Choose placement from reuse. A browser cache avoids a network trip; a CDN serves reusable public content near readers; an application cache can share expensive results across requests. The key must include every dimension that changes the result, including tenant, locale, or data version where appropriate. Omitting a dimension can return a fast but incorrect—or unauthorized—answer.
- Key includes every input that changes output
- Capacity targets the hot working set, not the full corpus
- Negative caching prevents repeated misses
02Invalidation is product semantics
Ask how stale is acceptable. TTL gives bounded staleness and simple failure behavior. Event invalidation is fresher but can lose or reorder events. Versioned keys make old values unreachable and simplify rollouts.
A TTL bounds how long an entry may remain without refresh; it is not a guarantee that the cached value matches the source throughout that time. If an account balance changes immediately after fill, a one-minute TTL can serve the previous balance for almost a minute. Decide whether that is acceptable for the specific action. A display may tolerate a stale estimate while a transfer must check authoritative funds.
Invalidation has ordering races. A reader fetches version 4, a writer commits version 5 and deletes the cache, then the reader fills version 4. Deleting on write alone did not prevent a stale refill. Options include versioned keys, conditional fills, or accepting bounded staleness with a clear refresh policy. The right mechanism follows the product's freshness contract.
Jitter spreads expiry across many keys; it does not stop concurrent misses for one viral key. Combine protections deliberately: coalescing for the same key, early refresh or bounded stale responses where allowed, and source admission limits when fills fail. On permission revocation, decide whether cached data can remain and where authorization is rechecked. “The cache expires eventually” may be too weak for access control.
- Use jittered TTLs to avoid synchronized expiry
- Single-flight/coalescing suppresses stampedes
- Serve stale during source trouble only when semantics allow
03Performance is broader than caching
Batch network calls, paginate unbounded results, compress large payloads, precompute expensive aggregates, and pool connections. Optimize the bottleneck you can name and measure.
Break latency into queueing, computation, storage, and network time. A cache only helps the reusable portion. If most latency is a full table scan caused by a missing index, repairing the query can be simpler and more predictable than introducing cache invalidation. If payload transfer dominates, compression or pagination may help more. Measure before choosing a mechanism.
Tail latency is especially sensitive to saturation. As arrival rate approaches service capacity, small bursts have less spare capacity to drain, so queues grow and slow requests become much slower. Average latency can remain acceptable while p99 fails. Load-test the intended traffic distribution, including cold starts, expensive keys, and long payloads; a warm uniform-key benchmark is an optimistic special case.
Test the cache as a dependency that can disappear. If losing it sends all traffic to a source sized for 5% misses, the source may face twenty times its expected load. Protect it with bounded concurrency, admission, and a planned degraded response. Capacity and correctness must remain understandable during a cold fleet, failed refresh, and full cache outage.
- Batching trades latency for throughput
- Precompute trades write/storage cost for read latency
- Connection pools need queue and timeout bounds