Caching

A fast, usually in-memory store that keeps copies of frequently accessed or expensive-to-compute data so future requests can be served quickly without hitting the slower source of truth.

A hit is when requested data is found in the cache. A miss is when it isn't, forcing a fetch from the origin (DB/service) and typically a cache population.

The fraction of requests served from cache: hits / (hits + misses). A higher ratio means more requests avoid the origin. It is the primary metric for cache effectiveness.

Redis and Memcached are the most common distributed in-memory caches. Others include Caffeine/Ehcache (in-process JVM), and CDNs like Cloudflare/Akamai for edge caching.

Time-to-live: an expiry duration after which a cached entry is considered stale and removed or refreshed. TTLs bound how stale data can get without explicit invalidation.

Cache memory is limited and expensive, and cached data goes stale as the source changes. You cache hot, reusable, tolerably-stale data and evict/expire the rest to stay within memory and freshness constraints.

The rule for choosing which entry to drop when the cache is full — e.g. LRU (least recently used), LFU (least frequently used), FIFO, or random.

The application first checks the cache. On a hit it returns the value; on a miss it reads from the database, writes the result into the cache (usually with a TTL), and returns it. Only requested data is cached. Pros: simple, resilient to cache failures. Con: first request per key is always a miss, and there's a window where cache and DB can disagree.

Write-through: every write updates cache and DB synchronously, so the cache is always consistent but writes are slower. Write-back (write-behind): writes go to the cache and are flushed to the DB asynchronously; writes are fast and DB load is smoothed, but a cache crash before flush can lose data.

LRU evicts the item unused for the longest time — good for recency-driven access. LFU evicts the least-frequently accessed item — good when some items are persistently popular. LRU can evict a popular item after a burst of one-off scans; LFU resists that but needs frequency counters and can keep stale "once popular" items unless it decays counts (e.g. TinyLFU / W-TinyLFU).

Options: explicit invalidation (delete/update the key on write), short TTLs (accept bounded staleness), or event-driven invalidation (publish change events that caches subscribe to). Deleting on write plus cache-aside repopulation is common; versioned keys (embedding a version in the key) sidestep invalidation by making new data a new key.

Local (e.g. Caffeine): fastest (no network), but each instance has its own copy — memory duplication and consistency drift across nodes. Distributed (e.g. Redis): shared across all instances, consistent view, larger capacity, but adds a network hop and its own availability concerns. Many systems use both: a small local L1 cache backed by a shared L2.

Memcached: simple multithreaded key/value cache, great for pure string/object caching at scale. Redis: single-threaded (per shard) but offers rich data types (lists, sets, sorted sets, hashes), persistence, pub/sub, Lua scripting, and replication/cluster. Pick Redis when you need those features or durability; Memcached for the simplest, memory-efficient caching.

A single key (e.g. a celebrity's profile) receives a disproportionate share of traffic, concentrating load on the one cache node that owns it — a hotspot that can saturate that node's CPU/network. Mitigations: replicate the hot key across nodes, add a local cache tier in front, or split/shard the value.

Browser cache, CDN/edge, reverse proxy (NGINX/Varnish), application-level object cache, distributed cache (Redis), ORM/query cache, database buffer pool, and even OS page cache and CPU caches. Each layer trades freshness for proximity/speed.

Preloading the cache with likely-needed data before it takes production traffic (or before a known spike), so early requests don't all miss and stampede the origin. Needed after deploys/restarts that flush the cache, before flash sales, or when a cold cache would overload the database.

Cache per (query + filters + page) key with a short TTL, since result sets change and combinations explode. Alternatively cache the underlying objects by ID and only cache the ordered list of IDs per query, so object updates are reflected without re-caching every page. Beware unbounded key cardinality from arbitrary filter combinations.

When a popular key expires, many concurrent requests miss simultaneously and all hit the database to recompute the same value, spiking load. Prevention: (1) request coalescing / single-flight — only one request recomputes while others wait for the result; (2) a lock/lease so a single worker refreshes; (3) probabilistic early expiration — refresh slightly before TTL with a randomized window; (4) serve stale-while-revalidate. Combining coalescing with jittered TTLs is robust.

Cache penetration is repeated requests for keys that don't exist anywhere (often malicious), so every request misses the cache and hits the DB. Defenses: cache the negative result (store a "not found" sentinel with a short TTL), and use a Bloom filter of existing keys to reject queries for keys that definitely don't exist before touching cache or DB.

A large set of keys expire at (nearly) the same moment — or the cache tier restarts — so a flood of misses hits the origin at once, potentially collapsing it. Mitigations: add randomized jitter to TTLs so expirations spread out, use multi-level caches, replicate the cache for availability, and apply rate limiting / circuit breakers at the origin so a cold cache can't take down the database.

Classic race: reader misses and fetches value V1 from DB; meanwhile a writer updates the DB to V2 and deletes the cache key; then the slow reader writes V1 into the cache — now the cache holds stale V1 indefinitely. Mitigations: delete-after-write ordering with a short TTL as a backstop, "delete then delete again after a delay" (delayed double delete), or use versioned values / compare-and-set so a stale write can't overwrite a newer one. Change-data-capture-driven invalidation avoids the read-modify race entirely.

There is no perfect consistency without cost. Common approaches: (1) invalidate (delete) on write and lazily repopulate; (2) write-through for read-your-writes on the same key; (3) TTLs to bound staleness; (4) CDC/binlog tailing (e.g. Debezium) to invalidate caches on committed DB changes, decoupling it from app code. For strong needs, read from the DB with the cache only as a best-effort accelerator, or use leases/versioning. Accept eventual consistency where the domain allows.

It maps keys and cache nodes onto a ring so each key belongs to the next node clockwise. Adding or removing a node only remaps ~1/N of keys instead of nearly all keys (as plain modulo would), preserving most cache entries during scaling or failures. Virtual nodes balance load across physical nodes. This minimizes the miss spike and DB load when the cache cluster changes size.

Cache rendered feed pages/objects in a distributed cache with short TTLs plus invalidation on edits. For viral (hot) posts, add a local in-process L1 cache and replicate the hot key across nodes to avoid a single-node hotspot; use request coalescing to prevent stampedes on recompute. Serve stale-while-revalidate for smoothness, push heavily-read static assets to a CDN, and pre-warm the cache for trending content. Cap fan-out cost by caching per-object and assembling feeds from cached objects.

Expiration removes entries because their TTL elapsed (data-driven freshness). Eviction removes entries because the cache is out of memory (capacity-driven), regardless of TTL. It matters for capacity planning: if eviction is happening before TTLs, your working set exceeds cache memory and hit ratio suffers — you need more memory or better key scoping. Monitoring evicted-vs-expired counts reveals which pressure dominates.

Prefer not to cache the authoritative value, or cache with write-through plus read-your-writes on the same node, and always perform the actual balance-changing transaction against the database under proper isolation/locking. If caching is needed for reads, use very short TTLs and invalidate on every write, and never make financial decisions from a cached read — validate against the DB in the transaction. Often the right answer is "don't cache the truth; cache derived/display data."

A policy where, after an entry's freshness expires, the cache keeps serving the stale value to clients for a grace window while asynchronously refreshing it in the background. This eliminates latency spikes and stampedes at expiry at the cost of briefly serving slightly stale data. It's an HTTP cache directive and is ideal for content that tolerates seconds of staleness (feeds, dashboards, product listings).

Profile access patterns to find the working set and skew (often Pareto: ~20% of keys serve ~80% of traffic). Size memory to hold the hot working set so eviction stays low; measure hit ratio and the marginal hit-ratio gain per extra GB (diminishing returns). Cache items that are read far more than written, expensive to compute, and tolerant of staleness. Avoid caching rarely reused, huge, or highly volatile data. Continuously monitor hit ratio, latency, eviction rate, and memory fragmentation.

The system had no defense against losing the cache: all traffic fell through to a database sized only for cache-miss volume. Prevent it with cache high availability (replicas/cluster so one node loss isn't total), request coalescing and rate limiting/load shedding at the origin, circuit breakers, a local L1 fallback, and capacity headroom or graceful degradation (serve reduced functionality) when the cache is unavailable. Treat "cache is down" as an expected failure mode in load testing.