Observability and Caching Strategies at Scale

How the World's Top Companies Instrument and Cache Their Systems

Part A: Why Observability Matters Today

The World Has Changed

A decade ago, a company ran 3 servers, one database, and one application.
When something broke, an engineer logged into the server, looked at a log file, and fixed it.

Today, a company like Netflix runs 700+ microservices.
A single user action triggers 100+ service calls across dozens of teams' codebases.
When something breaks, no single engineer knows the full picture.

Observability is the ability to understand what your system is doing from the outside, purely by looking at the data it produces.

What Changed and Why Observability Became Critical

1. Microservices Shattered Visibility

• Monolith: One log file, one process, one thread dump. Debugging was local.
• Microservices: A checkout failure could originate in payment service, inventory service, fraud service, or the network between them.
• Without distributed tracing, you see symptoms (checkout failed) but not the cause (fraud service timed out because Redis was full).

2. Cloud Infrastructure is Ephemeral

• Virtual machines and containers start and die in seconds.
• There is no "log in to the server" - the server may not exist anymore.
• You must capture everything before the container disappears.
• Kubernetes restarts pods automatically. If you did not observe the crash, the evidence is gone.

3. Customer Experience is Revenue

• Amazon's study: Every 100ms of latency costs 1% in sales.
• Google found: A 500ms increase in search latency caused 20% fewer searches.
• Netflix measures: Every 1% increase in rebuffering ratio causes 0.4% increase in cancellations.
• Without observability, you cannot connect system metrics to business outcomes.

4. On-Call Engineers Cannot Know Everything

• A team manages 5 services they own. But their service depends on 20 others.
• At 3 AM when something breaks, the on-call engineer has minutes to diagnose.
• Observability gives them the answers without needing to know every service's internal design.

5. Regulatory and Compliance Requirements

• Banks must audit every transaction. Financial regulators require proof of system behavior.
• GDPR requires knowing exactly where data flows. You cannot prove this without observability.
• PCI-DSS requires monitoring all access to cardholder data. This is observability by regulation.

6. Cost of Downtime Has Escalated

• Facebook 6-hour outage in 2021: $100 million in lost revenue.
• AWS us-east-1 outage in 2017: Thousands of companies went down.
• Without observability, MTTD (Mean Time to Detect) and MTTR (Mean Time to Recover) are too long.

The Three Pillars of Observability

Pillar 1: Metrics

• Numerical measurements over time
• "How many requests per second?", "What is the error rate?", "How much CPU?"
• Aggregated data, stored efficiently as time series
• Good for dashboards, alerting, trending
• Limitation: Metrics tell you WHAT is wrong but not WHY

Pillar 2: Logs

• Textual records of discrete events
• "User 12345 attempted payment", "Database connection failed"
• High detail, high volume, expensive to store and query
• Good for debugging specific incidents
• Limitation: Logs from 100 services for one request are scattered and unconnected

Pillar 3: Distributed Traces

• End-to-end record of a single request as it flows through multiple services
• Shows every service involved, time spent in each, where failures occurred
• Connects the dots between Metrics (something is slow) and Logs (what happened)
• Essential in microservices architectures
• Limitation: High cost to store all traces - sampling is necessary

The Fourth Pillar (Emerging): Continuous Profiling

• Captures CPU, memory, and I/O usage at the code level in production
• Shows exactly which line of code is consuming most resources
• Tools: Pyroscope, Parca, Datadog Continuous Profiler
• Used By: Google (pprof), Uber, Netflix

Observability Tools and Who Uses What

SLI, SLO, SLA, Error Budget - The Business Language of Observability

Service Level Indicator (SLI)

• A specific metric that measures service quality from a user perspective
• Examples: Request success rate, latency P99, availability percentage

Service Level Objective (SLO)

• The internal target for an SLI
• Example: 99.9% of payment requests must succeed within 500ms
• The team's engineering commitment to themselves

Service Level Agreement (SLA)

• The external contractual commitment to customers
• Always looser than SLO (Buffer for the unexpected)
• Breach triggers financial penalties or credits

Error Budget

• Error Budget = 100% - SLO
• If SLO is 99.9%, error budget = 0.1% = 8.77 hours of downtime per year
• Error Budget is burned during incidents, deployments, and experiments
• Google SRE Philosophy: If error budget is healthy, ship features fast. If burned, freeze deployments and fix reliability.
• This creates alignment between product teams (who want to ship) and SRE teams (who want stability)

Part B: Caching Strategies - How the World's Top Companies Do It

Why Caching Exists

Every system has a speed hierarchy:

• L1 CPU Cache: ~1 nanosecond
• RAM / In-Process Cache: ~100 nanoseconds
• Redis / Memcached (same data center): ~500 microseconds
• Database Query (same data center): ~1-10 milliseconds
• Cross-region database read: ~100-300 milliseconds
• Third-party API call: ~200-2000 milliseconds

Caching moves frequently accessed data up this hierarchy to eliminate the expensive lower-level fetch.

Core Caching Patterns

1. Cache Aside (Lazy Loading)

• Application checks cache first
• On cache miss: fetch from database, store in cache, return to caller
• On cache hit: return from cache directly
• Cache is populated only when data is actually requested
• Used When: Data access patterns are unpredictable, not all data needs to be cached
• Risk: Cache miss causes a database hit (thundering herd if many misses at once)

2. Write-Through Cache

• Every write goes to cache AND database simultaneously
• Cache is always consistent with database
• Higher write latency (must wait for both cache and DB write)
• Cache always has fresh data
• Used When: Read-heavy workloads where cache consistency is critical

3. Write-Behind (Write-Back) Cache

• Write goes to cache immediately, database write is asynchronous (later)
• Very low write latency (only cache write blocks the call)
• Risk: Data loss if cache fails before async DB write completes
• Used When: Very high write throughput where slight data loss is acceptable

4. Read-Through Cache

• Application always reads from cache layer
• If cache miss, the CACHE layer fetches from database (not the application)
• Application code is simpler (always talks to cache)
• Used When: Cache provider supports read-through (Redis modules, NCache)

5. Refresh Ahead (Pre-Fetching)

• Cache proactively refreshes data BEFORE expiry
• Based on predicted access patterns
• Avoids cache miss latency for popular content
• Used By: Netflix (pre-positions next episode before you finish current one), CDNs

Cache Invalidation - The Hard Problem

"There are only two hard things in computer science: cache invalidation and naming things." - Phil Karlton

TTL-Based Expiry

• Set Time-To-Live on cache entries
• After TTL expires, entry is removed or refreshed
• Simple, works well for data that changes infrequently
• Risk: Stale data served until TTL expires

Event-Based Invalidation

• When database data changes, publish an event
• Cache listener receives event and invalidates the specific key
• Requires event streaming infrastructure (Kafka, Redis Pub/Sub)
• More complex but more accurate than TTL
• Used By: Facebook (McRouter invalidates Memcached via TAO events)

Write-Through Invalidation

• On every write to database, also update or delete the cache key
• Strong consistency but requires every write path to be cache-aware
• Used By: Twitter, Amazon product catalog

Stampede Prevention (Cache Dog-Piling)

• Problem: A popular key expires. Thousands of requests hit the database simultaneously.
• Solution 1: Probabilistic Early Expiration (PER) - Expire the key slightly before actual TTL based on random probability
• Solution 2: Mutex Lock - First request gets a lock, fetches from DB, others wait for the lock to release
• Solution 3: Background Refresh - Always serve (possibly stale) cached value, refresh in background
• Used By: Reddit (implemented mutex-based stampede prevention in Python), Facebook

Amazon - Caching Strategy

Scale Context

• 350 million products in catalog
• 1.5 billion items in shopping carts globally
• Prime Day: 100,000 orders per minute at peak

ElastiCache as the Primary Cache Layer

• Amazon uses ElastiCache (managed Redis/Memcached on AWS) extensively
• Two modes: Memcached for simple key-value, Redis for sorted sets, pub/sub, persistence
• Product detail pages cached in Memcached (simple key-value, no persistence needed)
• Session data cached in Redis (persistence and replication required)

Multi-Layer Cache for Product Pages

Browser / CDN Edge (CloudFront)
    -> API Gateway Cache (Response cache for identical queries)
        -> ElastiCache Redis (Product detail, pricing, availability)
            -> Aurora PostgreSQL / DynamoDB (Source of truth)

DynamoDB Accelerator (DAX)

• Amazon built DAX specifically as a write-through cache for DynamoDB
• Sits in front of DynamoDB tables
• Reads served from in-memory cache in microseconds instead of single-digit milliseconds
• Fully managed, automatically invalidated on writes
• Used for: Product inventory checks, shopping cart reads during checkout

Amazon Product Catalog Caching

• Product data rarely changes compared to how often it is read
• Cache Aside pattern: App checks Redis, on miss loads from database
• TTL: 24 hours for product description, 5 minutes for price (changes more often)
• Cache key: product_id + locale (same product has different data per country)
• Invalidation: When a seller updates product details, cache key is explicitly deleted

Session and Cart Caching

• Shopping cart stored in Redis (hash data structure)
• Cart persists across devices and sessions
• Redis persistence (AOF mode) ensures cart survives Redis restart
• TTL: 7 days for guest carts, 30 days for logged-in users
• On login: guest cart merged with saved cart (custom merge logic in application)

Search Result Caching

• Popular search queries (top 1% of queries = 80% of traffic) cached in Memcached
• TTL: 10-30 minutes for search results
• Cache key: normalized query string + filters + page number
• Popular flash deal search queries pre-warmed in cache before Prime Day opens

Recommendation Engine Caching

• Personalized recommendations computed by ML models in batch (every few hours)
• Stored in Redis as sorted sets (product_id -> score)
• Served from Redis directly on every page load
• TTL: 6-12 hours
• Fallback: Generic popular products if user has no personalization data

CDN Caching for Images and Static Assets

• All product images stored in S3, served via CloudFront CDN
• CloudFront caches images at 400+ edge locations globally
• Cache-Control headers set to 1 year for product images (images are immutable - new URL if image changes)
• When seller updates product image, new URL is generated (old URL still valid for CDN cache)

Twitter / X - Caching Strategy

Scale Context

• 400 million daily active users
• 500 million tweets per day
• Timeline generation is the most cache-intensive operation

Timeline Cache - The Core Problem

• Every user has a home timeline showing tweets from people they follow
• Recomputing this from scratch on every page load is too slow
• Twitter precomputes and caches timelines in Redis

Timeline in Redis Sorted Sets

• Each user has a Redis sorted set: timeline:{userId}
• Score: tweet timestamp (for chronological ordering)
• Member: tweet ID
• When User A tweets: Twitter fans out the tweet ID to Redis sorted sets of all followers
• When follower opens timeline: Read from Redis sorted set, fetch tweet content by ID
• Cache contains up to 800 most recent tweet IDs per user
• 800 tweets per user * 400 million users = 320 billion entries - managed across a Redis cluster

Multi-Layer Caching (L1, L2, L3)

L1: In-Process Cache (Caffeine/Guava) - Per JVM instance, microsecond access
    -> Hot tweet content, hot user profiles, frequently accessed objects
    -> Size: ~100 MB per instance
    -> TTL: 30 seconds (very short, very fresh)

L2: Memcached Cluster - Shared across all service instances
    -> Tweet content, user profile data, follower counts
    -> Size: Terabytes across cluster
    -> TTL: Minutes to hours

L3: Redis Cluster - Persistent, sorted sets for timelines
    -> Timeline data, rate limiting counters, trending topics
    -> Persistence enabled for timeline data

Trend Calculation Caching

• Twitter calculates trending topics using a stream processing pipeline
• Results stored in Redis with a short TTL (5 minutes)
• Trending data is regional (different trends for India, USA, UK)
• Cache key: trending:{region}:{category}
• Pre-computed globally and per-country, pre-populated in Redis every 5 minutes

Celebrity Problem and Cache Strategy

• Elon Musk has 180 million followers. Fan-out on write would update 180 million Redis entries.
• Twitter uses Hybrid Fan-Out:
  • Regular users (< 1 million followers): Fan-out on write (push tweet to follower timelines at write time)
  • Celebrities (> 1 million followers): Fan-out on read (pull tweets from celebrity at read time, merge with precomputed timeline)
• This prevents the Redis cluster from being overwhelmed when a celebrity tweets

Tweet Content Cache

• Tweet text and metadata stored in Manhattan (Twitter's own distributed database)
• Frequently accessed tweets cached in Memcached with LRU eviction
• Cache key: tweet:{tweetId}
• Hot tweets (viral ones) stay in cache due to frequent access
• Cold tweets (old, rarely viewed) are evicted by LRU

Write-Through for Counters

• Like counts, retweet counts, reply counts need to be fast to read and write
• Stored in Redis as atomic counters: INCR tweet:{tweetId}:likes
• Periodically flushed to persistent storage asynchronously
• Consequence: Counts may be slightly stale (eventual consistency accepted)

Google - Caching Strategy

Scale Context

• 8.5 billion searches per day
• 2 trillion searches per year
• Every search must return in under 100ms globally

Google's Bigtable as a Large-Scale Cache

• Bigtable is used as a massive cache layer for web content and search index
• Crawled web pages are cached in Bigtable before indexing
• Frequently accessed index shards are pre-loaded into memory on serving machines
• This is effectively a cache in front of the full index

Google Search Result Caching

• Popular search queries have results cached at the serving layer
• Top 10% of queries cover 80% of search traffic
• Cached results stored in distributed in-memory caches across serving nodes
• TTL varies: news-related queries (5-10 minutes), stable queries (hours)
• Cache key: normalized query + user language + safe-search setting
• Personalized results are NOT cached (different for every user)

Google Maps Tile Caching

• Map tiles at popular zoom levels and locations are heavily cached
• Three levels of tile caching:
  1. Client-side (browser/app): Tiles stored in local cache for offline use and faster re-viewing
  2. CDN Edge (Google's own CDN): Tiles at popular locations pre-cached globally
  3. Serving layer: Hot tiles in in-memory cache on tile serving servers
• Popular tiles (e.g., New York City at zoom level 14) cached indefinitely at CDN
• Less popular tiles (rural areas, high zoom) served fresh or cached with short TTL
• Tile invalidation: When map data changes (new road, new building), only affected tiles are purged

Google CDN Caching

• Google has its own global CDN infrastructure (not using Akamai or Cloudflare)
• Static assets (images, JS, CSS) cached at edge nodes close to users
• Cache-Control: immutable for versioned assets (changes in URL when content changes)
• Personalized content (Gmail messages, Drive files) never cached at CDN (only TLS terminated there)

YouTube Video Caching

• Popular videos cached at CDN edge nodes worldwide
• Netflix-style: "Nighttime pre-push" of trending content to edge nodes before peak hours
• Video chunks (not whole files) are cached - enables instant quality switching
• Adaptive bitrate: 720p chunks for popular content cached at more nodes than 4K chunks
• Cache eviction: LRU on storage at edge nodes, hot content stays, cold content evicted

Gmail Caching

• Email metadata (sender, subject, date, labels) cached in Bigtable with low TTL
• Email body fetched from GFS/Colossus on demand, not cached (too large, too infrequent per user)
• Thread list (which emails belong to same thread) cached aggressively (rarely changes)
• Attachment thumbnails cached at CDN (content-addressed - same attachment = same URL = cached once)

Netflix - Caching Strategy

Scale Context

• 300 million subscribers
• 250 million hours watched per day
• Personalization computed for every user, every time they open the app

EVCache - Netflix's Distributed Cache

• Netflix built EVCache (Ephemeral Volatile Cache) on top of Memcached
• Why not just use Memcached? EVCache adds:
  • Multi-region replication (cache data replicated across AWS regions)
  • Zone-aware reads (read from same AZ for lower latency)
  • Bulk operations
  • Observability hooks
• EVCache is the primary cache for Netflix API responses

Layered Cache Architecture

Netflix Client App
    -> Netflix API (Zuul Gateway)
        -> Microservice A -> EVCache -> Database
        -> Microservice B -> EVCache -> Database
        -> Microservice C (Recommendation) -> Compute Layer -> EVCache

Personalization Caching Strategy

• Netflix ML model computes personalized ranking for all titles for each user
• This is expensive: millions of users * thousands of titles = billions of scores
• Strategy: Batch computation nightly, store results in EVCache
• Cache key: user:{userId}:recommendations:{context} (context = home, search, similar)
• TTL: 6-24 hours (stale recommendations are better than slow recommendations)
• Cache warm-up: When user logs in, recommendations are pre-fetched in background

Video Manifest Caching

• When you press play on Netflix, your device requests a "manifest file" listing all available bitrates and chunk URLs
• Manifest files are different per title, device type, and subscription tier
• Cached in EVCache with TTL of a few minutes
• Invalidated when a new encode is available or DRM keys are rotated

Open Connect CDN - Netflix's Content Cache

• Netflix operates its own CDN (Open Connect Appliances - OCAs)
• OCAs are physical servers placed inside ISPs' data centers
• Netflix fills OCAs during off-peak hours (midnight) with content predicted to be popular next day
• How Netflix knows what to pre-cache: Collaborative filtering tells them what Region X will watch tomorrow
• Popular shows (Stranger Things, Squid Game release day) are pre-loaded to every OCA globally
• A Netflix stream never touches Netflix's main data center - all video is served from OCAs

Artwork and Image Caching

• Netflix shows different thumbnail images for the same title based on user preferences
• Image CDN cache must serve different images per user for the SAME movie
• Solution: Images are personalized at the edge with a small URL parameter (user segment ID)
• Images for the same segment cached at CDN; segments are broad enough to have high cache hit rate

Uber - Caching Strategy

Scale Context

• 150+ million monthly active users
• 25+ million trips per day
• Real-time location data for millions of drivers

Uber's Redis Usage for Location Data

• Driver location updated every 4 seconds
• Stored in Redis geospatial index: GEOADD drivers longitude latitude driverId
• Query for nearby drivers: GEORADIUS drivers longitude latitude 2 km
• Redis geospatial commands are O(N+log(M)) - fast for sparse data
• Location data is highly volatile - TTL of 30 seconds (driver offline = entry expires)

Trip State Caching

• Active trip state (trip_id, driver_id, rider_id, status, route) cached in Redis
• Strong consistency required for trip state (cannot lose or serve stale trip data)
• Redis used with replication: Write to master, reads from replica
• TTL: None (trip state removed explicitly when trip ends)

Surge Pricing Cache

• Surge multiplier for each geographic zone calculated every 1 minute
• Stored in Redis: surgeZone:{zoneId} -> multiplier
• All app servers read from Redis, never from database for this
• Why Redis not database? 100,000 app servers each reading surge for every trip request = database would die
• TTL: 90 seconds (refresh happens before expiry to avoid stale surge display)

Schemaless and Cassandra-Based Cache

• Uber's Schemaless (MySQL-backed distributed document store) acts as a cache for trip history
• Recent trip data (last 30 days) stored in Cassandra (fast reads)
• Older trips archived to slower storage
• Effectively tiered caching: hot data in Cassandra, cold data in archived storage

Notification Caching

• Push notification payloads cached to avoid recomputing for the same event
• Deduplication: Cache recent notification_id per user to prevent duplicate sends
• Redis set: sentNotifications:{userId} with SADD and 24-hour TTL

Banking (HDFC, ICICI, Axis, SBI Digital) - Caching Strategy

Scale Context

• SBI has 500 million account holders
• HDFC processes 1 billion transactions per month
• Internet banking peaks: Salary credit day (25th-31st of month) sees 10x normal traffic

What Banks Cache and What They Cannot

Banks have strict rules on what can be cached:

CAN be cached (low sensitivity, frequently accessed):

• Branch and ATM location data (static, rarely changes)
• IFSC codes and bank details (static reference data)
• Exchange rates (updated every 15-30 minutes, acceptable to show slightly stale)
• Interest rate tables for FD, loan products (updated by business, not real-time)
• UI configuration data (which features are enabled for which customer segment)
• Product offering metadata (loan products, credit card features)

CANNOT be cached or must be cached with extreme care (high sensitivity):

• Account balance (must always reflect latest transaction)
• Transaction status (pending vs completed vs failed)
• Available credit limit on credit card (changes with every swipe)
• OTP (one-time use, cannot be cached)

Read Replicas as an Implicit Cache

• Core banking databases have read replicas for all non-transactional reads
• Balance check API calls for internet banking hit read replica (may be 100ms stale)
• Actual fund transfer validates against primary database with locking
• This is not a cache per se but serves the same purpose

Redis for Rate Limiting and OTP Management

• OTP generation: SETEX otp:{mobile}:{purpose} 300 {otpValue}
  • TTL: 300 seconds (5 minutes)
  • Automatically expires after use window
• OTP attempt tracking: INCR otpAttempts:{mobile} with EXPIRE of 15 minutes
  • After 3 failed attempts, account locked
• Rate limiting login attempts: INCR loginAttempts:{userId} with EXPIRE
• All this runs on Redis with persistence disabled (OTPs must not survive Redis restart to prevent replay)

API Response Caching

• IFSC code lookup: Cached indefinitely (IFSC codes do not change)
• Exchange rate lookup: Cached for 15 minutes (TTL based, refreshed by batch job)
• Statement download: No caching (must always be fresh, regulatory requirement)
• Balance inquiry API: Read from database directly for regulatory compliance

CDN for Static Banking Content

• NetBanking login page, JavaScript, CSS: CDN cached with long TTL
• NEFT/RTGS timing, branch holidays: CDN cached for 24 hours
• Personal customer data (account number, name in page): NEVER served from CDN

Session Management

• Internet banking session stored in Redis cluster
• Session token (JWT-like) tied to Redis entry with remaining validity
• Session invalidated immediately on logout (Redis key deleted)
• Concurrent session control: Track all active sessions per customer, force logout of old ones
• TTL: 15-20 minutes of inactivity (banking regulatory requirement)

Zerodha - Caching Strategy

Scale Context

• 10 million active traders
• 1 million tick events per second during market hours
• Orders must be executed with pre-trade risk checks in under 10ms

Market Data Caching - The Most Critical Cache

In-Process Cache for Live Market Data

• Every application server maintains an in-process (JVM heap or C++ in-memory) copy of latest market prices
• Updated by a dedicated Market Data Service that consumes the NSE/BSE tick feed
• Published to all app servers via Redis Pub/Sub
• App servers maintain local copy to avoid inter-service calls during order placement
• This in-process cache is the fastest possible: nanosecond access for price checks during order validation

Tick Data in Redis for Recent History

• Last 60 seconds of ticks for each symbol stored in Redis sorted sets
• Used for: Real-time chart rendering, VWAP calculation, recent volatility checks
• Key: ticks:{symbol}:{date} as sorted set with timestamp as score
• TTL: 1 day (next day's data is all that matters)

Order Book State

• Open orders for each user cached in Redis hash: openOrders:{userId}
• Updated on every order placement, modification, and execution
• Used for: Displaying active orders, position calculation, margin check
• Strong consistency required: Write-through, Redis updated synchronously with DB write

Margin and Position Cache

• Available margin per user cached in Redis: margin:{userId} -> available balance
• Updated after every order execution with atomic Redis operations (DECRBY, INCRBY)
• Pre-trade risk check reads from this Redis key (not database) for speed
• Reconciled with actual database value every 5 minutes
• TTL: None (persistent for trading day, cleared at market close)

Instrument Master Cache

• List of all tradeable instruments (symbol, lot size, tick size, exchange) - 10,000+ instruments
• Loaded from database at market open (9:00 AM)
• Cached in-process for the entire trading day (static during market hours)
• No eviction, no TTL - refreshed only at next market open
• Used By: Every order validation to verify symbol exists and lot size is correct

Static Reference Data Cache

• ISIN to symbol mapping, corporate actions calendar, holidays, circuit limits
• Cached in Redis with 24-hour TTL
• Loaded from CDSL/NSDL, exchange data feeds
• Not performance-critical but reduces database load significantly

Universal Caching Principles Observed Across All Companies

Part C: Cache Topology, Sizing, and Resilience

C1 - The Five Cache Layers Explained

Most engineers think of Redis as "the cache". In reality, production systems use 4-5 distinct cache layers simultaneously. Each layer trades off speed, cost, consistency, and scope.

Request comes in from browser/mobile
        |
        v
[L1] Browser / Client-Side Cache     <- Fastest. Zero network cost. Per user.
        |
        v
[L2] CDN Edge Cache                  <- Geographic. Static/semi-static content.
        |
        v
[L3] API Gateway / Reverse Proxy     <- Per-endpoint response cache (nginx, Varnish)
        |
        v
[L4] Application In-Process Cache    <- In JVM heap. Per app instance. Nanoseconds.
        |
        v
[L5] Distributed Shared Cache        <- Redis / Memcached. Shared across all app instances.
        |
        v
[L6] Database Read Replicas          <- Implicit cache. Offloads primary DB.
        |
        v
[L7] Primary Database                <- Source of truth. Last resort.

Not every company uses all layers for every request. The decision is driven by:

• Consistency tolerance: How stale can this data be?
• Read:Write ratio: More reads = more layers justified
• Data size: Large data (images, video) goes to CDN. Small structured data goes to Redis.
• Personalization: Personalized data cannot go to shared CDN but can go to user-session Redis key.

C2 - Layer-by-Layer Breakdown

Layer 1: Browser / Client-Side Cache

What it is: HTTP caching built into browsers. Controlled by Cache-Control and ETag response headers your server sends.

What gets cached: CSS, JS, images, fonts, static API responses.

Headers used:

Cache-Control: public, max-age=31536000, immutable   <- Versioned assets (hash in filename)
Cache-Control: private, max-age=300                  <- User-specific data, 5 min
Cache-Control: no-store                              <- Never cache (bank balance page)
ETag: "abc123"                                       <- Conditional revalidation

Spring Boot setting the correct headers:

@GetMapping("/api/products/{id}")
public ResponseEntity<Product> getProduct(@PathVariable Long id) {
    Product product = productService.findById(id);
 
    return ResponseEntity.ok()
        .cacheControl(CacheControl.maxAge(5, TimeUnit.MINUTES).cachePublic())
        .eTag(String.valueOf(product.getVersion()))
        .body(product);
}

Who uses this: Every company. Amazon sets aggressive 1-year cache on versioned JS/CSS bundles. Flipkart, Swiggy do the same.

Layer 2: CDN Edge Cache

What it is: Servers placed at Points of Presence (PoPs) globally (Cloudflare has 310 cities, AWS CloudFront has 600+ edge locations). Your content is cached there, reducing latency from 200ms (origin fetch from Mumbai) to 5ms (served from Chennai PoP).

What gets cached: Images, videos, static files, public API responses that are NOT personalized.

What does NOT go to CDN: Anything with a user ID in it. Cart data, personalized recommendations, session tokens.

Key CDN concepts:

• Origin pull: CDN fetches from your server on first miss, caches it
• Cache key: Usually the URL + query params. Must normalize (sort query params) to avoid cache fragmentation.
• Purge/Invalidation: When you deploy new code, you push an invalidation request to CDN to clear old cached responses.
• Cache-Control from origin: CDN respects the Cache-Control header your server sends.

Who uses what CDN:

Layer 3: API Gateway / Reverse Proxy Cache

What it is: A cache at the entry point of your backend, before any application code runs. Nginx, Varnish, Kong, or AWS API Gateway can cache entire HTTP responses.

What gets cached: Public, non-personalized GET responses. Example: /api/categories, /api/trending-products, /api/cities.

Nginx response cache configuration (used by many Indian startups):

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:100m max_size=10g inactive=10m;
 
location /api/products/trending {
    proxy_cache api_cache;
    proxy_cache_valid 200 2m;                # Cache 200 responses for 2 minutes
    proxy_cache_key "$host$uri";
    proxy_cache_bypass $http_pragma;         # Allow forced refresh with Pragma: no-cache
    add_header X-Cache-Status $upstream_cache_status;  # HIT or MISS in response header
    proxy_pass http://backend;
}

When to use this layer: High-traffic read endpoints that return identical responses for all users. Category pages, featured banners, top search results.

When NOT to use: Any response containing user-specific data.

Layer 4: Application In-Process Cache (Local Cache)

What it is: A HashMap or cache library living inside your JVM process (inside each application server). No network hop. Access time is nanoseconds (< 1 microsecond).

Libraries used in Java/Spring Boot:

• Caffeine - Most popular in Spring Boot 2.x/3.x. High-performance, feature-rich.
• Ehcache - Older, still used in enterprise banking applications.
• Guava Cache - Simple, now largely superseded by Caffeine.
• Spring @Cacheable - Abstraction that works with any provider (Caffeine, Redis, Ehcache).

Spring Boot Caffeine configuration:

spring:
  cache:
    type: caffeine
    caffeine:
      spec: maximumSize=10000,expireAfterWrite=5m,recordStats

@Service
public class ProductCatalogService {
 
    // Result is cached in JVM heap of this application instance
    // No Redis call, no DB call on cache hit
    @Cacheable(value = "productDetails", key = "#productId")
    public ProductDetail getProductDetails(Long productId) {
        return productRepository.findById(productId).orElseThrow();
    }
 
    // Evict when product is updated
    @CacheEvict(value = "productDetails", key = "#product.id")
    public Product updateProduct(Product product) {
        return productRepository.save(product);
    }
}

The critical limitation: If you have 50 application server instances, each has its OWN local cache. A write on server-1 evicts from server-1's cache, but servers 2-50 still have stale data.

When this is acceptable: For data that changes rarely (product catalog, configuration, IFSC codes, PIN codes, exchange rates, master data).

When this causes bugs: For data that changes often or where consistency matters (user balance, order status, inventory count) - do NOT use local cache alone.

Who uses in-process cache heavily:

• Zerodha: NSE instrument master (300,000 instruments) loaded in-process at 9:00 AM. Doesn't change during market hours. Nanosecond access is critical for order routing.
• LinkedIn: Profile schema / field definitions cached in-process
• Google: Everything possible is pushed to in-process cache inside their servers

Layer 5: Distributed Shared Cache (Redis / Memcached)

What it is: An external cache server (or cluster) shared across ALL application instances. When server-1 writes to Redis, server-2 and server-50 see the updated value.

Redis vs Memcached - When to Choose Which:

Redis Data Structures and Real Use Cases:

Layer 6: Database Read Replicas

What it is: One or more replica copies of your primary database, updated continuously via replication log. Read queries are routed to replicas, offloading the primary.

Why it acts as a cache:

• Primary handles writes
• Replicas handle 80-90% of read traffic
• Replicas are typically in the same datacenter = low latency
• You get horizontal read scaling without a separate cache layer for some use cases

Replication Lag: Replicas are eventually consistent. Typical lag is milliseconds on a healthy setup. Under heavy write load, lag can grow to seconds.

When this is a problem: User just changed their password → reads from replica with 2s lag → still sees old password accepted → security bug.

Solution: Read-your-writes consistency: after a write, route the user's next read to primary for a short window (5s), then back to replica.

Spring Boot routing between primary and replica:

@Configuration
public class DataSourceConfig {
 
    @Bean
    @Primary
    public DataSource routingDataSource() {
        Map<Object, Object> dataSources = new HashMap<>();
        dataSources.put("primary", primaryDataSource());
        dataSources.put("replica", replicaDataSource());
 
        AbstractRoutingDataSource routing = new AbstractRoutingDataSource() {
            @Override
            protected Object determineCurrentLookupKey() {
                // Route read-only transactions to replica
                return TransactionSynchronizationManager.isCurrentTransactionReadOnly()
                    ? "replica" : "primary";
            }
        };
        routing.setTargetDataSources(dataSources);
        routing.setDefaultTargetDataSource(primaryDataSource());
        return routing;
    }
}
 
// Usage: @Transactional(readOnly = true) routes to replica automatically
@Transactional(readOnly = true)
public List<Product> getProductCatalog() {
    return productRepository.findAll(); // Goes to replica
}
 
@Transactional
public Order createOrder(OrderRequest request) {
    return orderRepository.save(new Order(request)); // Goes to primary
}

C3 - How Many Cache Servers? Sizing the Cache Tier

This is one of the most common system design interview questions. The answer follows a structured process.

Step 1: Calculate Working Set Size

"Working set" = the hot data that needs to be in cache (the 20% of data that gets 80% of reads).

Example: E-commerce platform with 10 million products

• All products: 10M × 5KB average = 50 GB
• Top 20% frequently accessed: 2M × 5KB = 10 GB working set
• Target cache hit rate: 95%
• Conclusion: 10-15 GB cache needed to achieve 95% hit rate

Example: Session store for 5 million concurrent users

• Session object: 1KB average
• 5M × 1KB = 5 GB raw session data
• Add 2x headroom for overhead: 10 GB

Step 2: Choose Eviction Policy

Redis eviction policy configuration:

maxmemory 10gb
maxmemory-policy allkeys-lru

Step 3: Calculate Number of Redis Instances

Single Redis node limitations:

• Max recommended memory per instance: 25 GB (beyond this, AOF rewrite and snapshot become slow)
• Single-threaded command processing (one CPU core for commands)
• Practical throughput: 100,000-500,000 commands/second per node

For 10 GB working set with high availability:

• 1 Redis Primary (10 GB) + 2 Redis Replicas = 3 nodes minimum
• Replicas are for read scaling and failover
• For 500,000+ operations/second: Redis Cluster with 3-6 primary shards + 3-6 replicas = 6-12 nodes

For very large caches (Netflix-scale, Facebook-scale):

• Netflix EVCache: hundreds of Memcached/Redis nodes across availability zones
• Facebook Memcached: tens of thousands of servers globally
• Rule: One cluster per availability zone for zone-local reads

Step 4: Sharding Strategy

When your data does not fit in one Redis instance:

Option A: Redis Cluster (built-in sharding)

• Automatically shards 16,384 hash slots across multiple primaries
• Each primary holds a subset of slots
• Clients must support cluster-aware routing
• Zero manual sharding logic

spring:
  data:
    redis:
      cluster:
        nodes:
          - redis-1:6379
          - redis-2:6379
          - redis-3:6379
        max-redirects: 3

Option B: Client-side consistent hashing

• Application decides which Redis node to use based on key hash
• Used when you have multiple Redis nodes but not Redis Cluster
• Consistent hashing minimizes cache misses when a node is added or removed (only 1/N keys need rehashing)

C4 - Keeping Cache Close to the Application

The Golden Rule: Network latency kills cache performance.

A Redis call across a datacenter takes 50-100ms. The same Redis call within the same datacenter takes 0.2-1ms. In-process cache takes 0.001ms.

Geographic Colocation Strategies

Strategy 1: Same Availability Zone (AZ) Deployment

Always deploy Redis in the same AZ as your application servers. Cross-AZ calls in AWS cost money AND add latency.

AZ us-east-1a:  App servers (10 nodes)  +  Redis primary
AZ us-east-1b:  App servers (10 nodes)  +  Redis replica (promotes on primary failure)
AZ us-east-1c:  App servers (10 nodes)  +  Redis replica

Netflix EVCache model: Each availability zone has its own Memcached cluster. App servers read from the local AZ cluster. Writes go to all AZ clusters. This means reads never cross AZ boundaries.

Strategy 2: Region-Local Cache

For global products, each region has its own independent cache. Data is either:

• Replicated to all regions (for global data like product catalog)
• Regional only (for user session, user-specific data in that region)

India Region (Mumbai):      App servers + Redis cluster
Singapore Region:           App servers + Redis cluster
US-East Region:             App servers + Redis cluster

Global data (product catalog): Replicated to all 3 regions
User session (user in India): Only in India Redis

Strategy 3: Sidecar Cache (Per-Pod Cache in Kubernetes)

In Kubernetes deployments, a Redis or Memcached container runs as a sidecar in the same Pod as the application container. Communication is over localhost (127.0.0.1). Zero network latency.

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: order-service
      image: order-service:1.0
      env:
        - name: REDIS_HOST
          value: "127.0.0.1" # Talks to sidecar on localhost
    - name: redis-sidecar
      image: redis:7.0
      resources:
        limits:
          memory: "512Mi" # Small local cache, not a full distributed cache

Used by: Uber for per-pod caching of configuration and feature flags.

Strategy 4: Read Replica Closest to App

For databases, deploy read replicas in the same datacenter as your application servers.

C5 - Cache Invalidation Strategies (The Hardest Problem)

"There are only two hard things in Computer Science: cache invalidation and naming things." — Phil Karlton

Strategy 1: TTL-Based Expiry (Simplest, Most Common)

Set a time-to-live. Cache serves stale data until TTL expires, then fetches fresh data.

Choosing TTL values:

Strategy 2: Event-Driven Invalidation (Precise, Complex)

When data changes, the writing service publishes an invalidation event. All caching layers subscribe and evict immediately.

// Product update triggers immediate cache eviction across all instances
@Service
public class ProductService {
 
    private final KafkaTemplate<String, String> kafkaTemplate;
    private final CacheManager cacheManager;
 
    @Transactional
    public Product updateProduct(Product product) {
        Product saved = productRepository.save(product);
 
        // Publish invalidation event to Kafka
        kafkaTemplate.send("cache.invalidation",
            "product:" + product.getId(),  // Key
            "EVICT"                        // Action
        );
 
        // Also evict local cache immediately on this instance
        cacheManager.getCache("products").evict(product.getId());
 
        return saved;
    }
}
 
// All application instances listen for invalidation events
@KafkaListener(topics = "cache.invalidation", groupId = "cache-invalidation-group")
public void handleCacheInvalidation(String cacheKey) {
    String[] parts = cacheKey.split(":");
    String cacheName = parts[0] + "s";  // "product" -> "products"
    String id = parts[1];
 
    cacheManager.getCache(cacheName).evict(id);
    log.debug("Evicted cache key: {}", cacheKey);
}

Used by: Flipkart (product updates during flash sales), Amazon (price changes propagate instantly), LinkedIn (profile changes).

Strategy 3: Write-Through (Always Consistent, Write Cost)

On every write to the database, also write to cache. Cache is always up-to-date.

Problem: Write latency increases. Every write is now two operations.
Benefit: Cache hit rate is very high since every written item is immediately in cache.

@Transactional
public User updateUserProfile(User user) {
    User saved = userRepository.save(user);
    // Write to cache immediately - cache always current
    redisTemplate.opsForHash().putAll("user:" + user.getId(), toMap(saved));
    return saved;
}

Used by: Twitter (timeline writes go to both DB and Redis Sorted Set simultaneously).

Strategy 4: Cache Stampede Prevention

When a hot cache key expires, thousands of requests simultaneously go to the database. The database gets overwhelmed. This is the "thundering herd" or "cache stampede" problem.

Solution A: Mutex Lock

public Product getProduct(Long id) {
    String cacheKey = "product:" + id;
    Product cached = redisTemplate.opsForValue().get(cacheKey);
    if (cached != null) return cached;
 
    // Only one thread gets the lock, others wait
    String lockKey = "lock:" + cacheKey;
    Boolean locked = redisTemplate.opsForValue()
        .setIfAbsent(lockKey, "1", 5, TimeUnit.SECONDS);  // SETNX with 5s TTL
 
    if (Boolean.TRUE.equals(locked)) {
        try {
            Product product = productRepository.findById(id).orElseThrow();
            redisTemplate.opsForValue().set(cacheKey, product, 10, TimeUnit.MINUTES);
            return product;
        } finally {
            redisTemplate.delete(lockKey);
        }
    } else {
        // Another thread is refreshing. Wait briefly and retry from cache.
        Thread.sleep(50);
        return redisTemplate.opsForValue().get(cacheKey);
    }
}

Solution B: Probabilistic Early Expiry (XFetch algorithm)

Instead of waiting for TTL to hit 0, proactively refresh the cache before it expires with increasing probability as expiry approaches. No lock needed.

public Product getProduct(Long id, double beta) {
    CacheEntry<Product> entry = redisTemplate.opsForValue().get("product:" + id);
 
    double remainingTtl = entry.getRemainingTtlSeconds();
    double delta = System.currentTimeMillis() - entry.getLastFetchTimeMs();
 
    // Probabilistic early refresh: more likely to refresh as TTL decreases
    if (-delta * beta * Math.log(Math.random()) >= remainingTtl) {
        // Refresh now, before expiry, to prevent stampede
        Product fresh = productRepository.findById(id).orElseThrow();
        redisTemplate.opsForValue().set("product:" + id,
            new CacheEntry<>(fresh, System.currentTimeMillis()),
            10, TimeUnit.MINUTES);
        return fresh;
    }
 
    return entry.getValue();
}

C6 - What to Do When Redis Goes Down

This is a critical resilience design question. Redis going down is not "if", it is "when". Your system must survive it.

Scenario 1: Redis Single Node Goes Down

Symptom: 100% cache misses. All reads fall through to database. DB CPU spikes to 100%. Application latency goes from 5ms to 500ms. Users see slowness.

Mitigation Strategies:

A. Redis Sentinel (High Availability - automatic failover)

Redis Sentinel Setup:
    Primary Redis (read + write)
    Replica Redis 1 (standby)
    Replica Redis 2 (standby)
    3 Sentinel processes (monitor primary, vote on failure, elect new primary)

Failover time: 30-60 seconds
Data loss: A few seconds of writes that were not replicated before failure
Spring Boot config:

spring:
  data:
    redis:
      sentinel:
        master: mymaster
        nodes:
          - sentinel-1:26379
          - sentinel-2:26379
          - sentinel-3:26379

B. Redis Cluster (High Availability + Sharding)

6 nodes (3 primary + 3 replica)
Each primary shard holds 1/3 of the data
If one primary fails, its replica is promoted automatically
Failover time: 1-15 seconds
Application continues with 2/3 of the cache (the failed shard's slot requests miss until promoted)

C. Circuit Breaker Pattern for Cache

@Service
public class CachedProductService {
 
    private final CircuitBreaker cacheCircuitBreaker;
    private final ProductRepository productRepository;
    private final RedisTemplate<String, Product> redisTemplate;
 
    public Product getProduct(Long id) {
        // If Redis is down, circuit opens, bypass cache entirely
        return cacheCircuitBreaker.executeSupplier(() -> {
            Product cached = redisTemplate.opsForValue().get("product:" + id);
            if (cached != null) return cached;
 
            Product product = productRepository.findById(id).orElseThrow();
            redisTemplate.opsForValue().set("product:" + id, product, 10, TimeUnit.MINUTES);
            return product;
        });
    }
}

When circuit is open (Redis is down), the fallback automatically goes to the database. Application stays functional, just slower.

Using Resilience4j circuit breaker:

resilience4j:
  circuitbreaker:
    instances:
      cacheCircuitBreaker:
        slidingWindowSize: 10
        failureRateThreshold: 50 # Open circuit if 50% of last 10 calls fail
        waitDurationInOpenState: 30s # Wait 30s before trying Redis again
        permittedNumberOfCallsInHalfOpenState: 3

Scenario 2: Redis Memory Full (OOM)

Symptom: Redis returns OOM command not allowed when used memory > maxmemory error. New writes fail.

Solutions:

1. Eviction policy (set maxmemory-policy allkeys-lru) - Redis automatically evicts least-recently-used keys instead of returning error.
2. Memory alerting - Alert when Redis memory exceeds 75% capacity. Gives time to scale before hitting 100%.
3. maxmemory configuration - Always set maxmemory to leave OS headroom (never let Redis use 100% of system RAM — risk of OOM kill).

# Redis maxmemory best practice
# For 16GB server, set maxmemory to 12GB (leave 4GB for OS and Redis overhead)
maxmemory 12gb
maxmemory-policy allkeys-lru

Scenario 3: Cache Corruption / Stale Data Bug

Symptom: Users see wrong prices, wrong inventory counts, wrong profile data due to a cache bug that stored wrong data.

Solutions:

1. Cache key versioning: Include a version prefix in all cache keys. When you detect corruption, bump the version prefix. All old keys become "miss" and are naturally refilled.

private static final String CACHE_VERSION = "v3:";  // Bump to v4 to invalidate all old keys
 
public String buildCacheKey(Long id) {
    return CACHE_VERSION + "product:" + id;
}

2. FLUSHDB with caution: Clears the entire Redis database. Effective but causes a thundering herd - all requests hit the database simultaneously. Only do this during low-traffic period.
3. Selective eviction by pattern: redis-cli --scan --pattern "product:*" | xargs redis-cli del

Scenario 4: Network Partition Between App and Redis

Symptom: Redis is alive, but app servers cannot reach it. Looks like Redis is down from the app's perspective.

Solutions:

1. Timeout configuration - Set aggressive connect and read timeouts. Do not wait 30s for a response; fail fast.

spring:
  data:
    redis:
      timeout: 500ms # Command timeout - fail if Redis takes > 500ms
      connect-timeout: 1s # Connection establishment timeout
      lettuce:
        pool:
          max-active: 50
          max-wait: 200ms # Do not wait more than 200ms for a connection from pool

2. Fallback to local cache - If Redis is unreachable, serve from local in-process Caffeine cache for the duration of the partition.
3. Graceful degradation - If neither cache level works, serve from DB with a rate limit (protect DB from full traffic during cache miss storm).

C7 - Cache Strategy Selection Guide: Which Strategy for Which Data

This is the summary companies use when designing their caching layer for new features.

C8 - Real Company Cache Topology Summary

How Zerodha Caches (Trading Platform - Latency-Sensitive)

[NSE Feed] -> In-process (Java ConcurrentHashMap, 200 nanoseconds) <- Market data
[User requests] -> In-process Caffeine (instrument master) <- Static all-day
               -> Redis Cluster (positions, margins - DECRBY/INCRBY) <- Per-trade update
               -> PostgreSQL Read Replica (historical queries)
               -> PostgreSQL Primary (order writes, settlement writes)

Redis Config: 3 primary shards + 3 replicas, same datacenter as app (NSE co-location Mumbai)
Failover: Redis Sentinel, 30s automatic failover
Redis Down: Circuit breaker opens, falls back to direct DB read, alert fires

How Flipkart Caches (Flash Sales - Throughput Focused)

[User request]
    -> CloudFront CDN (static assets, category pages)
    -> Nginx API Gateway cache (trending, banners - 2min TTL)
    -> Caffeine L1 (product catalog, 5min TTL)
    -> Redis Cluster L2 (inventory counts, session, cart)
    -> Aurora Read Replica (long-tail product queries)
    -> Aurora Primary (order writes)

For Big Billion Day:
    Redis pre-warmed 2 hours before sale start
    Inventory counts in Redis (Redis DECRBY, atomic)
    Oversell protection: Lua script (check count, decrement if > 0, atomic)
    Redis Cluster: 6 shards, 12 nodes total

How HDFC/ICICI Bank Caches (Regulated - Safety First)

[User request]
    -> Akamai CDN (static assets, FAQ pages only)
    -> Nginx (no response caching for logged-in users)
    -> Caffeine L1 (branch master, IFSC data, interest rate tables - 24h TTL)
    -> Redis (OTP with SETEX 300s, session with 15min TTL, rate limiting)
    -> Oracle Read Replica (statement history, product info)
    -> Oracle Primary (balance, transaction writes - always real-time)

Balance: NEVER cached. Every balance query hits Oracle Primary.
OTP: Redis with SETEX. Attempt count tracked with INCR. Block after 3 failures.
Redis HA: Active-Passive with Redis Sentinel. RTO = 30s. RPO = near zero (AOF enabled).
Redis Down: Session service falls back to DB-backed sessions. OTP service goes down (alerts fire, incident declared).

How Netflix Caches (Global Scale - Availability First)

[User request]
    -> Open Connect CDN at ISP (video chunks - primary delivery, zero latency)
    -> Browser Cache (manifests, up to 1min)
    -> API Gateway (public non-personalized responses)
    -> EVCache (Memcached multi-AZ, personalization, catalog - 6-24h TTL)
    -> Cassandra (metadata primary store)

EVCache Key Property:
    Every AZ has its own EVCache cluster
    Writes replicate to ALL AZ clusters asynchronously
    Reads ALWAYS from local AZ cluster (never cross-AZ)
    If local AZ cluster is down: Read from next AZ, alert fires

Number of EVCache nodes: Hundreds per region, thousands globally