Caching
Serve repeated reads from memory — cache layers, eviction, expiry and the hard problem of invalidation.
13 min read
You will learn
- Explain where caches sit (browser, CDN, server, database)
- Use the cache-aside pattern
- Explain TTLs, eviction policies like LRU, and stale data
- Implement an LRU cache
The fastest request is the one you don't make
Reading a product from a database might take 5–50 ms; reading it from memory takes microseconds. When the same data is requested again and again — product pages, configuration, exchange rates, user profiles — a cache stores a copy close to where it's needed.
Caching is the single most effective performance technique in backend systems. It's also a classic source of bugs.
Cache layers
Caches exist at every level between the user and the database:
| Layer | Example | Controlled by |
|---|---|---|
| Browser | Images, CSS, API responses | Cache-Control, ETag headers |
| CDN / edge | Static files and public pages served near the user | Headers + CDN rules |
| Reverse proxy / output cache | Whole HTTP responses | Server config, ASP.NET Core output caching |
| Application (in-memory) | IMemoryCache inside one server | Your code |
| Distributed cache | Redis, shared by all server instances | Your code |
| Database | Buffer pool, query plan cache | The database |
An in-memory cache is fastest but each server has its own copy (and loses it on restart). A distributed cache like Redis is shared by all instances — consistent across servers, at the cost of a network hop.
Cache-aside
The most common pattern: the application checks the cache first, and fills it on a miss.
async Task<Product?> GetProductAsync(int id)
{
string key = $"product:{id}";
if (cache.TryGetValue(key, out Product? cached))
return cached; // hit
var product = await db.Products.FindAsync(id); // miss → source of truth
if (product is not null)
cache.Set(key, product, TimeSpan.FromMinutes(5)); // populate with a TTL
return product;
}Expiry, eviction and staleness
- TTL (time to live) — how long an entry stays valid. Bounds how stale data can be.
- Eviction — when the cache is full, something must go. LRU (least recently used) is the classic policy: drop what hasn't been touched for longest.
- Stale data — the source changed but the cache still serves the old copy.
Experiment with all three — and see what happens to correctness when data changes:
Caching
20 products, popular ones requested far more often · database 80 ms · cache 2 ms
Hit rate
0%
Avg response time
0 ms
Database queries
0 / 0
Stale reads
0
Recent requests
- Send some requests…
Cache contents (0/5) — most recently used first
Database: product #1 is at version v1.
Invalidation strategies
When data changes, how do caches find out?
- Expire with TTLs — simple; accept staleness up to the TTL.
- Invalidate on write — delete (or update) the cache entry whenever you change the data. Precise, but every code path that writes must remember to do it.
- Versioned keys — include a version in the key (
product:42:v7) so new data uses new keys. - Events — publish "product 42 changed" so every cache can evict it.
Cache stampede
When a hot entry expires, hundreds of concurrent requests may all miss and hit the database at once. Mitigations: let only one request rebuild the entry (a lock or HybridCache in .NET 9), add random jitter to TTLs, or refresh entries before they expire.
What (not) to cache
- ✅ Data read far more often than it changes; expensive computations; responses from slow external services.
- ⚠️ Per-user data (watch memory, never leak it to other users — include the user id in the key).
- ❌ Data that must always be exactly current (account balances during a payment), or that's cheap to fetch anyway.
Key takeaways
- Caches exist at every layer, from browser to database; in-memory is fastest, distributed (Redis) is shared.
- Cache-aside: check the cache, fall back to the source, populate with a TTL.
- TTLs bound staleness; eviction policies like LRU keep the cache within its size.
- Invalidation is the hard part — choose TTLs, invalidate-on-write or events deliberately.
Exercise
Implement a least-recently-used cache. Set adds or updates a value; TryGet returns it and marks it as recently used. When adding a NEW key to a full cache, evict the least recently used entry. Count is the number of entries.
C# runtime not loaded
Hint 1
Track usage order with a LinkedList<(TKey Key, TValue Value)>: the most recently used entry at the front, the least at the back.
Hint 2
Store Dictionary<TKey, LinkedListNode<(TKey Key, TValue Value)>> so you can find a node in O(1), then list.Remove(node) and list.AddFirst(node) to mark it as used.
Hint 3
On Set of a new key when Count == capacity: remove list.Last from both the list and the dictionary.
Show solution
var cache = new LruCache<string, int>(2);
cache.Set("a", 1);
cache.Set("b", 2);
cache.TryGet("a", out _); // "a" is now the most recently used
cache.Set("c", 3); // should evict "b"
Console.WriteLine($"a: {cache.TryGet("a", out _)}, b: {cache.TryGet("b", out _)}, c: {cache.TryGet("c", out _)}");
class LruCache<TKey, TValue> where TKey : notnull
{
private readonly int capacity;
private readonly Dictionary<TKey, LinkedListNode<(TKey Key, TValue Value)>> map = new();
private readonly LinkedList<(TKey Key, TValue Value)> order = new(); // front = most recently used
public LruCache(int capacity) => this.capacity = capacity;
public int Count => map.Count;
public bool TryGet(TKey key, out TValue value)
{
if (map.TryGetValue(key, out var node))
{
order.Remove(node);
order.AddFirst(node);
value = node.Value.Value;
return true;
}
value = default!;
return false;
}
public void Set(TKey key, TValue value)
{
if (map.TryGetValue(key, out var existing))
{
order.Remove(existing);
}
else if (map.Count == capacity)
{
var lru = order.Last!;
order.RemoveLast();
map.Remove(lru.Value.Key);
}
map[key] = order.AddFirst((key, value));
}
}Quiz
4 questions · 3 correct to pass1.In the cache-aside pattern, what happens on a cache miss?
2.What is a TTL in caching?
3.Which eviction policy removes the entry that hasn't been used for the longest time when the cache is full?
4.Why is cache invalidation hard?
To complete this lesson
- ○ Solve the exercise
- ○ Pass the quiz (3/4 correct)