Learn
Your cache holds ten thousand entries, and nobody asked how many bytes that is
You have put an in-memory cache in front of the product service, and you say it keeps the 10,000 most recently used entries. The interviewer nods and asks one more thing: how much memory is that? If the honest answer is "it depends on the entry", the limit you just gave does not bound anything.
The requirement
Each instance of the product service runs with a 2 GB heap. Product pages read a catalogue entry: name, price, and the list of variants. Most products have a handful of variants and serialise to about 4 KB. A few, such as a sofa sold in every fabric and size, have 2,000 variants at about 2 KB each, which makes one entry about 4 MB.
The requirement is not "cache 10,000 products". It is "keep the hit rate high and never let the cache take the process down". Say that second half out loud, because it is the one a count cannot promise.
What goes wrong, with the input that shows it
At 4 KB an entry, 10,000 entries take about 40 MB. At 4 MB an entry, 10,000 entries would take about 40 GB. The same limit covers a thousandfold range.
Now run the input that hurts. A sale puts the 500 largest products on the home page, and within the hour every one of them is read. All 500 fit comfortably under a limit of 10,000, so all 500 stay in the cache: 500 × 4 MB is about 2 GB, the whole heap. The service does not get slower first. It runs out of memory, restarts with a cold cache, and the next wave of reads fills it the same way.
Two designs and what each one costs
Option A: keep a count, but size it from the largest entry. If the cache may use 512 MB and the largest entry is 4 MB, the safe count is 128.
- What you get: the simplest cache there is, and a memory ceiling you can state.
- What you pay: 128 entries for a catalogue of many thousands, so most reads miss. Sizing the count from the typical entry instead brings the hit rate back and the out-of-memory risk with it.
Option B: bound the cache in bytes, and refuse entries that are too large. The cache keeps a running total of the bytes it holds and evicts the least recently used entries until the total is under budget. Anything above a per-entry cap is served without being cached.
from collections import OrderedDict
class ByteBoundedLRU:
def __init__(self, max_bytes, max_entry_bytes):
self.max_bytes, self.max_entry_bytes = max_bytes, max_entry_bytes
self.used, self.items = 0, OrderedDict() # key -> serialised bytes
def get(self, key):
if key not in self.items:
return None
self.items.move_to_end(key)
return self.items[key]
def put(self, key, value: bytes):
if len(value) > self.max_entry_bytes:
return # too large to cache: read it from the database each time
if key in self.items:
self.used -= len(self.items.pop(key))
self.items[key] = value
self.used += len(value)
while self.used > self.max_bytes:
_, old = self.items.popitem(last=False)
self.used -= len(old)
Trace it with max_bytes=10 and max_entry_bytes=6. Put a (4 bytes) and b (4 bytes): 8 used. Put c (7 bytes): refused. Put d (5 bytes): 13 used, so a is evicted and 9 remain. The cache stores serialised bytes on purpose, so len is the size, give or take object overhead.
With a 512 MB budget and a 1 MB cap, the cache holds about 128,000 typical entries and never more than 512 MB, whatever is on sale.
- What you get: a memory ceiling and a high hit rate at the same time.
- What you pay: every entry must be measured, which costs a serialisation or an estimate on each write. The largest products are never cached, so their reads always reach the database. One 1 MB entry pushes out up to 250 small ones.
What to say out loud
Ask first: "How far apart are the smallest and the largest entries, and what happens to the process if the cache fills with the largest?" Then commit: "If every entry is within a factor of two of the others, a count times the largest size is an honest budget, and I keep the count. If they span a thousandfold, I bound the cache in bytes, cap the size of one entry, and accept that the biggest products are always read from the database."
The exercise
- Pick a cache you have built or used, and write down its limit as it is configured today.
- Estimate the size of its typical entry and of its largest one, in bytes.
- Multiply the limit by the largest size, and compare the result with the memory of the process.
- Write the input that fills it with the largest entries: a sale, a bulk import, one tenant with a huge account.
- Say the deciding question aloud, then your choice for each answer to it, in under a minute.
Where this fits in Gadfly
System design is one of Gadfly's four rooms, alongside algorithms, behavioral and code review. The design is drawn on a shared board, guided or as a mock, and the session takes one trade-off at a time. In guided mode the tutor asks the next question, such as how large the largest entry is, and does not hand over the design. In a mock it behaves like an interviewer: no coaching during the round, then a debrief that quotes what you said rather than giving a grade.
Published 18 November 2026 · System design: one trade-off · all notes · feed
This note was drafted by Gadfly's scout — an automated writing agent run by Dmytro Hryshchenko — and checked by rules against the product's claims registry, not read by a person before it went live. The exercise is original. Spotted a mistake? [email protected].