Workshop · floor model for Survey of Software 1.047

cache me if you can

A cache hit in the standard library costs the same as a dict lookup. The cache most people install costs twenty times that — and neither number decides anything, because a cache is paid for by its misses. Below: what a hit and a miss cost on each tier, which eviction policy earns the hit rate on the access pattern you actually have, and the hit rate under which the cache makes your code slower. Measured in this tab.

Everything runs here. Press start, then change the access pattern, the cache size, and the cost of the function you are wrapping.

Plain language. Switch to Engineering for versions, byte counts and the measurements behind each claim.

The whole answer, first

Three tiers of cache, one question each. The numbers were measured for the survey on one machine; the panels below re-measure them on yours.

CPython 3.14.7, aarch64, single thread, best-of under a budget; one hit on the cheapest possible wrapped function, so the overhead is the whole number. The method and the full tables are in S2 §0 of the survey; the bench imports this page's core.py.

What a hit costs — and what a miss costs

Each cache wraps the cheapest function there is, k + 1, so the bar is the cache's own overhead. Left: a hit. Right: storing a new value, which is what every miss has to do on top of running the function.

1,000 keys, warmed, then 1,000 calls per batch, best-of under a budget (the browser clock clamps to 100 µs, so batches are calibrated to ≥ 5 ms first). Log scale. diskcache is SQLite on this tab's in-memory filesystem — its read cost is representative, its write cost is not, because nothing here is fsynced; the disk figure is in orange.

Why the standard library wins by twenty times: lru_cache is C — a hash lookup and a pointer move. cachetools is Python — a decorator, a key built from your arguments, a method call into a cache class, and for TTLCache a clock read and an expiry sweep. That is what you buy: a time-to-live, a size measured however you like, caches you can inspect and clear, five eviction policies. It costs a microsecond. A database query costs a thousand of those; an HTTP call ten thousand. If the function you are wrapping costs less than about ten microseconds, the answer is not to cache it.

Which policy earns the hit rate

The same requests replayed through each of cachetools' eviction policies, plus the standard library's lru_cache and the best any policy could possibly do if it could see the future. Pick the shape of your traffic and how much of it fits.

30,000 requests over 10,000 keys, deterministic (seed 7), so the numbers are identical here and in the survey's native run. Each policy is cachetools' own class — LRUCache, LFUCache, FIFOCache, RRCache — replayed with __getitem__ so the order is updated on a hit. The ceiling is Belady's offline MIN. "Hot set" is Zipf with s = 1 over a shuffled key space.

What the patterns say. On a stable hot set, every sensible policy lands within a few points of each other and a cache holding one key in a hundred already catches two requests in five. When the favourites move, LRU follows them and LFU does not — it keeps counting yesterday's hits. On a loop that does not fit, LRU and FIFO evict each key exactly one step before it is needed again, every time, at every size up to half; the only policies that score are the ones that refuse to follow the loop. With no favourites at all, the hit rate is the cache size and no policy can change it. Measure your pattern before choosing a policy — and if you cannot, LRU is the default for the right reason: it is the one that recovers.

Does it pay?

Take the hit rate from the pattern and size you chose above, the hit and store costs measured on this machine, and the cost of the function you want to wrap. The table says what you would actually save — and the hit rate under which each tier makes your code slower.

mean time with cache = h·hit + (1 − h)·(fn + store); pays when h > store ÷ (fn + store − hit). The network rows use the survey's figures (orange): a unix socket is the tier's best case — no network — and this machine's TCP loopback its second-best.

The miss is the price. A hit on diskcache costs a few microseconds; the write that follows a miss costs hundreds on a real disk. So for a function that takes 100 µs, diskcache needs more than eighty hits in a hundred just to break even — and for one that takes 10 ms, four in a hundred. Caching pays for slow functions and punishes fast ones, and the line between them is a number you can now read off.

What could not be measured here

Start your build

Take what you just measured and hand it to whichever AI you use — the costs on your machine, the pattern and hit rate you chose, and a brief for what to do next.

Clipboard envelope — no vendor deep link, no API key, no dependency on anyone's roadmap.

Open the lab

The measurement is one file, core.py, fetched by this page and imported by the survey's bench. uvx marimo edit a notebook that imports it, or read it — it is short.