Building a Local Dataset Cache: Hot Data in RAM, Cold Data on Disk
Exploring a hot/cold architecture where RAM acts as a cache, disk stores the full dataset, and performance behaves very differently than expected.
Source code: Here.
It's an experiment in building a local dataset cache where RAM acts as a hot cache and disk remains the source of truth.
Why Build This?
Memory is one of the most expensive resources in a small system.
Large companies can often solve storage problems by adding more machines, deploying another cache cluster, or scaling their infrastructure horizontally. Most systems don’t have that luxury.
A typical service might run on a single machine with a limited amount of RAM while its dataset continues to grow over time.
The obvious solution is to keep everything in memory.
The problem is that most data is rarely accessed.
A small percentage of records receives the majority of reads, while the rest simply occupies memory waiting for requests that may never arrive.
That observation led me to a simple question:
Do we really need the entire dataset in RAM?
Instead of treating memory as the source of truth, what if RAM only stored the hot working set while cold data lived on disk?
The architecture sounded straightforward.
What I didn’t expect was that the hardest part wouldn’t be storage at all.
Rethinking Capacity
Most in-memory caches share a simple assumption:
The dataset must fit into memory.
That assumption provides excellent performance, but it also means capacity is directly tied to available RAM.
I wanted to explore a different model.
Instead of storing all values in memory:
RAM stores indexes and frequently accessed values.
Disk stores the complete dataset.
Memory acts as a cache rather than the source of truth.
At a high level, capacity is no longer limited by memory.
The practical limit becomes disk space.
Of course, this comes with trade-offs.
Cold reads are slower because they require disk access, and the index itself still consumes memory. A machine cannot store an infinite number of keys.
The design only makes sense when access patterns are skewed enough that a relatively small working set receives most requests.
For workloads with strong locality, this can significantly reduce memory requirements without sacrificing access speed for hot data.
.
Shared-Nothing Shards
Once RAM became a cache layer rather than the entire store, the next challenge was concurrency.
I wanted to avoid locks in the hot path.
Rather than sharing a global HashMap across threads, the store is divided into shards.
Each shard owns its data.
A key is hashed and routed to a specific worker.
That worker is the only component allowed to modify its shard.
No mutexes.
No shared writes.
No coordination around the data itself.
Conceptually, the system becomes a collection of small independent stores running in parallel.
Each shard manages:
An in-memory cache.
An index.
Persistent storage files.
The architecture is heavily inspired by shared-nothing systems where ownership replaces synchronization.
At least in theory.
Request Multiplexing
The next problem was request ordering.
Traditional request-response systems often process replies in the same order requests arrive.
That sounds simple, but it creates a form of head-of-line blocking.
One slow operation can delay every response behind it.
Instead, each request carries a request identifier.
The server echoes that identifier back in the response.
Responses can be returned in any order.
The client matches responses using the request ID.
This shifts responsibility for ordering away from the server.
The server no longer needs to maintain large ordering structures or wait for slower requests before returning faster ones.
If this sounds familiar, it follows a similar idea to the transition from HTTP/1.1 pipelining to HTTP/2 multiplexing.
Write-Through Persistence and Lossless Tiering
The cache is not the source of truth.
Disk is.
Whenever a value is written:
The value is persisted to disk.
The value is inserted into the in-memory cache.
This means eviction becomes extremely simple.
Removing a value from memory does not remove the data itself.
The value already exists on disk.
The cache entry disappears.
The underlying record remains.
As a result, memory usage can stay bounded while the full dataset remains available.
The cache only contains the working set.
Everything else lives on disk until it becomes relevant again.
Where This Architecture Fits
This project is not intended to replace mature storage engines or distributed cache clusters.
It targets a much smaller space.
A single machine.
Limited RAM.
A dataset larger than memory.
Simple operational requirements.
The architecture works best when:
Hot data represents a relatively small portion of the dataset.
Capacity matters more than absolute latency.
Running additional infrastructure would be overkill.
It also has obvious limitations.
There is no perfect compression of memory usage.
Indexes still consume RAM.
Cold reads still touch disk.
And many of the scaling benefits of sharding only become meaningful when operations are expensive enough to justify the coordination cost.
Final Thoughts
I started this project thinking the difficult part would be storage.
It wasn’t.
The difficult part was everything around storage.
The cache.
The scheduler.
The benchmark harness.
The communication between tasks.
The assumptions I carried into the design.
The biggest lesson wasn’t about caches, disks, or HashMaps.
It was that architectures that look obviously faster on paper often tell you very little about real-world performance.
The expensive part is usually hidden somewhere else.
Measure first.
Then measure the thing you used to measure with.
If you'd like to explore the implementation, benchmarks, or architecture diagrams, the project is available here: My repo.
Author’s Note
This is my first technical write-up.
The goal of this article is not to present a production-ready storage engine or prove that this architecture is superior to existing solutions.
Instead, I wanted to document the reasoning behind the design, the trade-offs I encountered, and the lessons learned while building it.
Some conclusions are based on experiments performed on a single machine and should be treated as observations rather than universal truths.
If you spot mistakes, alternative approaches, or interesting ideas, I’d love to hear them.