
Where does the write go? To the cache, to the backing store, or both?

This question defines your caching strategy. Get it wrong and you lose data or tank performance. Get it right and you unlock the same trick that makes [LSM trees](/post/btrees-vs-lsm) and NVMe drives fast.

### Write-through

Every write goes to both the cache and the backing store. The write isn't acknowledged until both complete.

```
App → Cache → Backing Store
         ↓
      (wait for both)
```

Simple and safe. If the cache disappears, your data is already persisted. The downside: you're bottlenecked by the slower backing store on every write.

### Write-back

Writes go to the cache only. The backing store gets updated later, asynchronously.

```
App → Cache → (ack immediately)
         ↓
      (flush later)
         ↓
   Backing Store
```

Your writes are fast because you're only hitting the cache. The backing store gets batched updates when convenient.

The risk: if the cache dies before flushing, you lose data. For important data, you need replication or a write-ahead log to recover.

This is how LSM trees work. The memtable is the cache. Disk is the backing store. Writes hit memory, get acknowledged, and flush later as SSTables.

### Write-around

Writes bypass the cache entirely and go straight to the backing store. The cache only gets populated on reads.

```
App → Backing Store (writes)
App → Cache → Backing Store (reads, on miss)
```

Useful when you're writing data you don't expect to read soon. Avoids polluting the cache with write-once data.

### Read-through (pull-through)

On a cache miss, the cache fetches from the backing store, stores it, then returns it. The application doesn't talk to the backing store directly.

```
App → Cache (miss) → Backing Store
         ↓
   (cache stores it)
         ↓
   (return to app)
```

The cache is transparent. Your app just asks for data; the cache handles the rest.

### Cache-aside

The application manages everything. On a miss, the app fetches from the backing store and populates the cache itself.

```
App → Cache (miss)
App → Backing Store (app fetches directly)
App → Cache (app stores it)
```

More control, more code. You decide what gets cached and when.

### The trade-off matrix

| Strategy      | Write latency | Read latency | Durability        | Complexity |
| ------------- | ------------- | ------------ | ----------------- | ---------- |
| Write-through | Slow          | Fast (hit)   | High              | Low        |
| Write-back    | Fast          | Fast (hit)   | Needs replication | Medium     |
| Write-around  | Slow          | Fast (hit)   | High              | Low        |
| Read-through  | N/A           | Fast (hit)   | N/A               | Low        |
| Cache-aside   | N/A           | Fast (hit)   | N/A               | High       |

### Where you see this

Write-back shows up everywhere performance matters:

| System               | Cache layer    | Backing store   | Durability strategy       |
| -------------------- | -------------- | --------------- | ------------------------- |
| LSM trees            | Memtable (RAM) | SSTables (disk) | WAL (write-ahead log)     |
| NVMe SSDs            | DRAM cache     | NAND flash      | Flush on fsync            |
| CPU                  | L1/L2/L3 cache | RAM             | None (volatile anyway)    |
| Linux page cache     | RAM            | Disk            | fsync, O_DIRECT to bypass |
| Database buffer pool | RAM            | Disk            | WAL, checkpoints          |

The pattern is the same: absorb writes in fast storage, flush to slow storage in batches.

### The NVMe parallel

NVMe drives have a DRAM write-back cache in front of the NAND flash. When you write, data lands in DRAM first. The drive reports success immediately, then flushes to flash in the background.

This is why databases use fsync to force data to stable storage. Without it, a power loss can lose recent writes that the drive already acknowledged.

Same trade-off as LSM trees: write-back gives you speed, but you need a strategy for failures.

### The combo: write-back + read-through

To make a disk fast, you want both:

1. Write-back for writes: data lands in cache, gets acknowledged immediately, syncs to backing store lazily
2. Read-through for reads: reads hit the cache first, which has the freshest data (including unflushed writes)

The cache becomes the source of truth for recent data. Reads never see stale data because they check the cache first, which contains writes that haven't hit disk yet. The backing store is just a durability layer that catches up asynchronously.

```
Write: App → Cache (ack) .......... (lazy sync) → Backing Store
Read:  App → Cache (has fresh data, including unflushed writes)
              ↓ (miss)
          Backing Store
```

This is how LSM trees work: reads check the memtable first (the write-back cache), then fall through to SSTables on disk. The memtable always has the freshest view. Compaction syncs everything to disk in the background.

Same pattern in a database buffer pool, CPU cache, or NVMe drive. The fast layer absorbs writes and serves reads. The slow layer provides durability.

### When to use what

- **Write-through**: durability matters more than speed, or you can't trust your cache
- **Write-back**: you need write performance and have a replication or WAL strategy
- **Write-around**: you're writing data that won't be read soon (bulk loads, cold data)
- **Read-through/cache-aside**: orthogonal to write strategy; depends on whether you want the cache to be transparent

### So what

Write-back is the key to making writes fast. But it's a bet: your cache won't die before flushing. LSM trees, NVMe drives, and CPU caches all make that bet. The difference is how they handle losing it.


Cache strategies, explained

When do you organize your data? Every write, or... later?

Hash tables and B-trees said: every write. LSM trees said: later. Turns out, procrastinating is sometimes the right call.

### Hash tables: The baseline

Hash tables give you O(1) lookups. You hash a key, go directly to that slot, done. Hard to beat.

But they have limitations:

- **No ordering**: you can't ask "give me all keys between A and M"
- **No range queries**: scanning requires touching every slot
- **Resizing is expensive**: when the table fills up, you rehash everything

For pure key-value lookups (memcached, Redis), hash tables are great. But databases need more.

### B-trees: The workhorse

B-trees solved the ordering problem. They keep keys sorted in a tree structure, giving you:

- O(log n) lookups
- Efficient range scans (just walk the leaves)
- Good cache/disk utilization (nodes sized to pages)

This is the backbone of most relational databases: PostgreSQL, MySQL InnoDB, SQLite. On every write, you:

1. Traverse the tree to find the right page
2. Read that page from disk
3. Modify it in place
4. Write it back
5. Maybe rebalance, triggering more writes

Your data is always organized. Lookups are predictable. But you're paying the bookkeeping cost on every operation, and that cost is random I/O.<a href="#ref-1">[1]</a>

### The log-structured insight

In 1991, Rosenblum and Ousterhout asked a different question: what if we never modified data in place?<a href="#ref-2">[2]</a>

Their Log-structured File System (LFS) treated the entire disk as an append-only log. Writes always go to the end. Old data gets garbage collected later.

This sounds wasteful, but it converts random writes into sequential writes. Sequential I/O is 10-100x faster on spinning disks, still 5-20x faster on SSDs.<a href="#ref-3">[3]</a>

The same insight shows up in filesystems:

| Filesystem | Write strategy | Philosophy |
| ---------- | ------------------- | ----------------------------------------- |
| ext4, XFS | In-place + journal | Established: fix it now |
| btrfs, ZFS | Copy-on-write | Next-gen: never overwrite, clean up later |
| bcachefs | Copy-on-write + LSM | Next-gen: both techniques combined |

Copy-on-write filesystems are basically applying log-structured patterns to the filesystem layer.

### LSM trees: Log-structured meets sorted

LSM trees (Log-Structured Merge trees) took the append-only insight and applied it to sorted indexes.<a href="#ref-4">[4]</a> This is what powers RocksDB, LevelDB, Cassandra, and ScyllaDB.

The design:

1. Writes go to memory first, in a sorted buffer called a memtable
2. When it fills up, flush to disk as a sorted file called an SSTable
3. Periodically merge SSTables in the background (compaction)

Instead of organizing on every write, you batch it up and do it asynchronously. The bookkeeping still happens, just later, in bulk, when sequential I/O makes it cheap.

### Don't you end up doing the same work?

No. Way less.

By batching hundreds of writes into a single sequential flush, you're amortizing the I/O cost across all of them. One disk seek instead of hundreds.

### But reads must be slower?

In theory, yes. You might check multiple SSTables to find your data.

In practice:

- **Bloom filters** on each SSTable tell you which files definitely don't have your key
- Recent data is usually still in the memtable (in memory)
- Hot data gets cached at multiple levels

For many workloads, read performance is comparable to B-trees. For write-heavy workloads, LSM trees dominate.

### Temporal vs spatial locality

Here's where it gets interesting.

Spatial locality assumes data near each other in address space should be stored together. B-trees optimize for this: record #1000 is stored near record #1001.

Temporal locality assumes data written at the same time should be stored together. LSM trees optimize for this: everything flushed in the same batch ends up in the same SSTable.

Why does temporal locality matter? Because it matches how systems actually behave:

| Scenario | Why temporal locality helps |
| ------------------ | ----------------------------------------------------------- |
| `npm install` | Thousands of files written in a burst; they belong together |
| App startup | Reads config and state that was all persisted together |
| Checkpoint restore | Reading a consistent snapshot |
| Transactions | Records committed together are accessed together |

Data you need right now was usually written together at some point. Grouping by write-time often matches access patterns better than grouping by key.

### The trade-offs (there's always trade-offs)

LSM trees optimize for writes at some cost:

- **Write amplification**: data gets rewritten multiple times during compaction
- **Space amplification**: you temporarily store redundant data
- **Read amplification**: worst-case reads check multiple levels

For write-heavy workloads (logging, time-series, event streams), the trade-off is worth it. For read-heavy OLTP with lots of point queries, B-trees still make sense.

### So what?

Back to our opening question: when do you pay the bookkeeping cost? B-trees say now, on every write. LSM trees say later, in bulk, when sequential I/O makes it cheap.

That's the trick. Procrastination, done right, can be an advantage.

We're seeing this pattern show up in unexpected places. Cloud block storage systems are adopting log-structured designs to handle virtualized storage at scale.<a href="#ref-5">[5]</a> And with AI training flushing massive checkpoints every few minutes, the demand for write-optimized storage is only growing. This might be the architecture for the next generation of storage infrastructure.

And that's my love letter to the LSM tree. It's not real magic, but it's damn close.



[1] Graefe. ["Modern B-Tree Techniques"](https://doi.org/10.1561/1900000028). Foundations and Trends in Databases, 2011.

[2] Rosenblum & Ousterhout. ["The Design and Implementation of a Log-Structured File System"](https://people.eecs.berkeley.edu/~brewer/cs262/LFS.pdf). SOSP, 1991.

[3] Jacobs. ["The Pathologies of Big Data"](https://queue.acm.org/detail.cfm?id=1563874). ACM Queue, 2009.

[4] O'Neil et al. ["The Log-Structured Merge-Tree"](https://www.cs.umb.edu/~poneil/lsmtree.pdf). Acta Informatica, 1996.

[5] Zhang et al. ["What's the Story in EBS Glory: Evolutions and Lessons in Building Cloud Block Store"](https://www.usenix.org/conference/fast24/presentation/zhang-weidong). USENIX FAST, 2024.

The art of procrastination, in storage

### Creating the Systemd Unit

Systemd is a process manager that can start, stop, and manage processes ("_Units_") on boot or shutdown of your machine!<a href="#ref-1">[1]</a>
In this walkthrough, we'll be starting a long-running Node.js web server that lives in `/var/www/`.

To create a new _Unit_, we need to create a file in `/etc/systemd/system/` (the default location on most Linux distributions).
This file should have the `.service` extension.

Let's create `/etc/systemd/system/webserver.service`:

```ini
[Unit]
# Since we're going to bind to a port on localhost, we need to wait for the
# network to be available:
After=network.target
Description=My fancy web server

[Service]
# Set the working directory to where your application lives
WorkingDirectory=/var/www
# Set any environment variables your application needs
Environment=NODE_ENV=production
Environment=PORT=3000
# Start your webserver (absolute path required - find yours with `which node`)
ExecStart=/usr/bin/node server.js
# Restart the service if it crashes
Restart=always
# Wait 3 seconds before restarting
RestartSec=3

[Install]
# Start this service when the system boots to multi-user mode
WantedBy=multi-user.target
```

After you've created the file, update it to have the correct permissions (`664`):

```bash
chmod 664 /etc/systemd/system/webserver.service
```

Don't be intimidated by the file above! A lot of it is boilerplate.
Read the comments placed in the above file to understand what each section is for!

> **Note**: Comments in systemd unit files must be on their own line—inline comments after values are not supported.<a href="#ref-2">[2]</a> The `ExecStart` directive requires an absolute path to the executable.<a href="#ref-3">[3]</a>

You can learn more about the [_Unit_](https://www.freedesktop.org/software/systemd/man/systemd.unit.html) or [_Service_](https://www.freedesktop.org/software/systemd/man/systemd.service.html) sections in the systemd documentation.

The _Install_ section is used to configure when the service will start. Most commonly, you'll want `multi-user.target`.<a href="#ref-4">[4]</a>
The `WantedBy` value directly corresponds to different [Linux runlevels](https://en.wikipedia.org/wiki/Runlevel):

| runlevel | WantedBy value | Description |
| -------- | ------------------- | ------------------------------------------------------ |
| 0 | `poweroff.target` | Run before the computer shuts down |
| 1 | `rescue.target` | Run when the system is in single-user mode |
| 2-4 | `multi-user.target` | Non-graphical multi-user system (networking available) |
| 5 | `graphical.target` | Run when the display manager has started |
| 6 | `reboot.target` | Run before the computer reboots |

### Enabling the Systemd Unit

Now that the unit has been created, we need to reload the Systemd process.
This will allow Systemd to recognize the new file.

Run the following command:

```bash
systemctl daemon-reload
```

> **Tip**: You need to execute the above command to reload Systemd **every time** you edit your `.service` file(s).

Next, you'll need to enable the service. Running this command will create a symlink into your `WantedBy` target.

```bash
systemctl enable webserver
```

Finally, since your system is already booted, run the following command to start the service:

```bash
systemctl start webserver
```

### Cheatsheet

**Enabling (creating) or Disabling (deleting) your service**

```bash
systemctl enable webserver # enable service to start on boot
systemctl disable webserver # disable service from starting on boot
```

**Starting, Restarting, or Stopping your service**

```bash
systemctl start webserver
systemctl restart webserver
systemctl stop webserver
```

**Checking the status of your service**

```bash
systemctl status webserver
```



[1] freedesktop.org. ["systemd.unit — Unit configuration"](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html). systemd documentation.

[2] freedesktop.org. ["systemd.syntax — General syntax of systemd configuration files"](https://www.freedesktop.org/software/systemd/man/systemd.syntax.html). "Lines beginning with '#' or ';' are ignored."

[3] freedesktop.org. ["systemd.service — Service unit configuration"](https://www.freedesktop.org/software/systemd/man/systemd.service.html). "The first argument must be either an absolute path or a simple file name without any slashes."

[4] Red Hat. ["Working with systemd unit files"](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/using_systemd_unit_files_to_customize_and_optimize_your_system/assembly_working-with-systemd-unit-files_working-with-systemd). RHEL 9 Documentation.

Walkthrough: How to start a long-running command on boot with systemd

`xargs` reads space and line delimited strings from stdin and executes a specified command with the input as arguments.<a href="#ref-1">[1]</a>

This makes `xargs` a very powerful command when you want to run a command against arbitrary input.

For example, let's run the following command:

```bash
❯ echo 'hello there\nbig world' | xargs echo
```

`xargs` automatically parses `hello`, `there`, `big`, and `world` as individual inputs.
Then, these arguments are passed to the specified command (in this case, also `echo`).

```bash
❯ echo 'hello there\nbig world' | xargs echo

# What gets executed:
# echo 'hello' 'there' 'big' 'world'

# Output:
hello there big world
```

> **Note**: All computed input arguments were passed to a single `echo` command.

For more practical usage, you may find yourself wanting to run a command once per argument.
For example, when the command you're using only accepts a single argument.

A common command I find this useful for is `aws s3`.
`aws s3` doesn't support passing multiple arguments, so each command may only accept a single S3 path.

For example, using brace expansion and `xargs` to pipe S3 paths to `aws s3 rm` doesn't work:

```bash
❯ echo s3://my-bucket/partition-{1..3}/ | xargs aws s3 rm --recursive

# What gets executed:
# aws s3 rm --recursive 's3://my-bucket/partition-1/' 's3://my-bucket/partition-2/' 's3://my-bucket/partition-3/'

# Output:
Unknown options: s3://my-bucket/partition-2/,s3://my-bucket/partition-3/
```

To solve this, `xargs` accepts an `-n <number>` argument.<a href="#ref-2">[2]</a>
The `-n` argument configures how many arguments should be per command invocation.

So, since `aws s3 rm` only accepts one argument, we can pass `-n 1` to `xargs` to execute one command per argument:

```bash
❯ echo s3://my-bucket/partition-{1..3}/ | xargs -n 1 aws s3 rm --recursive

# What gets executed (sequentially):
# aws s3 rm --recursive 's3://my-bucket/partition-1/'
# aws s3 rm --recursive 's3://my-bucket/partition-2/'
# aws s3 rm --recursive 's3://my-bucket/partition-3/'

# Output:
success
success
success
```

> **Note**: The `aws s3 rm` command was invoked three times, once per argument.

Success! 🎉

Now that we're running one command per argument, it's important to note it's done so sequentially.
However, you may also want to parallelize execution of long-running commands.

`xargs` has an argument for parallelizing execution: `-P <number>`.<a href="#ref-3">[3]</a>

The above `aws s3 rm --recursive` example may take a long time per path, so we can maximize efficiency by running all the commands at once.
To do this, we'll pass the `-P 3` argument:

```bash
❯ echo s3://my-bucket/partition-{1..3}/ | xargs -n 1 -P 3 aws s3 rm --recursive

# What gets executed (in parallel):
# aws s3 rm --recursive 's3://my-bucket/partition-1/'
# aws s3 rm --recursive 's3://my-bucket/partition-2/'
# aws s3 rm --recursive 's3://my-bucket/partition-3/'

# Output:
success
success
success
```

> **Tip**: To run one command per CPU, you can use this handy shortcut:
>
> ```bash
> xargs -n 1 -P $(getconf _NPROCESSORS_ONLN) command
> ```

Good luck! 🏎



[1] IEEE Std 1003.1. ["xargs — construct argument lists and invoke utility"](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html). The Open Group Base Specifications.

[2] GNU. ["xargs invocation"](https://www.gnu.org/software/findutils/manual/html_node/find_html/xargs-options.html). GNU Findutils Manual. <code style="font-size:inherit">-n max-args</code>: Use at most max-args arguments per command line.

[3] GNU. ["Controlling Parallelism"](https://www.gnu.org/software/findutils/manual/html_node/find_html/Controlling-Parallelism.html). GNU Findutils Manual. <code style="font-size:inherit">-P max-procs</code>: Run up to max-procs processes at a time.

timer.blog

Posts

Cache strategies, explained

The art of procrastination, in storage

Walkthrough: How to start a long-running command on boot with systemd

xargs(1), Shell Brace Expansion, and Single-Line Execution