Twenty-Five Microseconds

Any sufficiently advanced measurement is indistinguishable from a lie.

I want to start with something you can do right now, without leaving this page. Write a megabyte to a file. Time the call. See what comes back.

On an ordinary machine, on an ordinary day, it comes back in about twenty-five microseconds.

Twenty-five microseconds is about how long it takes light to run seven and a half kilometres. Your megabyte did not go anywhere near that far.

Now, if you run that a hundred times, the number stops registering. It's small, the write worked, the test passed, and there's a whole afternoon of other work waiting. I have done exactly this and thought nothing of it. But twenty-five microseconds is the loudest thing that will happen to you all week, and this chapter is about learning to actually hear it.

We are not going to look anything up. By the end you will know exactly where that megabyte is sitting, and we will have gotten there with four numbers and some long division, which is honestly all this ever takes.

1.1 Two Candidates and a Missing Number

Let's start where everybody starts. Where could the data actually be?

Either it's still in memory, or it's out on the disk. That's a reasonable first cut. And the reflex that follows, the one I have had a hundred times, is to say disk is slower than memory and feel like some progress has been made.

It hasn't. Not really. That's a direction, and a direction cannot decide anything by itself. Both candidates are still standing, and we have no idea which way to lean, because we never asked the question that actually separates them:

How much slower?

This is the habit the whole book is built on, so it is worth saying plainly. When you name two possibilities, name the ratio between them in the same breath. The ratio is what tells you whether the question was worth asking at all.

1.2 The Axioms

Here is the part nobody quite says out loud. Some of this we cannot derive. Not because we aren't clever enough, but because there is nothing underneath it to derive from.

A cache line is 64 bytes because somebody, once, in a meeting, chose 64. DRAM latency is whatever physics and market economics happened to settle on between them. There is no first-principles path to either number. You just have to know it, the way you know a stop sign is red.

So let's take them as given. There is no shame in being handed a constant. The only real shame is needing to be handed the conclusion too.

A dozen constants is fewer than most people carry for their own family's birthdays, and considerably more useful at two in the morning.

A dozen numbers roughly like these make up most of the toolkit. Twelve numbers. That's it. What we do with them is the entire skill.

1.3 Doing the Division

One megabyte, three plausible paths home.

Before you read on, write down what you think the answer is. I mean it, actually write it down. Not because the arithmetic is hard, but because committing to a value you might be wrong about is the whole exercise, and reading past this sentence without doing it is how you get the comfortable feeling of learning without any of the learning.

1 MB ÷  10 GB/s  =  100 µs     memory copy
1 MB ÷   2 GB/s  =  500 µs     NVMe
1 MB ÷ 150 MB/s  =  6.7 ms     spinning disk

Now let's put those next to the thing we actually measured.

Sit with that a moment, because it is not the outcome anybody was set up to expect.

Twenty-five microseconds beats every option on the list. Every one. Including the option that never touches storage at all. So the conclusion is not "it must be in memory." The conclusion is much better than that: our list of candidates was wrong.

The ratio test did not just rank our candidates against each other. It caught that the question itself was malformed. Nobody told us we were buying that when we started dividing, and it is the most useful thing in the chapter.

Let me digress for a paragraph, because this rule has a family resemblance to something you have probably already lived through.

Years ago I watched somebody optimize a numerical loop and report a speedup of about four hundred times. Four hundred! Everyone was delighted. There was a graph. What had actually happened was that the result of the loop was never read, so the compiler noticed this and deleted the entire computation, and the benchmark was faithfully timing an empty program. The number was real. The measurement was honest. The work had simply not occurred.

The graph was beautiful. Somebody had put the four hundred in bold.

That is the same rule as the one in the box, wearing different clothes, and it is why I trust it more than almost anything else in this book. A result that is too good does not mean you won. It means you should go looking for the step that got skipped. Nobody moved a megabyte anywhere in twenty-five microseconds, and now we get to find out what got skipped.

1.4 Turning It On Our Own Answer

The textbook reflex here is page cache. The kernel took the bytes, parked them in its own memory, and returned without ever touching the device. Deferred write-back. It is clever, it is well documented, and it feels like an answer the instant you say it out loud.

So let's turn the floor test on it, exactly the way we would turn it on somebody else's claim.

Copying into page cache is still a memory copy. It is still a megabyte moving through a single core at ten gigabytes a second. That is 100 µs, a number we derived four paragraphs ago and now get to reuse for free.

Page cache is four times too slow to explain what we saw.

It does not survive contact with our own arithmetic. Not because we read something that contradicted it, not because an expert corrected us, but because numbers we already had in hand would not let it stand. That is a lovely way to be wrong. It costs nothing and it happens in ten seconds.

So the kernel never copied our data either. Which leaves something stranger and more interesting. Maybe nothing copied it at all.

1.5 Reading the Band

We have a floor. Now let's get a ceiling, coming at it from the other direction.

plain function call, off the stack  ~1–5 ns
null syscall (post-Spectre)         ~0.5–5 µs
1 MB memcpy                         ~100 µs

Twenty-five microseconds is five thousand times too slow for a bare function call, and four times too fast for a full DRAM-bandwidth copy. It is wedged between two floors, and the gap is narrow enough to mean something specific.

Twenty to fifty microseconds is the signature of a memcpy into a buffer that is already hot in the CPU cache. L2 and L3 can move data at 30–50 GB/s on a modern core, three to five times the ~10 GB/s you get when the destination lives in main memory. The bytes never left your process. No syscall happened at all.

That is exactly what fwrite, BufWriter, and the default buffered file API in most languages do. They keep your bytes in the process's own memory and defer the syscall until there is enough to make the trip worthwhile. At twenty-five microseconds, the floor test does not reject this: 1 MB in 25 µs is roughly 40 GB/s, which is comfortable L2/L3 cache bandwidth on a modern core.

So the explanation here is simpler than it first looked. The data was copied, but only into a cache-hot userspace buffer sitting in the CPU's own backyard. No kernel crossing, no device, no page cache. Just a fast, local memcpy that the DRAM-bandwidth floor fails to catch because the destination was never in DRAM.

What survives is the buffered copy: a memcpy into a cache-hot userspace buffer that never crossed into the kernel.

And here is a detail I find genuinely delightful. Buffering explains plenty of fast returns, just smaller ones. Ask for four kilobytes instead of a megabyte and the copy costs well under a microsecond, and buffering becomes the likeliest answer in the room. The mechanism we land on depends on the size we asked about. Which is why the megabyte was in the question at all, and why "how fast is a write" is not a question that has an answer.

1.6 Resisting strace

Here is where most of us reach for a tool. I want you to resist for one more second, and ask what the tool would actually buy.

We are separating "no syscall," which is nanoseconds, from "syscall that enqueued," which is microseconds. Three orders of magnitude apart. No amount of measurement noise, scheduler jitter, or thermal drift closes a 1000× gap. The stopwatch already answered it.

Nobody has ever been criticized in a review for adding more instrumentation, which is most of what is wrong with reviews.

For the day you genuinely need them: strace -c counts syscalls, perf trace times them. They are excellent tools and you should need them far less often than you think.

1.7 So Where Is It

Four layers, in the order they would survive:

   YOUR PROCESS
 ┌──────────────────┐
 │ userspace buffer │  gone if your process crashes
 └──────────────────┘
          │  write()        ~1–5 µs
          ▼
   THE KERNEL
 ┌──────────────────┐
 │   page cache     │  survives a crash
 └──────────────────┘  DIES on power loss
          │  fsync()        ~100 µs – 10 ms
          ▼
   THE DEVICE
 ┌──────────────────┐
 │   drive cache    │  still volatile
 └──────────────────┘
          │
          ▼
 ┌──────────────────┐
 │  FLASH / PLATTER │  actually durable
 └──────────────────┘

At twenty-five microseconds, you are at level one. Not two. Not four.

Now, the names in that diagram are worth complaining about for a moment. write() is the syscall that leaves userspace, which the name rather undersells. Your data lands in page cache. And fsync() is not "send it to the kernel," because it is already there. fsync is what pushes it out of the kernel, toward the media.

write() does not write and fsync() does not send. Both names are older than I am and neither is going to be fixed now.

Most people know about fsync. What catches people off guard is subtler, and it is the thing I would tattoo on a wall if I could: the default path in nearly every file API stops one level short of durable, and you have to opt into that last step on purpose. Closing a file flushes userspace into the kernel. It does not sync.

Which sets up everything in Part II. That last arrow turns out to be expensive enough that entire database architectures exist for no reason other than to avoid taking it more often than they must.