The Barrier

fsync is not a save button. It is a purchase.

Here is a loop you can write in about a minute. Take an 8-byte counter. Append it to a file. Do that over and over.

In memory, that loop runs fifty million times a second. Now add one call, fsync() after every append, and it runs two thousand times a second.

Twenty-five thousand times slower. To persist eight bytes.

Not eight megabytes. Eight bytes. Hold that number next to the size for a moment, because something has gone badly wrong with our intuition about what we just bought.

If fsync were charging us for the data, eight bytes would be the cheapest thing we could possibly ask it to do. So either the data is enormously more expensive to move than we think, or we are paying for something else entirely and the data is beside the point.

Those two possibilities are the whole chapter, so let's take them seriously one at a time.

3.1 Two Candidates and a Missing Number

The first candidate is the intuitive one. fsync is slow because it moves our bytes to the device, and moving bytes takes time proportional to how many bytes there are. Eight bytes is a small trip, but a trip to a disk is a long way, and that is what we are paying for.

The second candidate says the byte count has nothing to do with it. We are being charged for something that would cost the same if we had handed over nothing at all.

Those two lead to opposite fixes, so it is worth an hour to tell them apart. And we can do better than an hour, because there is a division here that settles it in about ten seconds.

Name the ratio before you pick. At even a modest 2 GB/s of device bandwidth, eight bytes should cost:

8 B ÷ 2 GB/s ≈ 4 nanoseconds

We measured about 100 microseconds for that same call.

That is not a near miss. That is 25,000× over the bandwidth floor. The first candidate is not slightly wrong, or wrong in the details. It is wrong by more than four orders of magnitude, which is the kind of wrong you can see from orbit. Set it aside.

Being wrong by four orders of magnitude is oddly restful. There is no danger whatsoever that you were nearly right.

Which leaves candidate two, and a question we cannot answer by staring at it any harder: if we are not paying for bytes, what exactly are we paying for? For that we need some numbers.

3.2 The Axioms

We cannot derive what an fsync costs. It depends on a device, and a device is a fact about the world rather than a consequence of anything. So here are the three numbers, handed over, for the three kinds of storage you are likely to be sitting on:

Roughly a hundredfold spread top to bottom, which is worth knowing. But the spread is not the interesting part of that table, and if you take only one thing from it, take this instead.

Every one of those numbers stays nearly flat whether you sync 8 bytes or 8 kilobytes.

A cost that ignores a thousandfold change in the payload is trying to tell you something, and it is not being subtle about it.

That flatness is the whole tell. If a cost barely moves when you change the payload by a factor of a thousand, then the payload was never the dominant term. It was not even close to being the dominant term. You are paying for something else, and the size of your data is a rounding error on the bill.

So we are paying for an event rather than a quantity. The next question writes itself: what event?

3.3 Doing the Division

The answer is that fsync is not really a write at all. It is a barrier.

It blocks until every write queued before it is provably sitting on non-volatile media. And it makes a second promise the name hides completely: a promise about order. Nothing after the barrier gets to be considered durable before everything ahead of it is.

Once you see it as a barrier rather than a save button, the flat cost stops being mysterious. You are not paying to move bytes. You are paying for a round trip: a request out to the device, a confirmation back, and a wait in between that no amount of shrinking your payload will shorten.

And that reframing immediately changes what we should do about it. Paying the barrier once for a batch of a thousand records, and paying it a thousand times for those same thousand records one at a time, are not remotely the same purchase. The first is one round trip. The second is a thousand of them, each one carrying almost nothing.

3.4 What This Rules Out

Take that rule seriously for a moment and notice what it costs us, because it is more than a tuning suggestion.

Twenty-five thousand times slower, for eight bytes. You could hand-deliver the eight bytes.

It rules out the intuition that durability is something you sprinkle on. "Just fsync after every write, to be safe." Safety is not the axis that scales badly here. Call count is. A service that fsyncs every record has chosen to pay a toll of 100 µs to 10 ms on every operation, however small, because the toll booth does not care what is in the trunk. It charges for the stop.

Now let me push on something we have been quietly treating as one thing, because I think it is really two, and separating them is what the next chapter is built on.

Durability and ordering are different purchases. You can imagine a system that needs its writes applied in order but does not need each one durable the instant it lands. You can imagine the reverse too: durability, with no opinion at all about relative order. Those are genuinely different requirements, and a careful engineer might well want to buy them separately.

fsync will not let you. It sells both, bundled, whether or not you wanted the bundle.

Two guarantees, one price, and no itemized receipt.

And once you notice that, the design that follows is almost forced. Put only the thing that must be ordered and durable behind the barrier, and let everything else move lazily along behind it. That is chapter 4, and that is the entire idea. Before we get there, it is worth seeing the barrier drawn out, because the picture makes the waste obvious in a way the arithmetic does not.

3.5 The Pictorial

Here it is drawn out. Three records queued above the line, one crossing, and everything below it provably on the media:

 record 1 ─┐
 record 2 ─┤  queued, unordered w.r.t. the barrier
 record 3 ─┘
           │
     ══════╪══════  fsync()  ←  the barrier: one round trip,
           │                    ~100 µs – 10 ms, size-insensitive
           ▼
 [ record 1 | record 2 | record 3 ]  ← provably durable, in this order,
                                        only past this line

Three records behind the barrier, or three thousand, at the same price. Enjoy it. Very little else in this book is on offer.

That is the shape you want, and it costs one round trip no matter how many records are stacked above the line.