Group Commit

A constant that works at one load level is a bug waiting for load to change.

Say we have read chapter 3 and taken it seriously. Fsync-per-record was costing 25,000× what the bytes warranted, so we batch. Collect a thousand records, fsync once, acknowledge all of them together.

Under load, throughput jumps fifty-fold. We ship it. We are pleased with ourselves, and I think reasonably so.

A hundred seconds. Long enough to make a cup of tea while your database decides whether your write happened.

Then overnight, traffic drops to ten events a second. And the same code, unchanged, takes one hundred seconds to acknowledge a single write.

Now, nothing broke. I want to be careful about that, because the instinct is to go hunting for the bug. There is no bug. The arithmetic is exactly the arithmetic we should have expected: a thousand records arriving at ten per second takes a hundred seconds to fill the batch. We did not introduce a defect. We hard-coded a load level, and then the load politely changed on us.

5.1 Two Candidates and a Missing Number

So the constant was wrong. The interesting question is whether we should pick a better one or stop picking one, and those are genuinely different positions rather than degrees of the same one.

The first: pick a bigger fixed batch for throughput, a smaller one for latency, and tune the constant to match our traffic.

The second: stop picking a constant at all.

Name the ratio, because the ratio is the whole argument. A fixed batch of 1000 costs us:

at 100,000 events/sec:  1000 ÷ 100,000  =  10 ms  to fill    (fine)
at      10 events/sec:  1000 ÷      10  = 100  s  to fill    (not fine)

Ten million times worse. Same constant. Just a quieter night.

The constant did not change. The world did, and the constant had no way of finding out.

And that number is the argument against the first all by itself. There is no single value of N that is safe across a range that wide. Whatever you pick is correct at the traffic level you tuned it for and wrong by orders of magnitude everywhere else, and the place it is most wrong is the place nobody is watching, which is three in the morning.

5.2 The Axioms

One fact, and chapter 3 already handed it to us: an fsync round trip costs about the same whether it carries one record or a thousand.

Look at that table for a second longer than it seems to deserve, because that equality is the entire opportunity.

Only one of these questions requires guessing about the future, which is a good reason to prefer the other one.

If a barrier crossing is nearly free per additional record, then the thing worth optimizing is not how many records we force ourselves to wait for. It is how many records happen to arrive during a crossing we are already paying for. Those are completely different questions, and only one of them requires us to guess about the future.

5.3 Doing the Division

So let's not set N at all. Let's set a rule instead.

When the in-flight fsync returns, immediately start the next one, and sweep in everybody who arrived while the first was in flight.

t=0         fsync #1 issued (covers whatever's queued right now)
t=0..100µs  requests 2, 3, 4 arrive; they queue behind fsync #1
t=100µs     fsync #1 returns → ack 1; fsync #2 issued, covers {2,3,4}
t=100..??   whatever arrives now queues behind fsync #2
t=200µs     fsync #2 returns → ack 2,3,4; fsync #3 issued, covers {...}

Now watch what the batch size does. Nobody set it. It falls out of the arrival rate on its own.

At high load, dozens of requests pile up during one 100 µs window and ride across together, so we get the throughput win without ever naming a number. At low load, a request often finds nobody else queued at all, its "batch" is size one, and it waits exactly one fsync round trip. Not a hundred seconds. One round trip, which is the least it could possibly have waited.

The best batch size is the one nobody chose. There is a lesson in there about several other constants in your codebase.

That is the pleasing part. The scheme is not a compromise between the two cases. It is optimal at both ends, and it got there by refusing to answer a question it did not have to answer.

5.4 What This Rules Out

This rules out fixed-count and fixed-window batching as general answers. Not because either is wrong exactly, but because each one encodes an assumption about load that the adaptive version simply does not need to make. Fixed-count breaks at the low end. Fixed-window taxes everyone, always, including when nobody is waiting.

There is one wrinkle the naive adaptive version misses, and it is worth chasing down because it is the sort of thing that only shows up in production.

At moderate load, just under one arrival per fsync-latency window, batches regress toward size one anyway, and we find ourselves firing an fsync for very nearly every record. That is not a correctness problem. Latency is fine. Throughput is fine. But it is the maximum possible call rate, and on flash media, call rate correlates with write amplification, and write amplification is measured in years off the drive's life.

The fix is a small floor: hold the batch open for a minimum of roughly 200 µs, even if the in-flight fsync would have returned sooner. That caps how often the device gets hit, at the cost of a bounded, small, constant addition to latency.

The rejected idea comes back in the last act, behaving itself. This happens more often than the tidy version of engineering admits.

Which is the same trade as the fixed window we rejected two sections ago. Exactly the same trade. The difference is that here it is sized to be negligible instead of dominant, and I think that is worth saying out loud: the idea was never bad, it was just badly sized.

5.5 The Pictorial

The clearest way to see the difference is to put both policies on the same axis at two very different loads, and notice which one changes shape when the traffic does: