Write-Ahead

Never make a promise you cannot reconstruct.

I want to show you a bug that is worse than a lie, because everybody involved was telling the truth.

Suppose fsync() returns EIO. A real hardware write failure, correctly detected, correctly surfaced. So we do the responsible thing: we log it, and we retry the fsync.

The most dangerous error message is the one that says everything is fine.

It returns 0. Success.

The bytes it just claimed to have persisted were never written, and they never will be. The kernel is not lying to you. It is telling you the honest truth about the only attempt it still has any record of.

This was a real bug, found in PostgreSQL in 2018, and once you see why it happened, it changes what the phrase "handle the error" is even permitted to mean.

4.1 Two Candidates and a Missing Number

Before we decide what to do about that, we have to settle what actually happened, and there are two readings of it that lead somewhere very different.

The first: an fsync error behaves like any other I/O error. Log it, back off, retry, and eventually it either succeeds or you escalate.

The second: retrying does nothing at all, because there is nothing left to retry.

This is not a ratio we can divide our way out of. It is a question about what the kernel does at the instant writeback fails, so let's just trace it through, slowly, because the answer is stranger than it looks.

A dirty page fails to reach the device. The kernel reports the error to whoever calls fsync next, and it reports it once. Then it marks the page clean and drops it.

Read that again. Not clean because it succeeded. Clean because the kernel gave up and stopped tracking it as pending work. The data that failed to write is now simply gone from the page cache. There is nothing dirty left for a second fsync to flush.

Nobody lied. The kernel answered exactly the question it was asked, which was not the question anybody meant.

So the second call returns success. Honestly. From where it is standing, there is genuinely nothing left to do.

4.2 The Axioms

There is no arithmetic to do here, which is unusual for this book. What we need instead are three facts about kernel behaviour, none of which you can reason your way to from first principles, and all three of which have to be true at once for the bug above to happen.

A backup on the same disk is not a backup. Everybody agrees with this, and roughly half of us keep one anyway.

Put those together and you get something that sounds almost harsh: an fsync failure is not a transient fault to be retried. It is a terminal event. The only sound response is to treat that data as lost and rebuild it from somewhere the kernel did not just erase.

4.3 Doing the Division

That "independent, already-durable source" has a name. It is the write-ahead log, and the pattern that survives this failure mode is narrow and specific. Every clause in it is carrying weight:

1. write the record to the WAL          (userspace buffer)
2. fsync the WAL                        (the one barrier you pay for)
3. ACK the client                       ← the only line that matters
4. apply the change to the data pages   (lazily, async, whenever)

Steps 1 through 3 are the whole durability contract. Step 4 can happen a second later, a minute later, or after a crash and a replay, and it genuinely does not matter which, because if the crash arrives before step 4 finishes we do not need step 4's result. We need the WAL, and we already made certain that one thing was durable before we told anybody we were done.

The data pages, meanwhile, are allowed to be wrong. Half-written. Entirely missing. At any instant you care to inspect them.

That sounds reckless until you see why it is fine: they are derived state. They can be rebuilt. The WAL is the only object in this picture that cannot be reconstructed from anything else, and that is exactly why it is the one thing that gets the expensive, synchronous, ordered barrier we bought in chapter 3.

The data pages are allowed to be wrong. It takes a while to be comfortable writing that sentence down.

One barrier. On the one irreplaceable thing.

4.4 What This Rules Out

This rules out the natural instinct to spread fsync calls evenly across everything that looks important.

Not all durable-looking writes are equally irreplaceable. A data page is a cached, derivable projection of the log. Losing an unflushed data page after a crash costs you a replay, which is to say it costs you some seconds. Losing the log entry costs you the fact itself. There is no second copy anywhere, and no amount of retrying a broken fsync brings it back.

Now here is a question that looks entirely unrelated, and I promise it is the same question.

Why does PostgreSQL's torn-page protection (full_page_writes) take its reference image from the in-memory buffer pool, and never by re-reading the on-disk file it is trying to protect?

It would be so much simpler to read the file. The file is right there.

But the disk file is precisely the thing we do not trust. That is the entire premise of needing torn-page protection in the first place: a page write that crashes partway through leaves a sector-level mixture of old and new bytes. If our known-good reference came from reading that same file back, we would be verifying the disk against itself, and a corrupted page would cheerfully certify its own corruption.

Asking the suspect to vouch for the suspect. It passes every test you can think of, which is the trouble.

The buffer pool copy is the one version of that page our process actually validated. It is the only candidate in the room that is not circular.

Same rule as before, wearing different clothes: your recovery source has to be independent of the thing that failed.

4.5 The Pictorial

Four boxes and one arrow that matters. Watch where the acknowledgement sits, because everything in the design follows from its position:

 client request
        │
        ▼
 ┌──────────────┐   write()    ┌──────────────┐   fsync()    ┌──────────┐
 │  WAL record  │─────────────▶│  page cache  │─────────────▶│  device  │
 └──────────────┘              └──────────────┘              └──────────┘
                                                                        │
                                                         ACK client ◀───┘
                                                         (the only line
                                                           that matters)
        │
        ▼ (lazy, async, any time after the ACK)
 ┌──────────────┐
 │  data pages  │  ← derived. Can be rebuilt from the WAL.
 └──────────────┘     Never the thing we promised on.