Runbooks that survive a reorganisation

Most runbooks are written by the person who just fixed the problem, while the fix is still warm. That is the worst possible moment: everything obvious to them is invisible, and everything they had to discover is already forgotten.

The reader you are writing for is the third one. Not you, not your colleague who was on the call — the engineer who joins in eighteen months, opens this page at three in the morning, and has never seen the system healthy.

Name the failure, not the procedure

A runbook titled Restarting the ingestion service is filed under an answer. The reader arrives with a symptom, not an answer: the queue depth alarm fired, or the dashboard is stale, or a customer says their order vanished.

Title the page after what the reader sees:

• Queue depth above 10k for more than five minutes — not Draining the queue
• Stock figures differ between the store app and the warehouse — not Reconciliation job
• Login succeeds, then the next request 401s — not Rotating the signing key

One symptom can lead to several procedures. That is fine; the branch belongs inside the page, where the reader can see why they are being sent one way rather than the other.

Write the check before the action

Every step that changes state should be preceded by the observation that justifies it and followed by the observation that confirms it. Without the first, the reader is guessing. Without the second, they cannot tell whether they helped.

1. Check   SELECT count(*) FROM outbox WHERE published_at IS NULL;
2. Expect  a number that keeps rising between two runs
3. Do      restart the publisher on one node only
4. Confirm the same query returns a falling number within 60s
5. If not  stop here and escalate — the queue is not the problem

Step 5 is the one people leave out, and it is the one that matters. A runbook that has no exit tells a tired engineer to keep applying a procedure that is not working.

Record the decision, not only the command

The command will change. The reason it was the right command tends not to.

We restart one node rather than the whole deployment because the publisher takes a lease; restarting all of them at once means every node races for the lease and none of them wins.

That sentence survives three refactors of the deploy script. kubectl rollout restart does not.

Keep it honest by using it

A runbook nobody has followed since it was written is a hypothesis. The cheapest way to test it is to have someone who did not write it follow it, out loud, during a quiet afternoon — an exercise that takes twenty minutes and reliably finds two steps that no longer exist.

The measure of an operational document is not whether it is complete. It is whether the next person can act from it without finding you.