gtmjosh | Plan for Stop, Undo, and Recovery

Plan for Stop, Undo, and Recovery

Build a kill switch people will pull, route failures by category, and show four states instead of a spinner.

A switch that turns the whole system off is a switch people hesitate to pull at exactly the moment they should pull it, because pulling it has its own cost and the person with their finger on it is not certain yet.

The version that gets used is graduated. It disables the expensive or dangerous path while leaving the safe one working: paid reveals off, free matching still running. Now the decision to pull it is cheap, so it gets pulled early, which is when it is useful.

Transient and permanent are different failures

This is the fundamental retry taxonomy and almost everything else follows from it.

A transient failure is one that might succeed if you try again: a timeout, a rate limit, a service having a bad minute. It goes back in the queue. A permanent failure will never succeed no matter how many times you try: a malformed configuration, a record that does not exist, a credential that is wrong. It goes to a person.

Retrying a permanent failure a thousand times is a misconfiguration a thousand times, and it produces a log nobody can read on top of the original problem. Routing a transient failure to a person wastes their attention on something that would have fixed itself.

There is a third category the previous guide already named: the stall, where nothing failed and nothing is happening because a person has not acted. It is neither transient nor permanent and it needs the third destination.

Four states, not a spinner

A status display has to represent four things independently: running, stale, partially failed, and last successful.

Most show one number and a spinner, which collapses all four. The user cannot tell whether the number in front of them is current, an hour old, or the surviving half of a run that mostly failed.

And the rule that makes it work: a failed run must never erase the last successful result. If tonight's scan fails, yesterday's numbers stay on the screen, clearly labelled as yesterday's. The alternative is a blank screen during exactly the incident when somebody needs the numbers.

GTM Lab

Saved locally

Workflow Control Plan · 0 of 8 sections started

Saved locally to your browser.

Build a switch people will pull

During the incident above, two people spent the useful part of an hour deciding whether things were bad enough yet. Fix the switch, then route the failures.

Fixture: total-switch

The switch nobody pulled

FIXTURE-TOTAL-SWITCH

What it does: Stops the entire system
Cost of pulling it: Every workflow halts, including the safe ones
During the incident: Two people discussed whether it was bad enough yet
Time before it was pulled: Long enough to matter
Times it has been pulled since: Zero

What shape should the kill switch be?

Graduated: disables paid reveals and outbound sending, leaves free matching, flagging, and everything read-only running.
Now the decision to pull it is cheap, so it gets pulled early, which is when it is useful.Total: one switch that stops the entire system.
Unambiguous, impossible to get half-right, and guaranteed to stop whatever is going wrong.Graduated, and requiring two named people to agree before it can be pulled.
Prevents a single person from halting revenue-generating automation on a hunch.

Five failures. Where does each one go?

Timeouts and rate limits back in the queue. Bad configuration and bad credentials to the operations owner. The unreviewed draft to the owner’s manager.
Transient goes back, permanent goes to a person, and the stall is the third category: nothing failed, and nothing is happening.All five back in the queue with a retry limit.
One mechanism, no classification to maintain, and a retry limit stops any infinite loop.All five to the operations owner.
Guarantees a human sees every failure, which is what most incident policies ask for.

What must the status display be able to show?

Running, stale, partially failed, and last successful, each independently, with a failed run never erasing the last good result.
Four states because they are four different situations, and the fourth one is what keeps the screen useful during an incident.The current figure, with a spinner while a run is in progress.
What almost every dashboard shows, and it answers the question people usually ask.Running, failed, and succeeded, with the display cleared when a run fails.
Never shows a stale number, so nobody can mistake old data for current data.

Chapter 6 of 80 complete