gtmjosh | Notify and Escalate to the Right Owner

Notify and Escalate to the Right Owner

Give every alert an owner and a remediation, and watch the two numbers everybody ignores.

Every alert names two things or it is not an alert: who hears it, and what they do about it.

The second one is the one that gets skipped, and it is the reason alert fatigue exists. An alert that says something is wrong, to a person with no procedure, produces one of two behaviors. They improvise, differently each time, or they learn to ignore it. Usually both in that order.

An alert with a remediation attached is a task. An alert without one is an interruption.

The runbook metric set

Ten numbers, and this list is worth copying directly. Request volume. Queue age. Processing latency. Success and failure by category. Retry count. Freshness of whatever the workflow depends on. Budget or quota state. Manual override rate. Reconciliation drift. User-visible error state.

Two of those get ignored and both are diagnostic in a way the others are not. Reconciliation drift is Chapter 5's convergence measure, and a drift that is not trending to zero means the loop is not converging. Manual override rate is the only number in the list that measures whether people actually trust the system, and a rising one is the earliest warning you will get.

Commitments are structured work items

When the workflow produces a commitment, a promise that somebody will do something, model it as a real work item: the trigger event, the requested timing, the owner, and the disposition. Close it out through explicit completion, reschedule, or escalation states.

A promise living in somebody's memory is not tracked work, and it is not visible to anything in this plan.

GTM Lab

Saved locally

Cross-System Operating Plan · 0 of 8 sections started

Saved locally to your browser.

Make every alert a task

The loop above needs somebody told, and told what to do. Write the alerts, then choose the numbers worth watching.

Fixture: diff-that-never-zeroes

A loop chasing its own tail

FIXTURE-DIFF-NEVER-ZEROES

Cycle after cycle: The diff always has work to do
How it looks on a dashboard: Healthy activity
The same rows: Recurring, cycle after cycle
Row count over time: Growing, on a schedule
Complaints: Arriving much later than the numbers did

Four alerts are proposed. Which set is complete?

All four name a condition, an owner, and the first thing that owner should do.
An alert with a remediation is a task. An alert without one is an interruption, and that is the whole of alert fatigue.All four name a condition and an owner; two of them leave the response to the owner’s judgment.
Trusts the people who know the systems best rather than prescribing a fixed procedure.All four name a condition and a remediation, posted to the operations channel.
Everybody who could act on it sees it, which is faster than routing to one person.

Two of the runbook metrics get ignored and both are diagnostic. Which two?

Reconciliation drift and manual override rate.
Drift not trending to zero means the loop is not converging. Override rate is the only number in the set that measures whether people trust the system, and a rising one is the earliest warning you get.Request volume and processing latency.
The two numbers everybody quotes in a status meeting, and the two that move first when something breaks.Retry count and queue age.
Both are early indicators that a downstream system is struggling before users notice.

Chapter 6 of 80 complete