gtmjosh | Compare Expected and Actual Outcomes

Compare Expected and Actual Outcomes

Find out whether the system was right, not just whether it ran.

Everything so far measures whether the system ran. This chapter is about whether it was correct, which is a completely different question and the one most operating reviews never ask.

The comparison

Take what the system said would happen and compare it with what happened. The records it recommended routing, against the ones that produced a result. The confidence it attached, against how often that band turned out right. The outcome it predicted, against the outcome recorded.

None of that is possible unless both sides were preserved, which is why this chapter is short and its dependency is long. 4a Chapter 4 is what makes this chapter possible, and a reader who skipped it will find this one theoretical.

Confidence bands are checkable

If the system attaches a confidence to its outputs, that confidence is a claim and it can be scored. Take the low-confidence decisions and ask how often they turned out wrong compared with the high-confidence ones. If there is no difference, the confidence is decorative and should either be fixed or removed, because a decorative confidence signal is worse than none: people act on it.

The anecdote and the base rate, again

Reviews attract stories. Somebody brings the deal the system got badly wrong, and the story is usually true and usually memorable.

Pair it with the base rate before deciding anything. One vivid failure out of a large population may be the system working as designed, and the same story out of a small population is a serious problem. The story is what gets the item onto the agenda. The base rate is what decides the response.

GTM Lab

Saved locally

Operating Review and Improvement Plan · 0 of 9 sections started

Saved locally to your browser.

Find out whether it was right

A true, memorable, expensive story is in front of the review and a decision is about to be made from it. Define the comparison, then handle the story.

Fixture: one-bad-deal

The story that got onto the agenda

FIXTURE-ONE-BAD-DEAL

What happened: The system routed it wrong and it was expensive
Is the story true?: Yes
Is it memorable?: Very
Base rate: Not asked for
Decision about to be made: Change the rule for everyone

How do you find out whether the system was right?

Compare what it recommended with what was recorded, grouped by the confidence attached at the time.
Only possible because both sides were preserved on every entry. Grouping by confidence also scores the confidence, which is itself a claim.Compare its earlier outputs against its later ones on the same records.
Uses only data the system produced, so it needs nothing extra to have been stored.Survey the people using it about how accurate it feels.
Goes straight to the people with the most context on each individual case.

The story is true and the outcome was bad. What do you do first?

Ask for the base rate before deciding anything.
The story gets the item onto the agenda; the base rate decides the response. Asking is uncomfortable in a room where somebody is upset, and it is the right move.Change the rule so it cannot happen again.
Responds directly to a real failure with a real cost, which is what people in the room want.Note it as anecdotal and move on.
Avoids over-reacting to a single case, which is a genuine risk in any review.

Chapter 3 of 90 complete