gtmjosh | Show Sample Size, Missing Data, and Blind Spots

Show Sample Size, Missing Data, and Blind Spots

Print the thing next to the number that stops it being misread.

This is the chapter most worth taking away, and it contains no statistics.

Show the count beside the average

A real breakdown displays a per-segment average next to the number of records in that segment, and carries a note saying the highest-scoring segment rests on too few records to be treated as a benchmark.

One extra column. That column prevents the single most common reporting error there is, which is somebody quoting the top segment's rate in a meeting as a target.

Note what it does not do: it does not suppress the number. Suppress the conclusion, not the data. Showing a rate on a handful of records with the count attached is honest. Hiding it is a different distortion, because now the reader does not know the segment exists.

The ratio that flatters

A real analysis reports two groups a couple of points apart and describes the difference as roughly double. Both statements are arithmetically true. They land completely differently.

“Three in a hundred versus five in a hundred” and “almost twice as likely” describe the identical finding. The first invites the question of whether two points is worth reorganizing around. The second sounds like a strategy.

Neither is a lie. The rule is simply: when you state a ratio, state the difference too. Both, together, one sentence. The reader can then decide which framing matters for their decision, which is their job and not yours.

Publish the blind spots

The best single practice in this space: a metric that ships with a section listing what it cannot see.

A real engagement metric publishes that a particular kind of activity earns nothing at all, that a scheduled event nobody attended still scores full marks until the date passes, and that events predating the record are invisible. Every one of those is a real limitation, published, next to the number.

A metric with a published blind-spot list is trustworthy in a way one without cannot be, and the reason is not that it has fewer blind spots. It is that somebody went looking.

GTM Lab

Saved locally

Metric Dictionary and Reporting Brief · 0 of 9 sections started

Saved locally to your browser.

Print the thing that stops it being misread

Three sentences above describe one identical finding. All three are true. Choose which belongs in the brief, then decide where a caveat is needed and where it is noise.

Fixture: three-honest-sentences

One finding, three framings, all true

FIXTURE-THREE-SENTENCES

As a difference: A couple of points apart
As a ratio: Almost twice as likely
With the sample size: A couple of points apart, on small groups
Arithmetically: All three identical
In a meeting: Three different decisions

Which framing goes in the brief?

Both: the multiple and the absolute difference, in one sentence, with the record counts.
The reader can then decide which framing matters for their decision, which is their job rather than yours.The ratio. It communicates the significance of the gap.
Ratios travel further in a meeting and are what people remember afterwards.The absolute difference. It is the conservative statement.
Avoids overstating a small gap, which is the more common failure.

Which metrics need a sample-size disclosure?

The ones on small or partial populations. The one on a large complete population states why it does not.
Waiving a disclosure is a decision, so it gets a reason. A waiver with no reason is indistinguishable from having forgotten.All of them. Consistency makes caveats credible.
Nobody can accuse the brief of disclosing selectively.None. Suppress any figure whose sample is too small to be reliable.
Prevents a weak number from being quoted at all, which is the outcome the caveat is trying to achieve.

What else travels next to a metric?

A published list of what the metric cannot see.
A metric with a published blind-spot list is trustworthy not because it has fewer blind spots, but because somebody went looking.A confidence rating for the metric overall.
One compact signal the reader can weigh without reading a list.

Chapter 6 of 90 complete