Treat Confidence Carefully

A confidence figure now arrives alongside every answer. The team does the sensible-looking thing and uses it: anything above 0.8 routes automatically, anything below goes to review.

Two months later somebody checks. Among the answers the system rated above 0.8, roughly one in seven was wrong. Nobody had assumed 0.8 meant a four in five chance of being right, exactly, but everybody had assumed it meant something close to that. It does not. It is an internal number reflecting how strongly the inputs pointed one way. It is not a probability of correctness, it was never calibrated against outcomes, and nobody had checked whether the two tracked.

Confidence is a description of the input, not a promise about the output

Three things get confused, and separating them costs one paragraph and saves a lot. Confidence is how strongly the available evidence pointed toward the answer given. Accuracy is how often the system is actually right, and it can only be known by comparing answers to outcomes after the fact. Evidence sufficiency is whether there was enough to decide at all, and it is the one that usually matters most operationally. A decision can be highly confident on thin evidence, and that combination is the dangerous one.

Two rules follow. Never present a confidence number as a probability of being right unless you have measured the relationship. If you have not, say “strong signal” and “weak signal” rather than 0.87, which invites arithmetic nobody is entitled to do. Gate on evidence sufficiency, not on confidence. A confident answer built on two of five required inputs should not act automatically no matter how confident it is.

False precision has a second source

Even without a model, a number can claim more than it knows, and composite scores are where it happens. GTM Lab's engagement score subtracts a penalty for time in stage from a base built on meeting recency, then adds points for recent email. The subtraction is floored at zero before the email points are added.

That ordering is invisible and it changes what the number means. A badly stalled deal with a recent email thread scores exactly its email points, because the base was ground to zero first. A score in the low twenties on an old deal is almost always “two emails and nothing else,” not “a somewhat healthy deal.” Two records showing 22 can have nothing in common.

The fix is not a better formula, because the formula is doing what it should. It is that a composite score must carry its components. Store the parts alongside the total. Then 22 is readable as “base 0, email 22” and the meaning is recovered. The general rule: if a reader cannot reconstruct why a number is what it is, the number invites false precision, and somebody will eventually make a decision on the difference between a 22 and a 26 that does not exist.

Benchmarks need their denominators

The same discipline applies to any aggregate the decision produces. “Deals in this segment average 58” is a benchmark. “Deals in this segment average 58, across six open deals” is a fact, and it is a fact that stops somebody setting a target against noise. A segment average over a handful of records is not a benchmark, and publishing it as one is how a number acquires authority it never earned. Carry the denominator everywhere the aggregate goes. It is the cheapest honesty available.

Carry this into your business

Find a threshold in your business expressed as a confidence number and ask whether anyone has compared answers above it to what actually happened. If not, the threshold is a guess wearing a decimal point, and knowing that is more useful than changing it.