Inspect Errors, Exceptions, and Overrides

Errors. Things that failed. Easy to find, usually already alerting, and the least informative, because a system failing loudly is a system somebody is already fixing.

Exceptions. Things that were refused, deferred, or blocked. More interesting: a refusal rate of zero almost never means everyone is well behaved, and a rising one usually means a rule is catching something it was not built for.

Overrides. Things a person decided differently from what the system recommended. This is the one that matters most and the one almost nobody has.

Override rate is the trust measurement

An override rate tells you something no other number here does: whether the people using the system believe it.

Adoption tells you whether they are using it, which is often mandatory. Accuracy tells you whether it was right, judged by somebody with hindsight. Override rate tells you what the person closest to the situation thought at the time, and a rising one is the earliest warning available that the system has started being wrong in a way nobody has articulated yet.

Three questions to ask of it: how often, in which direction, and is it concentrated. Overrides clustered in one segment mean the rules are wrong for that segment specifically, which is a much more actionable finding than a rate that has drifted up overall.

And you can only ask any of this if both sides were stored. 4a Chapter 4 made that decision. This chapter is what it was for.

Read the records

A standing agenda item, not a one-off audit: for at least one field-derived metric, go and read the records behind it. Nobody would ever find a lying field from the summary, because the summary is computed from the field, and the field is exactly the thing that was wrong.

Every finding type needs a route to a decision

An inspection that produces no decision path is a report, which is the failure this guide opened on. For every finding type on your list, write where it goes: who sees it, what they decide, and what the possible outcomes are.

If a finding type has no route, you have either found a missing owner or an inspection nobody was ever going to act on, and both are worth knowing before you commit a quarterly hour to it.