Evidence requirements, structured outputs, and confidence

A decision is only as trustworthy as what it was allowed to read. Three things need stating: which evidence is required, what the decision must refuse to infer, and how an absence is handled.

Three roles, and the third is the one that gets skipped

The prohibited list is what stops a decision quietly acquiring inputs. Without it, somebody adds a signal that correlates well in testing and nobody notices that the correlation came from a bulk import, or that the field encodes something the business has decided not to decide on.

For each required item, record where it comes from, what authority it carries, and how fresh it has to be. The vocabulary is already established in the CRM field dictionary; reuse it rather than inventing a parallel one.

Estimation must announce itself

When a required value is absent, a decision may estimate. Three constraints make that safe:

1. It must record that it estimated, on the result.
2. It must record the basis, specifically enough that somebody could disagree.
3. It must never write the estimate into the field that holds observed values.

The third is the one that causes lasting damage. Once an estimate lands in an observed field it is byte-identical to a measurement, and every downstream consumer, report, and future decision treats it as one. There is no way to un-mix them afterwards. See AI-generated CRM fields are not a source of truth.

The shape of a decision result

A decision that returns only an outcome cannot be reviewed, explained, or improved. The minimum useful result:

{
  "decision": "inbound-routing-v1",
  "version": "1.4",
  "subject": "003xx000004TmiQ",
  "outcome": "keep-and-nurture",
  "confidence": "medium",
  "deciding_rule": "company-size-below-threshold",
  "evidence": [
    {"field": "company_employee_count", "value": 40, "source": "provider", "observed_at": "2026-08-14", "kind": "observed"},
    {"field": "stated_need", "value": "evaluating search relevance", "source": "form", "observed_at": "2026-08-23", "kind": "observed"},
    {"field": "annual_revenue", "value": 6000000, "source": "estimated", "basis": "industry and headcount", "kind": "inferred"}
  ],
  "evidence_not_used": ["job_title_seniority"],
  "counterfactual": "Above the size threshold, this would have routed to a seller.",
  "state": "evaluated",
  "decided_at": "2026-08-23T09:14:22Z"
}

Four fields earn their place and are usually missing.

deciding_rule. Showing eleven inputs is not an explanation. Naming the one rule that determined the outcome is.

kind per evidence item. Observed and inferred must be distinguishable at the field level, not summarised at the top.

evidence_not_used. Showing what the decision was not permitted to consider stops a reviewer assuming it weighed something it never saw.

state, separate from outcome. Pending, evaluated, needs-review, suppressed, and failed are states. Collapsing failed into an outcome is how an outage becomes a population of disqualified records.

Confidence: bands, not numbers

A numeric confidence invites two errors. It gets compared across records where the comparison means nothing, and it gets treated as a probability it was never calibrated to be.

Use three or four named bands, and define each by what happens rather than by a range:

Two constraints keep it honest.

Low confidence is not a review trigger on its own. Routing everything uncertain to a person builds a queue nobody can clear. Combine confidence with consequence: a low-confidence decision with a reversible, internal action can proceed and be sampled.

Confidence must be scored against outcomes. If low-confidence decisions turn out no worse than high-confidence ones, the confidence is decorative and should be fixed or removed. A decorative confidence signal is worse than none, because people act on it. See error, exception, and override reporting.

Evidence sufficiency is a separate question from confidence

A decision can be highly confident and under-evidenced: certain about the answer the available fields imply, when the fields that would change the answer are absent.

Record them separately. Evidence sufficiency asks whether the required items were present. Confidence asks how strongly the present evidence supports the outcome. A result that is high-confidence and low-sufficiency is exactly the one to route for review, and a single blended number hides it.

Audit one week of results

Take every decision the system returned in a week and cross-tabulate sufficiency against confidence. The high-confidence, low-sufficiency cell is the one to read record by record, because those are the answers that look strongest and rest on least, and they are the ones people act on without checking.

Two other numbers are worth having. What share of results carried an estimated value, and whether anyone downstream can tell. And whether the confidence bands separate anything: if the accuracy of the high band and the medium band is the same, the bands are decoration and the thresholds behind them were never calibrated.

Related: Outcome sets and unknown states, Explaining decisions and review queues, and CRM field dictionary for where source and freshness are recorded. The interactive walkthrough is Make the Decision Without Faking Certainty.