Compare Expected and Actual Outcomes

Find out whether the system was right, not just whether it ran.

See the situation

Start with the situation below. Next, we will work through an example together before you try the activity.

Elena · Reporting lead

The case

A disputed recommendation beside a prepared comparison

Observed outcomes can inform evaluation, but lag, missing capture, selection, and changed human actions affect what they prove.

What happened: The system routed it wrong and it was expensive
Is the story true?: Yes
Is it memorable?: Very
Base rate: Not asked for
Decision about to be made: Change the rule for everyone
Comparison population: 12 cases per confidence band per quarter
Outcome capture: Recorded booked conversation within 60 days; missingness not modeled in this prepared illustration
Important limit: A human may have changed the recommended action before the observed outcome

Your task

Compare the decision-time evidence with the later outcome. Keep the recommendation and human action separate.

Need a little guidance?

You’re done when: Differences are interpreted only where expected and actual are genuinely comparable.

Observed outcomes are not automatically ground truth for the original recommendation.

Need a reminder? Reopen the worked example

Watch one example first

A model recommends a call; the seller instead sends an email; a booking follows. That outcome cannot isolate the effect of the unperformed call recommendation.

Why this follows from the evidence

Preserve what was known at decision time, the recommendation, actual human action, and mature outcome. An observed association is not automatically causation.

GTM Lab

Saved locally

Operating Review and Improvement Plan · 0 of 9 sections started

Saved locally to your browser.

Case evidence

A disputed recommendation beside a prepared comparison

What happened: The system routed it wrong and it was expensive
Is the story true?: Yes
Is it memorable?: Very
Base rate: Not asked for
Decision about to be made: Change the rule for everyone
Comparison population: 12 cases per confidence band per quarter
Outcome capture: Recorded booked conversation within 60 days; missingness not modeled in this prepared illustration
Important limit: A human may have changed the recommended action before the observed outcome

A case is vivid, and a rule change is about to be made from it. Define a bounded comparison, then decide what the single case can establish.

Check the case below to save your activity result and this chapter’s prepared example plan section. You can inspect that template in the saved plan above; no live records are changed.

Review the case

Inspect what was known, recommended, changed, and observed

Compare the recommendation, the action taken, and the later result. Use the original groups so the comparison stays fair. It can show a relationship, but can’t prove what caused it.

Score bands and mature outcomesCompare the original state with the proposed change. This preview does not change live records.

Score bands and mature outcomes
Item	Before	Prepared outcome	Why
High score at decision time	20 records; 16 mature	12/16 successful (75%)	14 recommendations were followed; compare human action and outcome separately.
Medium score at decision time	20 records; 15 mature	9/15 successful (60%)	10 recommendations were followed; compare human action and outcome separately.
Low score at decision time	20 records; 12 mature	6/12 successful (50%)	9 recommendations were followed; compare human action and outcome separately.

High score at decision time: Before20 records; 16 mature; Prepared outcome12/16 successful (75%); Why14 recommendations were followed; compare human action and outcome separately.
Medium score at decision time: Before20 records; 15 mature; Prepared outcome9/15 successful (60%); Why10 recommendations were followed; compare human action and outcome separately.
Low score at decision time: Before20 records; 12 mature; Prepared outcome6/12 successful (50%); Why9 recommendations were followed; compare human action and outcome separately.

Case W-12Known at 09:00 CRM snapshot: Score 0.82; required integration present; source freshness 1 daySystem recommendationApprove for reviewHuman actionHold after a customer callLater informationBudget restriction documented at 14:00Observed outcomeNo booking within 60 days

Which cohort rule supports a fair comparison?What must remain distinct when a person changes the action?How long is the prepared outcome window?What can the group conclude?

Restoring this working packet…

Optional: compare alternative reasoning

The case tool above saves your result and completes this activity. These comparisons are extra practice; they do not change your saved plan.

Work the case · 0 of 2 sections drafted

Find out whether it was right

Compare these alternatives one section at a time. Check a choice to see its consequence; the draft summary below keeps the proposed changes together.

Step 1 of 2

How do you find out whether the system was right?

Survey the people using it about how accurate it feels.Compare its recommendation with a defined later outcome source. State the capture window, missing outcomes, and comparison population.Compare its earlier outputs against its later ones on the same records.

This is a proposed draft until the chapter check runs. Selecting a choice alone does not complete the chapter.

4c · Proposed

Improvement plan draft

Improvement plan draft · current draftCompare the original state with the proposed change. This preview does not change live records.

Improvement plan draft · current draft
Item	Before	Proposed	Why
How do you find out whether the system was right?	No saved change in this exercise draft	Not drafted yet	Choose a proposal to see what it would add to the draft.
A case suggests the recommendation may have been wrong. What do you check before changing the rule?	No saved change in this exercise draft	Not drafted yet	Choose a proposal to see what it would add to the draft.

How do you find out whether the system was right?: BeforeNo saved change in this exercise draft; ProposedNot drafted yet; WhyChoose a proposal to see what it would add to the draft.
A case suggests the recommendation may have been wrong. What do you check before changing the rule?: BeforeNo saved change in this exercise draft; ProposedNot drafted yet; WhyChoose a proposal to see what it would add to the draft.

A choice appears here as a proposal, not as evidence that a live system changed.

Your place saves as you go.

Chapter 3 of 90 complete