Chapters
- 01 · Pick the Right Decision Shape
- 02 · Apply Simple Rules First
- 03 · Use AI Where Interpretation Helps
- 04 · Handle Missing and Conflicting Evidence
- 05 · Treat Confidence Carefully
- 06 · Return a Consistent Result
- 07 · Test Plausible Edge Cases
- 08 · Capstone: Run the Qualification Set
15 min · Interactive exercise
Capstone: Run the Qualification Set
Run every design section against a full set of records at once.
Each section holds up alone. The shapes are right, rules run before the model, absence is distinguished from bad, confidence is honest, the output shape never varies, and the test set covers the known failures.
What is untested is the design as one thing. The seams are where the failures live: a rule ordering that is correct until a record is missing the input the first rule reads, an output contract that cannot represent the outcome the timeout produces, a segment calculation that skips the conflict check the general one applies.
Run the records, not the sections
The capstone runs the whole design against every record in the test set at once, plus the qualification cases from Define the Decision, and checks the interactions rather than the individual answers.
What this actually buys you
A design that passes this can be changed. That is the real return, and it is worth more than whatever accuracy the first version has.
Every decision system is wrong in some way you have not found yet. The question is whether, when you find it, you can tell which part was wrong, change that part, and know whether the change helped. The deciding-rule field tells you which part. The version stamp tells you what changed when. The test set tells you whether you broke something that used to work. Without those, the first correction is a guess, and the second one is a guess about the first.
Carry this into your business
Ask whether your team could change a rule in an automated decision this week and know by next week whether it helped. If the answer is no, the missing thing is almost never the model. It is one of the three fields this guide asked you to record.
GTM Lab
Saved locallyDecision Design
sample-decision-design · v1 · draft
Run the qualification set
Eight records, each drawn from a failure that actually happened. Every section held up alone; the failures that matter are the ones between sections.
- Complete evidence, no conflicts, nothing estimatedRoute to a seller
The path every design handles. Proves nothing on its own. - A renewal sitting in one stage for months, behaving normallyScored by the renewal calculation, or blank outside the window
Exposes a whole population being scored by rules built for a different one. - Old stage date, genuinely recent meetingThe meeting survives the capped penalty
Exposes an unbounded term cancelling the most informative fact on the record. - Required activity date never stampedNot scored, flagged for logging
Exposes absence being read as a bad signal rather than a missing one. - Large company, message reads as a testStopped by the interpretive veto
Exposes rule ordering: cheap numeric rules first would route this. - Genuine interest far below the scale thresholdKeep and nurture
Exposes a missing middle outcome. - Two sources disagree on the deciding fieldReview, both values recorded
Exposes a decision quietly picking a side. - The decision never returnsDefined default, marked as defaulted
Exposes a record stranded behind a gate with no escape.
Run all 12 validation checks
The checks test the seams between sections, since every section was written while looking at that section. Each runs by its stable ID and reports pass or fail with a reason.