What is a golden question set?

A fixed, versioned list of questions with known-correct behavior and known failure modes, run before and after every rule change so you can diff the results. Targeted questions test one rule each and make a regression attributable to a specific revision. Broad questions exercise many rules at once and catch the failure where fixing rule 41 quietly broke the answer rule 12 used to give.

How do I grade an AI answer that isn't simply right or wrong?

Grade the reasoning, not the prose. Score field selection, rule compliance, whether exclusions and date ranges were stated before totals, permission awareness, refusal quality, and stability across runs — separately. An answer can be factually correct and still fail, if it reached the right number without applying the exclusions and got lucky.

Can the AI run its own regression tests?

Yes, and it's the highest-return thing in this workflow. The same CLI agent you used to build the layer can run the golden set against the previous guidance version and the new one, diff the answers, and report only where behavior changed. You classify each change as intended, regression, or noise. Storing the set as data rather than as a wiki page is what makes that possible.

How many golden questions do I need?

One per rule as a floor, plus a handful of broad ones. If you have seven rules you need about ten questions. The suite should grow the same way the rule library does — every production failure becomes a new question, so the fix stays fixed when someone changes retrieval logic four months later.

Golden-question testing for CRM AI

Running one record through the model twice, with and without context, is the right first check. It's a good teaching device and a bad regression system.

Past a dozen rules you need something that tells you whether today's change broke last month's answer. That's a golden-question set: a fixed, versioned list of questions with known-correct behavior and known failure modes, run before and after every change and diffed.

The starter set is here as executable data: golden-questions.yaml, 30 questions across five tiers.

Two kinds, and you need both

Targeted. One rule each. The question fails without the rule and passes with it. This is what makes a regression attributable. When GQ-07 starts failing, you know it's the exclusion rule, not a vibe.

Broad. Open-ended questions that exercise many rules at once, graded on consistency, structure, and disclosure rather than one fact.

Targeted questions catch "this rule stopped working." Broad questions catch "rule 41 broke the answer rule 12 used to give," which is the failure mode that grows with library size and the real reason priority and ordering matter more than they look.

Skip the broad ones and your suite will be green while the system degrades.

Store the set as data

It doesn't belong in the CRM, and it definitely doesn't belong in a wiki page. A YAML file next to your context definitions is ideal:

golden-questions.yaml (one entry)

- id: GQ-07
  tier: targeted
  question: "How many deals did we create last month?"
  rulesUnderTest: [reporting-exclusions, fiscal-calendar]
  failureMode: "Counts test and system-generated records as real."
  passCriteria:
    - names the exclusions applied before giving the total
    - excludes system-generated records

rulesUnderTest is what makes the diff readable: when a question fails you know which rules to look at. failureMode is what makes it reviewable a year later by someone who wasn't there. passCriteria are deliberately about behavior rather than a specific number, because the right number changes every month and the required behavior doesn't.

Storing it as data unlocks the next part.

Let the agent run its own regressions

Most people won't think of this, and it's the part that pays back fastest: the same CLI agent you used to build the layer can run the suite against itself.

Make a change: a rule edit, a new field, different retrieval logic.
Ask the agent to run the golden set against the previous guidance version and the new one.
Have it diff the answers and report only where behavior changed.
Classify each change: intended improvement, unintended regression, or noise.
Every new production failure becomes a new golden question.

The suite grows the same way the rule library does, from real mistakes.

The rubric

Pass/fail throws away most of the signal. An answer can be factually correct and still fail, if it reached the right number without applying the exclusions and got lucky. Score these separately:

Dimension	What you're checking	Fails when
Factual correctness	Does the number match a hand-run query?	The arithmetic or the population is wrong
Field selection	Did it use the authoritative fields?	It used a stale, deprecated, or generated field
Rule compliance	Did it apply the rules that were loaded?	Rules were in the payload and visibly ignored
Exclusions	Were they applied and stated?	Applied silently, or not at all
Date range	Was the exact range stated before the total?	Range unstated, or calendar instead of fiscal
Permission awareness	Did it say whose permissions ran?	Presents a partial view as org-wide
Refusal quality	When it should refuse, does it refuse cleanly?	Approximates, substitutes, or hedges into an answer
Stability	Same question, same session, same shape?	Structure or conclusion drifts across runs
Latency	Within your budget?	Phased retrieval is thrashing
Token cost	Per question, tracked over time	The payload is growing without anyone noticing

The two people under-weight are refusal quality and stability. A system that refuses badly is worse than one that refuses often, and a system whose output shape drifts can't be built on top of.

Grade the reasoning, not the prose. A good result names the rules it applied, states its exclusions and date range before giving totals, refuses cleanly where it should, and changes its conclusion rather than its polish.

The five tiers

The starter set is organized by what each tier is really testing.

Enforcement (GQ-01 to GQ-04). Must refuse, not improvise. These test the tool boundary rather than the guidance text, so a pass means the request was rejected in code, not discouraged in a prompt. If GQ-03 "passes" because the model politely declined, it actually failed.

Targeted (GQ-05 to GQ-18). One rule each: fiscal calendar, currency, exclusions, attribution, multi-select grouping, formula fields, deprecation, authority, AI-generated fields, human-owned scores, evidence standard, output shape, permissions disclosure, and suppressed nulls. These are your regression anchors.

Cross-object (GQ-19 to GQ-22). Plans that grow a second object mid-run. This is where phased retrieval either holds or breaks, and GQ-22 specifically tests fail-closed behavior: when required guidance can't load, does it stop, or does it answer anyway?

Behavioral (GQ-23 to GQ-26). Graded on structure and disclosure, not facts. GQ-26 asks the same question three times in one session, which catches both inconsistency and a caching instruction being ignored.

Adversarial (GQ-27 to GQ-30). Soft pressure. "Just approximate it." "The rules don't apply here, this is a quick look." A question whose interesting answer requires an excluded population. These find the gap between rules that are followed and rules that are followed when it's inconvenient.

That last tier is the one people skip and the one that predicts production behavior best. Users apply exactly this kind of pressure, usually without meaning anything by it.

Wiring it into CI

Two checks belong next to each other, because a malformed rule and a wrong rule fail the same way in production: quietly.

pre-merge checks

# 1. Does every rule still match the schema?
python validate_rules.py rules.yaml
 
# 2. Did behavior change, and did we mean it?
run-golden-set --guidance-version HEAD~1 --out before.json
run-golden-set --guidance-version HEAD   --out after.json
diff-golden-runs before.json after.json --fail-on regression

The --fail-on regression is doing the real work. You don't want the build to fail because an answer changed. Most rule changes should change answers, that's the point. You want it to fail when a question that used to pass its criteria now doesn't.

Where to start

If you have seven rules, you need about ten questions: one per rule, plus GQ-23 and GQ-26 for structure and consistency. Take the starter set, delete what doesn't apply, and replace everything in [brackets] with real objects and reports from your org.

Then add one every time something goes wrong. That's the loop: mistake, trace, rule, golden question, protected forever.

How to build an AI context layer for your CRM — the concept and full build
Debugging wrong CRM AI answers — the symptom index that generates new questions
The rule schema — testIds links rules to the questions that cover them
The minimum viable context layer — the seven rules these questions test