Test Unsafe Requests

Every control in this plan is a claim. The claim is that when a particular thing happens, the system will not do a particular other thing.

An untested claim of that shape is worth very little, and worse, it is confidently believed. The usual way people find out that a control was never wired up is that nothing was ever rejected, which reads like good behavior and is actually silence.

There is a published version of this point about tool boundaries in AI agents: a rejection rate of exactly zero almost never means users are well behaved. It usually means the boundary was never connected. The same is true of every control in this chapter.

Four kinds of unsafe request

The one that is correct and unacceptable. Every rule passes and a forbidden state applies. This is the Chapter 4 test, and it is the one most likely to be missing, because it is the only category where the system has to refuse something it believes in.

The one that arrives at volume. A single request that is fine and ten thousand of them that are not. Bulk is its own failure category: things that are safe individually stop being safe when the rate changes.

The one that succeeds partially. Half the batch worked. The interesting question is not whether the failure was handled but what the system reports afterwards, and whether the half that worked is distinguishable from the half that did not.

The one where the repair is the hazard. A reconciliation or cleanup loop with a defect does not fail quietly. It amplifies, because it runs repeatedly and each run acts on the state the previous one left. This category is the reason the next guide spends a whole chapter on measuring convergence.

Write them as expectations

Each test names the request, the control that must refuse it, and the observable result. “The enrichment rule must refuse a record on an account in an active renewal, and the refusal must state the forbidden state by name” is testable. “The system should be careful about renewals” is not.