Measure Adoption, Cost, Speed, and Quality

The most useful thing in this chapter is a specific, surprising finding.

In a real system, the cost of the AI work turned out to track the volume of per-record signal sent rather than the number of calls made. The intuitive lever, calling less often, barely moves the bill. The effective lever, sending less per call, is invisible unless you measure it.

Generalize that: measure the actual cost driver before optimizing. Almost every system has one input that dominates the cost and it is frequently not the one whose name is on the invoice.

A measurement that produced a design change

A system was re-processing a measurable share of active records unnecessarily, because ordinary activity churn kept triggering refreshes that produced nothing new. Somebody measured it. The fix was a per-record cooldown: a record refreshed recently is skipped by the real-time path, and the scheduled batch catches it up.

That is the entire loop this guide is about, in one paragraph. Measure, find the waste, change the design, and the change is small because the measurement was specific.

Adoption, and what it does not tell you

Adoption is easy to measure and easy to over-read. A high adoption rate on something mandatory tells you compliance. Pair it with the override rate and you get something better: high usage with rising overrides is a system people have to use and have stopped believing, which is a specific and urgent state that neither number shows alone.

Also worth watching: abandonment. Not people who never started, but people who used it and stopped. That population knows something.

Publish what is not metered

In that same real system, only one lane is instrumented and per-call cost is not logged. The documentation says so. A team that knows exactly which part of the bill is invisible can put something else on that side; a team that believes the number is total will optimize confidently in the wrong place.