Upfront rules vs. dynamic retrieval: I ran the numbers

Once you have a context layer grounding your CRM AI, there are two ways to get those rules to the model.

Load them upfront. Put the whole rule set in the system prompt. With prompt caching, that prefix gets cached, and cache reads run about a tenth the price of a fresh input token. Call this Full.

Fetch them on demand. Let the agent pull only the rules a question actually needs, as tool calls, during the conversation. Fewer rules per answer, fewer tokens. Call this Dynamic.

Dynamic sounds cheaper. It reads less. I believed that for a while. Then I instrumented the real token usage and ran an 18-question golden set both ways.

Dynamic read fewer tokens and cost more.

Per 18-question run	Full	Dynamic
Total input tokens read	3.54M	2.67M (−25%)
Billed cost (weighted)	842K	1.24M (+48%)
Wall-clock time	1,033s	1,158s (+12%)
Correctness	same	same

So Dynamic read 25% fewer tokens, cost 48% more, ran 12% slower, and answered exactly as well. On every count that isn't raw token volume, it lost.

Why fewer tokens costs more

It's the caching math. Your cached prefix reads at about 0.1× a fresh token. Tool results don't live in that prefix. They land in the message history, and message history is fresh input at full price on every following turn.

Dynamic moves your guidance out of the cheap cached prefix and into expensive per-turn history. Uncached input more than doubled. And every tool result gets re-sent on every later turn.

Look at one 19-turn question. The cached prefix read 723K tokens but only weighed about 72K after the cache discount. The uncached input was 221K, and it weighed the full 221K. That's three times the cost at a third of the volume. A rule the agent fetched at turn 3 gets paid for 16 more times before the question ends.

Fewer tokens isn't the goal. Cheaper tokens is.

The change that actually cut spend

The win wasn't the retrieval strategy. It was caching the conversation itself.

Most setups cache the system prefix and stop there. But you can mark the last message each turn, which makes turn N's history a cacheable prefix for turn N+1. Each call then reads the prior conversation at 0.1× and only writes its small delta.

Use a short time-to-live on that conversation breakpoint. A conversation only has to survive one question, so paying the write premium for a long TTL buys nothing. In my data, a 5-minute TTL beat an hour.

Full-mode billed input	Today	1h TTL	5m TTL
Weighted tokens	842K	613K (−27%)	540K (−36%)

That's directional. It assumes history grows in a straight line and ignores the minimum cacheable size, so expect the real number a bit under 36%. Answers don't change. It's the same rules, cached better.

When dynamic might still be worth it

There's one case where Dynamic has a real shot: a question about a single record that already carries fresh context.

This is where I have to name two things that sound the same.

• The context layer is the general rule set: what your fields mean, which to trust, what the agent may touch. It applies to every record.
• Record context is a small, pre-computed briefing written onto one specific record ahead of time. It's specific to that record.

When a record already has fresh record context, the agent usually answers in one or two passes instead of exploring. That's exactly the case where Dynamic's extra turns stop hurting, because there aren't many.

The tempting shortcut is to route on object count, or on which page the user is standing on. Both misfire. Two questions asked from the same record page went opposite ways: a renewal question won big on Dynamic, a campaign-pipeline question lost by more than double. Page context tells you where someone is standing, not how many objects the answer needs, and you only learn that after the agent digs in. Guessing wrong is lopsided too: a bad Dynamic call can cost several times more, while a good one saves a little.

The signal that actually predicts a Dynamic win is "does this record already have fresh context?" You know that at submit time. Object count you don't.

Where I landed

Default to loading your rules upfront in a cached prefix. It's cheaper, faster, and just as accurate.

Then cache the conversation, not just the system prompt. That's the real lever, and it's a config change, not a rewrite.

Keep dynamic retrieval in your back pocket for single-record questions on records that already have context. And don't turn it on until you can measure it, because the cheaper-looking option was the expensive one, and I'd never have known without the numbers.