DIY CRM Database Health Center architecture

This page is an advanced Reference implementation, not a starting point. Work through buy, configure, connect, or build first. Most data-quality gaps resolve with tools you already own; in the worked example that page runs through, only one of three broken capabilities actually justifies a build. This page describes the full architecture a credible custom system uses when several capabilities genuinely do justify one: adjudicated deduplication, rule-driven enrichment, and spend governance, none of which a CRM's native tooling or most off-the-shelf products combine into one system, so you can build only the piece your own scoring justifies instead of the whole thing.

The shape: one health surface over several working systems

A DIY health center isn't one feature; it's a small set of systems sharing a data model and a common operating surface:

Health overview: a snapshot of population totals, inactive counts, duplicate candidates, field-population gaps, and queue depths, refreshed on a defined cadence.
Duplicate detection and merge: the adjudicated pipeline described in duplicate detection, merge review, and survivorship, running as scheduled batches with full job history.
Rule-driven enrichment: the eligibility and trigger model from enrichment eligibility and moment-of-use enrichment, configured through rules rather than hard-coded logic.
Cost and approval governance: the spend ledger and approval queue from enrichment cost controls and approval queues.
Field-population analytics: fill-rate visibility by field, object, and segment, so enrichment spend targets records that are actually thin instead of re-buying data you already hold.

Each system can, in principle, be built independently. The reason to unify them under one health surface is that they share state: a duplicate that's about to merge shouldn't also be mid-enrichment, an over-cap enrichment job shouldn't queue work for records a same-day merge is about to disqualify, and a field-population view is only useful if it reflects the same snapshot the other systems are working from.

Scan runtime: four states, one promotion rule

The audit checklist's four-state model (running, last successful, stale, partial failure) is the runtime contract this whole system depends on. Implement it as an explicit state machine, not an implicit "did the job return 200" check:

A scan starts in running state.
Work happens in chunks (batch processing, not one long transaction), each chunk producing findings tagged with the scan's run ID and rule-set version.
Only after every required rule family completes does the run get promoted to last successful, replacing the previously current snapshot.
A run that doesn't complete every required rule family is marked partial failure and stays visible, but never becomes the current snapshot.
If no run has completed inside the expected cadence, the current snapshot is marked stale so the UI can say so honestly instead of silently presenting old data as current.

Retry transient failures (a timeout, a rate limit). Surface a distinct error category for a structural problem (a broken rule definition, a permissions gap) so it doesn't get silently retried forever while looking like ordinary flakiness. Every remediation action a person takes off a finding should be idempotent (running it twice produces the same result as running it once) and separately audited, so a bad merge or a bad correction can be traced back to the specific job and person that made it.

Duplicate pipeline: detect, adjudicate, merge, log

The pipeline itself runs as a chain: detect candidate pairs (exact-match signals first, fuzzy signals second, scored across the full population rather than only within a batch chunk), route each pair through the confidence tiers from duplicate detection, merge review, and survivorship, auto-merge what clears the high-confidence bar with the survivorship rules applied, and hold everything else in a review queue with its match evidence attached.

An AI or rules-based adjudication step between "matched" and "merged" is what catches the false-positive class that pure similarity scoring can't: two distinct people who happen to share identifying signals, like a surname and an employer domain. Adjudication doesn't replace the protected-record check; both run. A pair can pass adjudication and still get blocked from auto-merging if either record is protected (open Opportunity, active CS relationship, recent marketing engagement, closed-won status).

Every scan, adjudication pass, and merge batch needs job history: what ran, what changed, who or what ran it, so a bad outcome is traceable to its cause instead of discovered by accident.

Rule-driven enrichment: configuration, not hard-coded logic

Enrichment eligibility and triggers (see enrichment eligibility and moment-of-use enrichment) should be admin-configurable rules, not logic buried in code: an object, a target field, a trigger (on-creation, on-update, on-handoff, on a schedule scoped to eligible segments), and a provider. Keep the provider layer pluggable. A rule framework that only works with one enrichment vendor becomes expensive to change later; one that treats "which provider" as a rule parameter rather than a hard dependency survives a vendor switch without a rebuild.

Every rule should be individually toggleable and show its own last-run time and processed-record count, so an admin can tell which rules are actually running and which have quietly gone stale, the same failure mode the scan-state model exists to prevent at the system level.

Cost governance: the gate that makes everything else safe to turn on

Every credit-consuming call funnels through a single gate that checks the metered ledger from enrichment cost controls and approval queues before the call and records consumption after it. Centralizing this in one gate, rather than letting each enrichment rule manage its own spend tracking, is what makes the cap and the kill switch actually reliable: there's exactly one place spend gets metered, and exactly one switch that disables paid calls without touching free matching or lookup functionality.

Be explicit about what the gate does and doesn't cover. A gate on your own automated callouts has no visibility into a rep spending credits directly inside a provider's own web UI or dialer. That's a real limitation worth stating plainly rather than implying the automated gate covers all organizational spend; it needs its own separate control, typically the provider's native per-seat limits.

Field-population analytics: target the gaps, don't re-buy what you have

A fill-rate view, filterable by object and segment, is what turns enrichment spend from "refresh everything" into "refresh what's actually thin." Without this visibility, enrichment defaults to re-buying data you already hold, which is how a per-credit provider becomes the most expensive line in the stack regardless of its subscription price. Pair this view with the eligibility segments from the enrichment-eligibility page so the two systems agree on which records are worth spending on.

What this design pattern has been worth, in one measured case

I built a version of this architecture and ran it in production. What follows is what it was worth in that one system. It is a single company's outcome, not a guaranteed result of building this pattern:

Consolidating overlapping enrichment and dedupe subscriptions onto one governed platform plus this kind of system saved about $50,000 a year.
The same database's duplicate rate fell from about 5% to under 1% after the adjudicated detection-and-merge pipeline replaced ad hoc manual cleanup.
Most core fields on that database now sit above 95% population, with the field-population view making it possible to see exactly which fields and segments are still thin, rather than treating the aggregate number as proof the whole database is trustworthy.
Moving enrichment from a blanket schedule to eligibility rules and moment-of-use triggers cut predicted monthly enrichment credit use by more than 80%.

Read none of these as an expected outcome. They describe what happened in one production system, against its own prior baseline, under its own constraints. Your starting duplicate rate, tooling contracts, and field coverage decide what the same design is worth to you.

GTM Lab: what "build" actually produces

Continuing the scoring worked through in buy, configure, connect, or build, GTM Lab's build-versus-buy record justifies building exactly one bounded piece of this architecture: the enrichment cost-governance gate (the atomic spend ledger, the defer-not-drop rule, and the approval queue), reusing the scan-state model, the rule-driven configuration pattern, and the single-gate cost governance described above. Deduplication stays on native platform tooling plus the review queue. Sequence-orchestration reconciliation stays on the sequencing tool's native CRM connector, with only a thin custom rule layer added on top of it for the lifecycle fields the connector doesn't recognize, not a system built from scratch.

This is a worked example, and it's a deliberately modest one: two of GTM Lab's three broken capabilities never needed this architecture at all. The specific piece GTM Lab builds follows directly from its own build-vs-buy scoring, not from a rule that says every data-quality gap eventually gets a custom system.

DIY health center checklist

The build was justified by a documented buy/configure/connect/build decision, not by default.
The scan runtime distinguishes running, last successful, stale, and partial failure, and a failed run never overwrites the last successful snapshot.
The duplicate pipeline includes an adjudication step and a protected-record check, not fuzzy matching alone.
Enrichment rules are admin-configurable, provider-pluggable, and individually toggleable with visible last-run state.
All credit-consuming calls funnel through one metered gate with a kill switch and a stated coverage boundary.
Field-population analytics are filterable by object and segment, and enrichment targeting uses them.
Every job, scan, and remediation action is logged, idempotent, and traceable to its cause.

Where this connects

This page assembles the systems defined individually in duplicate detection, merge review, and survivorship, enrichment eligibility and moment-of-use enrichment, enrichment cost controls and approval queues, and conflict resolution and overwrite policy. The scan-state model implements the cadence contract from CRM data quality audit.