Illustrative scenario: two generated answers cite the same policy but use different phrasing. An agent, supervisor and QA reviewer may need a consistent approved response for that question class.
This scenario motivates a design that combines retrieval with reviewed playbooks. Answer consistency and factual accuracy should be evaluated separately; neither is guaranteed by the architecture alone.
The fix is what we now call the crystallization loop. This post describes what it is, why it works, and how it gets built. It's an engineering post — there's specific architecture below — but the idea behind it generalizes to any AI deployment where consistency matters more than novelty.
The problem with pure RAG
Retrieval-augmented generation is a powerful default for grounding AI responses in source material. You retrieve the relevant chunks, you stuff them into the prompt, you let the model write the answer. The answer is grounded — every fact has a source — and the architecture scales gracefully as the corpus grows.
Two risks to evaluate are:
- Same question, different answer. The retrieval pulls slightly different chunks on different calls (because of embedding noise, recency biasing, or chunk-boundary effects), and the model phrases the answer differently each time. The substance is consistent. The wording isn't.
- The auditor's question. "Why did the system say what it said?" has the same answer every time: "Because the model produced it from these chunks." That answer is true and not useful. There's no policy decision to point at. There's no playbook version. There's no human in the chain.
For commercial CX where the cost of variance is low, pure RAG is fine. For regulated CX where the cost of variance is high — eligibility determinations, member benefit explanations, policy citations, hearing procedures — it is not.
What "crystallization" means
Crystallization is the process of taking a recurring generative response pattern and converting it into a deterministic playbook. The generative response is the seed; the playbook is the crystal. Once a playbook exists for a given question class, the system uses it. The model is invoked only when no playbook matches.
The architecture has three loops, running at different cadences:
The proposed serving loop checks the playbook library for an approved match and uses retrieval-assisted generation when no suitable playbook applies. Measure latency and response quality in the intended environment.
The capture loop watches the generative responses. It clusters them by intent, by retrieval signature, and by surface form. A cluster of responses that consistently answer the same question — across different agents, different calls, different days — becomes a candidate for crystallization.
The promotion loop takes a candidate cluster and produces a draft playbook: a deterministic response template with the key facts, the citations, and the canonical phrasing. The draft goes to a human reviewer (a domain expert at the customer agency) who edits, approves, and assigns a version number. The approved playbook enters the library. From the next call forward, that question class hits the deterministic path.
The serving loop in detail
The serving loop is conceptually simple but operationally precise. The order of operations matters:
- Intent classification. A small, fast model classifies the inbound question into one of the agency's intent categories. We use an embedding-based classifier with a calibrated similarity threshold; the alternative — letting the LLM classify — is too slow and too variable for the hot path.
- Playbook lookup. The intent category is the key into the playbook library. If a versioned playbook exists for the category, the system instantiates it with the case-specific variables (member ID, claim number, eligibility window, etc.) and returns the response. The lookup is sub-millisecond.
- Generative fallback. If no playbook matches — either because the intent is novel or because the existing playbook is missing variables — the system falls back to RAG. The retrieval pulls the relevant policy documents; the model produces the response with citations; the response is returned.
- Telemetry. Every interaction emits structured telemetry — intent classification result, playbook hit/miss, retrieval signature, response, latency — to the capture loop's input stream.
The proportion of requests handled by approved playbooks depends on the workflow and the quality of the knowledge library. Measure coverage, exceptions and answer quality during evaluation; no customer outcome is claimed here.
The capture loop in detail
The capture loop runs every six hours on the previous window's generative responses. It does two things: cluster, and score.
Clustering can combine intent classification, response embeddings and retrieval signatures. Thresholds should be calibrated against representative, human-reviewed data before operational use.
Scoring ranks clusters by crystallization value. A cluster scores high if it (a) appears frequently — the question is recurring; (b) has high within-cluster consistency — the answer is stable across instances; and (c) contains responses that the reviewing agents accepted without modification — the AI got it right repeatedly. High-scoring clusters become candidates.
The promotion loop in detail
Promotion is the human-in-the-loop step. We deliberately keep it human. The system drafts the playbook; a domain expert at the customer agency approves it.
The draft includes the canonical response, the variable slots (filled at runtime per case), the citations to source policy, and a metadata block (version number, approving reviewer, effective date, applicable intents). The reviewer edits in a structured editor that enforces the playbook schema; a published playbook cannot be missing required fields.
Once approved, the playbook is versioned (semver, with breaking-change discipline) and deployed to the playbook library. Every interaction served by the playbook records the playbook version in the audit chain. If a regulator ever asks "what was the policy on X on date Y," the answer is in the audit log, with the playbook version, with the reviewer who approved it, with the effective date — every variable named.
The crystallization loop is what makes generative AI safe in regulated work. The model produces variety; the loop converts variety into consistency; consistency is what an auditor can trust.
Why this matters beyond Assist
The pattern generalizes. Any AI system that handles recurring questions in a domain where consistency matters — and that includes essentially every regulated CX deployment — benefits from a crystallization architecture. The specifics differ: the cluster threshold, the promotion cadence, the human-review burden. The pattern doesn't.
The instinct of most AI vendors is to push more and more traffic through the model, on the assumption that the model gets better and the answer gets better. In regulated CX, the opposite is true: the system gets better when more traffic doesn't hit the model, because the model's variance is the source of the problem. The model is the seed, not the engine.
If you're building or evaluating an AI system for CX work where consistency matters, the crystallization architecture is worth thinking through carefully. We're happy to walk through specific implementations. Get in touch →.