A missing audit event can make an interaction difficult to reconstruct. This article explains why traceability should be part of the architecture from the start; it is not a report of a specific customer incident.
A missing event can reveal a gap in logging coverage. Design explicit failure handling, monitor coverage and test whether required events remain reviewable.
This post describes what "audit chain by design" actually requires across an AI platform — the architecture, the engineering practices, and the specific patterns that make audit a precondition rather than a postcondition. It's a long post because the topic is unforgiving and the details matter.
What auditors actually check
Review audit evidence with the relevant internal, external or regulatory reviewer. The required records depend on the applicable obligations and the system being evaluated:
- Completeness. Does every customer-facing interaction produce a log entry? Can the system process a request that doesn't get logged?
- Tamper-evidence. If an entry is altered after the fact, can the system prove it? (Not just notice it — prove it, in a way that holds up to a second auditor.)
- Time-stamp integrity. Are timestamps from a trusted source? Are they monotonic? Can someone backdate a log entry?
- Identity binding. Does each entry name the human, service account, or system that caused it? Is that identity verifiable from outside the log?
- Retention. Are logs retained for the period the regulation requires? Are they deleted on schedule? Is deletion itself logged?
- Access control. Who can read the logs? Who can write them? Who can delete them? Are those roles separated?
- Replayability. Given a log entry, can you reconstruct what happened — including the exact prompt sent to the model, the retrieval set, and the response?
Bolt-on logging fails most of these consistently. The failures are predictable: the new code path skipped the log. The log was written but the timestamp came from the application server, which the application could fake. The log was retained but the bucket allowed object overwrites. The log named a service account but the service account was shared across nine internal users. The log captured the response but not the retrieval set, so the answer can't be reproduced.
The three architectural commitments
Audit-by-design starts with three architectural commitments that have to be made before code is written, not after.
1. Write before display
Every customer-facing response — every chatbot reply, every agent-assist suggestion, every document classification — must be written to the audit log before it is exposed to the user. Not after. Not concurrently. Before. If the audit write fails, the response is not delivered.
Writing before delivery can reduce the risk of an unrecorded response, but it introduces availability and recovery tradeoffs. Define and test the failure policy with the organization responsible for the system.
The implementation pattern: serialize the response, hash it, write the hash and the body to a tamper-evident store, wait for confirmation, then return the response. The latency cost is real — usually 30–80ms for a high-throughput log path. The latency is worth it.
2. Customer-owned sink
The audit log must live in a sink the customer owns and the vendor cannot write to outside the prescribed write path. A proposed deployment can use a customer-controlled logging sink. Write, read and administrative permissions must be designed, configured and tested for that environment.
This separation is what lets a customer's auditor trust the log. If the vendor writes to its own sink and exports a copy to the customer, the auditor has to trust that the export is faithful. If the vendor writes directly to the customer's sink, the auditor only has to trust the cloud provider's write semantics.
3. Hash chain and time anchor
A proposed tamper-evidence design can chain event hashes and periodically sign a chain head with an appropriately controlled key. Key permissions, anchor frequency and independent retention require implementation and verification.
This is overkill for most enterprise SaaS. It is the bare minimum for regulated AI. The cost is one cryptographic hash per write and one signing operation per anchor interval — negligible at any reasonable throughput.
What gets logged
The temptation is to log selectively — only the "interesting" interactions. This is the bolt-on failure mode in disguise. The audit chain logs everything customer-facing, end to end:
- The inbound request: channel, identity, timestamp, raw payload (or hash if the payload is sensitive)
- The retrieval set: which documents/chunks were retrieved, with similarity scores
- The deterministic-rule match: which playbook (and version) matched, if any
- The model invocation: model name, version, prompt template version, token counts, request fingerprint
- The model response: full text, with PII redaction tokens replaced by reversible references
- The post-processing: redaction applied, citations attached, guardrail checks run
- The delivered response: the exact bytes shown to the user
- The user action, if any: did the agent accept the suggestion, override it, edit it
Auditors do not always look at all of this. Auditors care that it is all available, in a consistent format, with a defensible chain of custody, when they need it. The cost of capturing it is small. The cost of not capturing it, in the moment an investigation actually happens, is enormous.
If you cannot reconstruct exactly what happened in any past interaction — including the model prompt and the retrieval set — you do not have an audit chain. You have a history view.
Two patterns we explicitly avoid
Two patterns appear frequently in AI platforms and both fail audit pressure. We avoid both.
Pattern: log to the application database. The application database is operational. It gets restored from backups. It gets schema-migrated. It accumulates index writes. Treating it as an audit store conflates two failure modes — operational and forensic — and the operational concerns always win in a tradeoff. The audit store has to be dedicated, append-only, and out-of-band of the application database.
Pattern: vendor-owned export. Some platforms write logs to the vendor's sink and export to the customer on a schedule. This breaks tamper-evidence: the customer's copy is downstream of the vendor's, and the vendor's copy is the canonical record. The customer's auditor is reviewing a derivative artifact. The fix is direct write to the customer-owned sink, with the vendor's observability needs handled by a separate, less-sensitive telemetry stream.
What this gets you
An audit-oriented design should help an authorized team trace an interaction to its source material, configuration and recorded review actions. Retention, access and retrieval procedures need testing in the deployed environment.
Bolt-on logging cannot do any of this. By the time you need the audit chain, it is too late to build it.
If you're working through what your AI audit posture should look like, we run architecture reviews focused specifically on this question. Schedule one here →.