How to design retrieval-augmented generation around source quality, permissions, evaluation, and operations rather than a single retrieval component.
RAG is a content system before it is a model pattern
A retrieval-augmented generation prototype can be assembled quickly: split documents, create embeddings, retrieve similar chunks, and place them in a prompt. Production exposes everything the prototype left implicit. Documents conflict, access changes, tables lose meaning when split, old policies outrank new ones, and users ask questions whose answer does not exist. The vector store is one component in a larger publishing and evidence system.
The useful design question is not “Which database should we use?” It is “How does approved knowledge become searchable evidence, remain fresh, respect the current user, and support an answer that can be checked?” That framing brings content owners, identity, lifecycle, and evaluation into the architecture early enough to influence it.
Prepare sources for retrieval, not just ingestion
Source quality limits answer quality. Assign an owner, status, effective date, and audience to each knowledge source where possible. Preserve headings, document lineage, and relationships between prose, tables, images, and attachments. A chunk saying “this limit applies only to tier two” is useless if the heading and referenced definition disappeared during extraction.
- Normalize common formats while retaining source URL, version, section path, and access metadata.
- Chunk along semantic boundaries, then test the chunks people actually retrieve instead of choosing a universal token size.
- Represent canonicality and effective dates so a superseded policy does not compete equally with its replacement.
- Detect deletions and permission changes as first-class events, not only additions and edits.
- Quarantine extraction failures rather than silently publishing empty or malformed content.
Treat retrieval as an inspectable pipeline
Semantic similarity is helpful, but it does not understand every business distinction. Production retrieval may combine lexical search for exact identifiers, vector search for meaning, metadata filters for scope, and reranking for relevance. Query rewriting can help with conversational shorthand, but it must not broaden the request beyond the user’s authority or turn an ambiguous question into an unjustified assumption.
- Classify the request
Identify intent, entity hints, time scope, and whether the question should be answered from governed sources at all.
- Apply identity and metadata filters
Narrow the candidate set using current permissions, tenant boundaries, source state, locale, and effective date.
- Retrieve through complementary methods
Use exact and semantic signals where each adds value, then retain scores and provenance for debugging.
- Rerank and assemble context
Prefer diverse, authoritative evidence and fit complete ideas into the context budget rather than blindly taking the first chunks.
- Answer or abstain
Require the response to stay within retrieved evidence and make insufficient or conflicting support visible to the user.
Permissions and citations belong in the correctness model
Post-filtering a generated answer is too late. Retrieval should operate on the requesting identity or on an equivalently constrained permission snapshot. Cache keys must include the authorization context, and revocation needs a defined propagation path. Multi-tenant systems should prefer explicit tenant constraints over hoping embedding similarity never crosses a boundary.
Citations help users inspect an answer, but a link alone is not proof. Preserve the exact supporting passage and source version used at generation time, while still linking to the live source when appropriate. The interface should distinguish supported claims, synthesis across sources, and model-added suggestions. If evidence conflicts, surface the disagreement instead of letting prose smooth it away.
Measure each stage and operate the whole loop
Evaluate retrieval and generation separately. For a set of representative questions, label relevant sources and check whether they appear in the candidate and final context. Then evaluate whether the answer is supported, complete enough, appropriately cautious, and correctly cited. End-to-end preference scores alone cannot tell you whether a miss came from ingestion, permissions, retrieval, context assembly, or generation.
Production telemetry should include indexing lag, extraction failures, empty-result rate, retrieval composition, answer abstention, source opens, user corrections, and cost and latency by stage. Sample traces for qualitative review with privacy-aware logging. A RAG system stays useful through steady content operations: fixing sources, adding hard evaluation cases, and making failure modes visible. The model will change; that operating loop is the durable capability.