RAG (Retrieval-Augmented Generation) Architecture

RAG grounds the model in data you control

RAG is the default GenAI design because it solves two problems an LLM cannot solve alone: it does not know your private data, and it hallucinates confidently when it does not know something. RAG grounds the model by retrieving relevant passages from your own corpus at query time and stuffing them into the prompt with instructions to answer only from that context and to cite it. The model becomes a reasoning-and-phrasing engine over evidence you control, not an oracle. There are two halves: an offline ingestion pipeline and an online query path.

Ingestion (offline)

You parse each source document (PDF, HTML, Confluence, tickets) into clean text, then chunk it. Chunking is where naive systems die. A chunk that is too large dilutes the embedding and wastes context budget; too small and you shred the meaning across boundaries. A common baseline is 300 to 800 tokens with 10 to 20 percent overlap so a sentence split across a boundary survives in one chunk. Better is semantic or structure-aware chunking that respects headings, tables, and paragraphs. Each chunk gets an embedding (from a model like text-embedding-3-large or an open model like bge) and is written to a vector index alongside metadata: source id, title, ACL groups, timestamp, section. When a document changes you re-embed only the affected chunks; you do not rebuild the whole index. Deletes must propagate or you serve stale, retracted content.

Query path (online)

Order of evaluation

1. Embed queryone embedding call, and part of your latency budget
2. Hybrid retrievedense vector top-100 unioned with sparse BM25 top-100
3. Rerankcross-encoder reads query and chunk together, keep top-8
4. ACL filterdrop chunks this user may not see
5. Assemble contextdedup, budget to the window, add citation markers
6. Generateanswer only from context, cite sources, else say I do not know
7. Post-checkverify each cited claim maps to a retrieved chunk

Why a reranker and hybrid retrieval are mandatory, not optional. Dense vector search captures meaning but misses exact terms, error codes, product names, and rare acronyms. BM25 nails exact matches but misses paraphrase. Hybrid runs both and unions the candidates. Then the reranker matters because embedding similarity is a coarse first-stage filter: the vector top-20 is full of plausible-but-wrong chunks. A cross-encoder reranker reads the query and each chunk together and produces a far sharper relevance score, so the 8 chunks you actually put in the prompt are the right 8. Skipping the reranker is the single most common reason a demo RAG feels dumb in production.

Access control at retrieval time

You never filter after generation, because the model has already seen forbidden text. You attach the user's group memberships to the query and filter candidates by the ACL metadata on each chunk before assembly, ideally as a pre-filter inside the vector query so you do not retrieve what the user cannot read. Retrieval is the security boundary.

Instruct the model to say "I do not know" when context is weak, and verify citations by checking each cited claim resolves to a retrieved chunk. Measure the RAG triad: context relevance (did retrieval fetch the right chunks), faithfulness (is the answer supported by context), answer relevance (did it address the question). Without this triad you cannot tell a retrieval bug from a generation bug.

Interview nuance: when latency is probed, note that the reranker and embedding calls are the cost, not the vector search. Cache embeddings for repeated queries, run rerank on a small candidate set, and stream the answer so time-to-first-token hides generation latency.

Recap: RAG is ingestion (parse, chunk, embed, index with ACL metadata) plus a query path of hybrid retrieval, a mandatory reranker, ACL-filtered context assembly, grounded generation with citations, and the RAG triad for eval.