What RAG is for
Retrieval-augmented generation separates knowledge in model parameters from documents the enterprise can currently access. The system retrieves evidence first, then asks the model to answer from that evidence and return citations. RAG does not automatically solve permissions, document quality, or conflicting facts, but it makes freshness and citation boundaries explicit.
Reference architecture
files/web -> parse/clean -> chunk -> embedding -> vector index
| |
+-> ACL/source/time metadata ---------+
v
question -> rewrite -> hybrid retrieval -> rerank -> evidence window -> answer + citations
Smaller chunks are not always better. A useful starting point is 400–800 tokens while preserving headings, paths, update times, and access-control labels. Tables, code, and FAQs often need semantic handling of their own.
Retrieval quality comes first
Create an offline evaluation set with expected documents and unacceptable answers. Track recall@k, MRR, citation coverage, and refusal accuracy. Hybrid retrieval combines lexical and vector search; a reranker then orders the top candidates.
Require the model to use only the evidence window and to refuse clearly when evidence is insufficient:
if evidence.isEmpty() or evidence.confidence < threshold:
return "There is not enough evidence to confirm this."
return answer_with_citations(evidence)
Permissions, freshness, and cost
Apply ACL filters before retrieval, never retrieve private documents and rely on a prompt to hide them. Track which chunks are affected by an update and remove documents when access is revoked. Measure embedding, reranking, and generation separately; cache query rewrites and stable evidence sets where appropriate.
Further reading
See the RAG paper, the OpenAI embeddings guide, and pgvector documentation.
