← Back to learning path
Chapter · Published

RAG foundations: documents, chunking, and data contracts

Start with a traceable document pipeline and define content, version, permission, and deletion semantics.

Source: knowledge/p24-b/collection_6e8fbe1e8b1ea569a539e8a5/tutorials/rag-foundations-and-data-contracts.mdRAG knowledge engineering
Chapter progress 0%Learning path progress 0% · 0/10 chapters completedContinue learning path →

RAG uses retrieved external material as generation context. A reliable system first makes material traceable, updateable, and deletable before tuning vector models or prompts.

Core concepts

Every document should have a stable id, source, version, update time, tenant, and access tags. Keep the heading path and document id in every chunk so answers can cite the original and handle incremental updates.

from dataclasses import dataclass

@dataclass(frozen=True)
class Chunk:
    document_id: str
    version: str
    ordinal: int
    text: str
    access_tags: tuple[str, ...]

def chunk_id(chunk: Chunk) -> str:
    return f'{chunk.document_id}:{chunk.version}:{chunk.ordinal}'

Steps

  1. Define allowed file types, maximum size, and a source allowlist.
  2. Parse heading levels and split at semantic boundaries, preserving document version and chunk ordinal.
  3. Filter empty text and sensitive fields before generating embeddings.
  4. Preserve source locations. Remove or replace chunks by version when a document changes.

When to use it

It suits product manuals, internal procedures, and public technical documentation. Do not copy real-time transaction data into long-lived knowledge chunks.

Common questions

Is a larger chunk always better? No. Choose size with retrieval tasks, model context, and citation needs together.

How should updates work? Write the new version, switch the index pointer only after verification, and then clean the old version.

Do scanned PDFs require OCR? When needed, record OCR quality separately; do not treat empty text as valid knowledge.

Official references

OpenAI Embeddings; OpenAI Retrieval.

Last checked: 2026-11-16. Use the linked official documentation for current API behavior.

Related practical cases

KnowledgeCase study: blue-green RAG index release and rollbackKnowledgeCase study: replaying RAG retrieval regression with a fixed question set

Further reading

KnowledgeRAG for Enterprise Knowledge: Architecture and PracticeKnowledgeDesigning Reliable AI Automation WorkflowsKnowledgeAI Agents: From Core Concepts to Production
Product Hub Demo · V0.3