RAG uses retrieved external material as generation context. A reliable system first makes material traceable, updateable, and deletable before tuning vector models or prompts.
Core concepts
Every document should have a stable id, source, version, update time, tenant, and access tags. Keep the heading path and document id in every chunk so answers can cite the original and handle incremental updates.
from dataclasses import dataclass
@dataclass(frozen=True)
class Chunk:
document_id: str
version: str
ordinal: int
text: str
access_tags: tuple[str, ...]
def chunk_id(chunk: Chunk) -> str:
return f'{chunk.document_id}:{chunk.version}:{chunk.ordinal}'
Steps
- Define allowed file types, maximum size, and a source allowlist.
- Parse heading levels and split at semantic boundaries, preserving document version and chunk ordinal.
- Filter empty text and sensitive fields before generating embeddings.
- Preserve source locations. Remove or replace chunks by version when a document changes.
When to use it
It suits product manuals, internal procedures, and public technical documentation. Do not copy real-time transaction data into long-lived knowledge chunks.
Common questions
Is a larger chunk always better? No. Choose size with retrieval tasks, model context, and citation needs together.
How should updates work? Write the new version, switch the index pointer only after verification, and then clean the old version.
Do scanned PDFs require OCR? When needed, record OCR quality separately; do not treat empty text as valid knowledge.
Official references
OpenAI Embeddings; OpenAI Retrieval.
Last checked: 2026-11-16. Use the linked official documentation for current API behavior.
