← Back to learning path
Chapter · Published

RAG evaluation and citations: separate retrieval from generation

Use traceable question sets to measure recall coverage, citation accuracy, faithful answers, and justified refusal.

Source: knowledge/p24-b/collection_6e8fbe1e8b1ea569a539e8a5/tutorials/rag-evaluation-and-citations.mdRAG knowledge engineering
Chapter progress 0%Learning path progress 0% · 0/10 chapters completedContinue learning path →

RAG failures can come from missing results, a stale document version, truncated context, or a model that ignores citation rules. Evaluate retrieval and generation separately so the repair targets the right layer.

Core concepts

Build a question set with expected document IDs. Record retrieval hits, citation coverage, answer faithfulness, and whether refusal is appropriate. When the corpus has no evidence, an explicit lack-of-evidence result is correct.

Runnable example

def citation_coverage(answer: str, expected_sources: set[str]) -> float:
    cited = {token.strip('[]()') for token in answer.split() if token.startswith('[') and token.endswith(']')}
    return len(cited & expected_sources) / len(expected_sources) if expected_sources else 1.0

assert citation_coverage('see [doc-1]', {'doc-1'}) == 1.0

Steps and use cases

  1. Define expected document IDs and unacceptable answer ranges.
  2. Run retrieval first and record top-k before running generation.
  3. Classify no-hit, wrong-version, missing-citation, and unauthorized-candidate failures.
  4. Add failures to the next regression set instead of hiding a retrieval problem with a prompt change.

This fits compliance assistants, internal manuals, and evidence-based research tools.

Official references

OpenAI Evals and OpenAI Retrieval.

Last checked: 2026-11-16. Citation behavior and retrieval APIs can change; follow the linked documentation.

Code examples

def citation_coverage(answer: str, expected_sources: set[str]) -> float:
    cited = {token.strip('[]()') for token in answer.split() if token.startswith('[') and token.endswith(']')}
    return len(cited & expected_sources) / len(expected_sources) if expected_sources else 1.0

assert citation_coverage('见 [doc-1]', {'doc-1'}) == 1.0

Related practical cases

KnowledgeCase study: replaying RAG retrieval regression with a fixed question set

Further reading

KnowledgeRAG for Enterprise Knowledge: Architecture and PracticeKnowledgeDesigning Reliable AI Automation WorkflowsKnowledgeProduction Prompt Engineering: From Instructions to Measurable Contracts
Product Hub Demo · V0.3