RAG failures can come from missing results, a stale document version, truncated context, or a model that ignores citation rules. Evaluate retrieval and generation separately so the repair targets the right layer.
Core concepts
Build a question set with expected document IDs. Record retrieval hits, citation coverage, answer faithfulness, and whether refusal is appropriate. When the corpus has no evidence, an explicit lack-of-evidence result is correct.
Runnable example
def citation_coverage(answer: str, expected_sources: set[str]) -> float:
cited = {token.strip('[]()') for token in answer.split() if token.startswith('[') and token.endswith(']')}
return len(cited & expected_sources) / len(expected_sources) if expected_sources else 1.0
assert citation_coverage('see [doc-1]', {'doc-1'}) == 1.0
Steps and use cases
- Define expected document IDs and unacceptable answer ranges.
- Run retrieval first and record top-k before running generation.
- Classify no-hit, wrong-version, missing-citation, and unauthorized-candidate failures.
- Add failures to the next regression set instead of hiding a retrieval problem with a prompt change.
This fits compliance assistants, internal manuals, and evidence-based research tools.
Official references
OpenAI Evals and OpenAI Retrieval.
Last checked: 2026-11-16. Citation behavior and retrieval APIs can change; follow the linked documentation.
Code examples
def citation_coverage(answer: str, expected_sources: set[str]) -> float:
cited = {token.strip('[]()') for token in answer.split() if token.startswith('[') and token.endswith(']')}
return len(cited & expected_sources) / len(expected_sources) if expected_sources else 1.0
assert citation_coverage('见 [doc-1]', {'doc-1'}) == 1.0
