← Back to learning path
Case · Published

Case study: replaying RAG retrieval regression with a fixed question set

Use source ids, permission tags, and expected evidence to separate retrieval regressions from generation problems.

Source: knowledge/p27/rag-engineering/cases/rag-regression-replay.mdRAG knowledge engineering
Chapter progress 0%Learning path progress 0% · 0/10 chapters completedContinue learning path →

Problem and boundaries

Answer quality drops after a knowledge-base update, but the cause is unclear. This case uses fixed documents and questions to compare authorized retrieval only; it does not connect to a vector service or external model.

Design

Each question includes a tenant, allowed roles, and expected document ids. Replay filters permissions first and then compares retrieved sets; the generation layer receives only evidence that passed the assertions.

Minimal implementation

fixtures = [
    {'query': '退款', 'role': 'support', 'expected': {'refund-1'}},
    {'query': '工资单', 'role': 'support', 'expected': set()},
]
docs = {'refund-1': ('support', '退款需要订单号'), 'payroll-1': ('finance', '工资单仅限财务查看')}
for item in fixtures:
    found = {doc_id for doc_id, (role, body) in docs.items() if item['query'] in body and role == item['role']}
    assert found == item['expected']

Reproduction steps

  1. Create a redacted question set and expected source ids, recording the data version.
  2. Run permission filtering and retrieval against old and new indexes, saving set differences.
  3. Classify missing results, wrong versions, and unauthorized candidates instead of hiding retrieval differences behind answer text.
  4. Replay the same question set after the index fix and add failed samples to a versioned regression set.

Verification checklist

Verify tenant isolation, no-result behavior, document deletion, version switching, and stable source ids.

Common questions

Why not accept one model answer? Text output cannot identify whether retrieval or generation caused the problem.

Can test data contain production documents? Use redacted or synthetic data and record its version.

Further reading

Official references

OpenAI Retrieval; SQLite FTS5.

Last checked: 2026-11-17. Follow the linked sources for current interfaces, model behavior, and pricing; this case does not present one local run as a general performance result.

Further reading

KnowledgeRAG for Enterprise Knowledge: Architecture and PracticeKnowledgeProduction Prompt Engineering: From Instructions to Measurable ContractsKnowledgeAI Agents: From Core Concepts to Production
Product Hub Demo · V0.3