Problem and boundaries
Answer quality drops after a knowledge-base update, but the cause is unclear. This case uses fixed documents and questions to compare authorized retrieval only; it does not connect to a vector service or external model.
Design
Each question includes a tenant, allowed roles, and expected document ids. Replay filters permissions first and then compares retrieved sets; the generation layer receives only evidence that passed the assertions.
Minimal implementation
fixtures = [
{'query': '退款', 'role': 'support', 'expected': {'refund-1'}},
{'query': '工资单', 'role': 'support', 'expected': set()},
]
docs = {'refund-1': ('support', '退款需要订单号'), 'payroll-1': ('finance', '工资单仅限财务查看')}
for item in fixtures:
found = {doc_id for doc_id, (role, body) in docs.items() if item['query'] in body and role == item['role']}
assert found == item['expected']
Reproduction steps
- Create a redacted question set and expected source ids, recording the data version.
- Run permission filtering and retrieval against old and new indexes, saving set differences.
- Classify missing results, wrong versions, and unauthorized candidates instead of hiding retrieval differences behind answer text.
- Replay the same question set after the index fix and add failed samples to a versioned regression set.
Verification checklist
Verify tenant isolation, no-result behavior, document deletion, version switching, and stable source ids.
Common questions
Why not accept one model answer? Text output cannot identify whether retrieval or generation caused the problem.
Can test data contain production documents? Use redacted or synthetic data and record its version.
Further reading
Official references
OpenAI Retrieval; SQLite FTS5.
Last checked: 2026-11-17. Follow the linked sources for current interfaces, model behavior, and pricing; this case does not present one local run as a general performance result.
