Start with the retrieval task
The largest embedding model is not automatically the best choice. Define languages, document types, query length, latency budget, and multilingual needs. Build a set with positive matches, close distractors, and unanswerable questions before comparing models.
Design chunks and indexes together
Keep titles, paths, timestamps, and access labels with each chunk. Large chunks dilute the useful passage; tiny chunks lose context. Retain keyword search beside the vector index so exact names, identifiers, and short queries are covered by hybrid retrieval.
A compact evaluation flow is:
query -> top-k retrieval -> recall@k / MRR
-> reranking -> citation coverage and refusal accuracy
Normalize vectors consistently with the similarity metric. For an index upgrade, dual-write or rebuild offline, then compare hit rate and latency before switching the read path.
Authorization and deletion are retrieval concerns
Filter by ACL before retrieval. A prompt cannot repair a permission leak after recall. Remove vectors and caches when a document is revoked or deleted. Preserve source URL, version, and timestamp so answers can be cited and checked.
Sources
See Sentence Transformers documentation, Faiss documentation, and the original RAG paper.
