Use three evaluation layers
Offline evaluation validates a fixed sample set. Online evaluation observes the real distribution. Human review handles high-risk and disputed cases. None replaces the others. Offline samples should include common tasks, edge inputs, refusals, and tool errors.
Track factual accuracy, citation coverage, instruction following, refusal accuracy, tool success, time to first token, and cost in addition to answer similarity. For subjective work, use a clear rubric and double-sampled review instead of relying only on another model as a judge.
Connect events end to end
Assign a trace ID to each request and link retrieval, model calls, tool steps, and the final result. Keep only necessary summaries and mask sensitive fields. Slice metrics by model, prompt, retriever version, and locale to find regressions.
Define release gates
Set minimum pass rates and maximum latency before release. If production anomalies persist, degrade to a smaller model, a static answer, or a human queue. Feed reviewed failures back into evaluation, never straight into training.
Sources
See OpenTelemetry, NIST AI RMF, and MLflow Evaluation.
