Separate the signals
Likes, rewrites, follow-up questions, abandonment, and human corrections mean different things. Record events by task, locale, model version, and content source. A click is not proof that an answer was correct. Keep feedback decoupled from identity and store only the fields needed to improve the product.
Route feedback to a queue
Send low-confidence results, user corrections, and high-risk actions to human review. Classify issues as missing retrieval, factual error, formatting error, authorization error, or product experience. Each class needs a different fix; not every problem is a prompt problem.
Compare versions
Turn reviewed feedback into a regression set and compare models, retrievers, and prompts. Record sample coverage and known limits at release. If quality drops, roll back quickly. Unreviewed user text must not be published, trained on, or promoted.
Sources
See NIST AI RMF, OpenTelemetry Logs, and the Google People + AI Guidebook.
