Normalize inputs before inference
Define MIME allowlists, size limits, transcoding, and metadata for images, audio, and text. Do not pass arbitrary public files directly to a model. Store source, timestamp, and checksum during parsing for audit and deduplication.
A maintainable pipeline is:
upload -> security scan -> preprocess -> modality encoding -> fusion/retrieval -> generation -> quality check
Use modality-specific preprocessing: normalize image size and orientation, record audio sample rate and speakers, and keep page numbers and layout coordinates for PDFs. Cross-modal citations should point to the original fragment rather than only returning rewritten text.
Control cost and latency
Classify with a lightweight model before routing hard cases to a larger one. Cache immutable transcoding and embeddings; process long media in segments and asynchronous jobs. Show processing, partial completion, and retry states instead of pretending a long task is synchronous.
Quality and privacy
Keep separate samples for OCR, transcription, and image description. Redact or restrict faces, voices, and personal information as the use case requires. Deleting a source should also remove derived thumbnails, transcripts, and vectors.
Sources
See W3C Media Types, the OpenAI vision guide, and the NIST Privacy Framework.
