Measure the service target
Break latency into queueing, preprocessing, model execution, and postprocessing. Record time to first token, per-token latency, throughput, concurrency, and memory use. Without these measures, changing hardware or quantization is guesswork.
Continuous batching can improve throughput while increasing wait time. Give interactive and batch work separate queues and budgets, with a maximum wait to protect tail latency. Streaming responses still need a first-token timeout and an overall deadline.
Pin software and hardware
Record model weights, tokenizer, CUDA, driver, inference engine, and quantization. Generate a software inventory with the image and run a fixed prompt regression before deployment. Starting successfully is not proof of compatibility.
Design for failure and rollback
Health checks should distinguish process liveness, model readiness, and real inference. Expose clear states for out-of-memory, load failure, and upstream timeout. Limit queue and request size before scaling. Keep the last known-good model so a rollout can be reversed quickly.
Sources
See the vLLM documentation, NVIDIA Triton documentation, and KServe documentation.
