An open-source engine for high-throughput LLM inference and serving, with an OpenAI-compatible API and continuous batching. It suits teams with GPU operations capability that need multi-model or high-concurrency inference services.
AvailableAILLMInference servingGPU
Source overview
An open-source engine for high-throughput LLM inference and serving, with an OpenAI-compatible API and continuous batching. It suits teams with GPU operations capability that need multi-model or high-concurrency inference services.
The upstream project does not provide a product interface screenshot that can be embedded here. It is primarily used through APIs, the command line, or an SDK; see examples in the official repository and documentation.