Validated as a self-hosted inference-serving layer for private AI scenarios where throughput, batching, OpenAI-compatible serving, and operational control matter. Outcomes depend more on the operational maturity around it than on the server itself.
Throughput, tail latency, batching behaviour, and operational ergonomics for serving a 70B-class open-weights model behind an OpenAI-compatible endpoint, including mixed prompt-length traffic and sustained concurrent load.
Two-node GPU cluster, A100 80GB, Kubernetes deployment, internal model gateway in front, traces and metrics exported to an internal observability stack.
- Strong throughput and batching characteristics for self-hosted inference
- OpenAI-compatible API surface keeps client code portable across managed and self-hosted backends
- Mature continuous batching and KV cache behaviour under realistic concurrency
- Tunable enough to land specific tail-latency and throughput targets when capacity is sized honestly
- Requires real GPU infrastructure and capacity planning; not a fit for casual local experimentation
- Operational responsibility is non-trivial — deployment, scaling, and upgrades are the operator's problem
- Model and quantization compatibility varies; not every checkpoint runs cleanly without configuration work
- Observability, quotas, and multi-tenant fairness must be assembled around it rather than expected from it
- Scaling design (replication, routing, autoscaling) needs to be made explicit rather than assumed
- Underestimating GPU capacity planning is the most common failure mode
- Sizing on demo workloads instead of real traffic shapes
- Treating an OpenAI-compatible endpoint as a drop-in for a managed provider without owning the operational surface around it
- Locking application code to vendor-specific quirks rather than the compatible subset of the API
Sits as the inference-serving layer in a private AI architecture: open-weights models behind vLLM, a thin internal gateway in front for auth, quotas, and per-tenant rate limits, retrieval and agent layers calling it through the OpenAI-compatible API. Useful where private model hosting, throughput, or cost-control experiments matter; less appropriate when a managed provider already meets the requirements.