Inference throughput: SGLang and vLLM on A6000 GPUs
A post published on August 4, 2024 reports higher throughput with SGLang than with vLLM on four A6000 GPUs, but similar performance with smaller batches.
Source: Comparação de throughput entre SGLang e vLLM em GPUs A6000 (x.com). Text prepared with AI from this source.
What happened and what to do
In a post dated August 4, 2024, an engineer reports replacing vLLM with SGLang in a local data-generation pipeline. With Mistral Large 2 in 4-bit precision running on four A6000 GPUs, the reported throughput was about 1,100 tokens per second with SGLang, versus less than 750 with vLLM. Performance was similar with smaller batches. The account illustrates how the serving engine and batch size can affect throughput on the same hardware and model.
A company can measure these variables through a controlled evaluation on its own infrastructure, using representative workloads and metrics for latency, throughput, cost, and GPU utilization. The findings can inform serving configuration and capacity planning, without assuming that one option is best for every usage pattern.
How the consultancy can help
Wendelmaques can assess the inference workload and infrastructure, define a comparative evaluation with suitable criteria and metrics, and implement serving and monitoring. The scope can include ongoing operation and adjustments as workloads change.
Next step
Send a short description of your inference workload and available infrastructure to receive a proposal with a defined scope.
Let's assess your project
I reply with scope and a proposal.