vLLM, SGLang, or TensorRT-LLM? Picking an LLM Serving Stack

Introduction

LLM inference is no longer just about loading a model and calling generate(). That works for a single request, but the moment we move toward real serving workloads, the problem becomes much more interesting.

Now the system has to handle many requests at the same time. Some requests have short prompts, some have long prompts, some are in the prefill phase, and some are already decoding tokens. At the same time, the GPU has limited memory, the KV cache keeps growing, and latency can quickly become unstable if scheduling is not handled properly.

This is where inference frameworks like vLLM, SGLang, and TensorRT-LLM become important.

All three frameworks are trying to solve the same high-level problem: how do we serve LLMs with high throughput, low latency, efficient GPU memory usage, and stable performance under concurrency? But they approach the problem differently.

vLLM focuses on easy and efficient LLM serving with features like PagedAttention, continuous batching, prefix caching, quantization, and distributed inference.

SGLang focuses on fast serving, efficient prefix reuse, structured generation, and runtime-level optimizations such as RadixAttention.

TensorRT-LLM focuses on NVIDIA GPU-specific optimization, where models are converted and compiled into optimized TensorRT engines for high-performance inference.

In this blog, we compare these three frameworks from an engineer's perspective. The goal is not just to find which framework gives the highest throughput number. The goal is to understand how each framework behaves across different models, datasets, concurrency levels, and GPU setups. For benchmarking, we use dense and MoE models, ShareGPT and RULER 16K datasets, and concurrency sweeps across vLLM, SGLang, and TensorRT-LLM on H100 GPUs.

Why This Comparison Matters for Engineers?

Running an LLM locally and serving an LLM in production are very different problems.

For a single request, the flow is simple: tokenize the prompt, run prefill, decode tokens one by one, and return the response. But under real traffic, many requests arrive together. Each request can have a different prompt length, output length, and memory requirement. This creates multiple engineering challenges around batching, GPU memory usage, KV cache management, long-prompt handling, prefix reuse, and stable latency under high concurrency.

These challenges matter because LLM inference has two very different phases. During prefill, the model processes the input prompt. This phase is usually compute-heavy because many prompt tokens can be processed in parallel. During decode, the model generates one token at a time. This phase is often memory-bandwidth-bound because every new token requires reading model weights and attending over the existing KV cache.

So a good inference framework is not only about fast kernels. It also needs a strong scheduler, efficient KV cache management, good batching behavior, prefix caching, and stable memory usage. This is why comparing vLLM, SGLang, and TensorRT-LLM is useful. These frameworks make different design choices, and those choices directly affect real metrics like TTFT, TPOT, ITL, and throughput.

For engineers, the practical question is: which framework gives the best latency-throughput tradeoff for a given model, dataset, hardware setup, and serving workload? That is the main question this benchmark tries to answer.

vLLM

vLLM is one of the most widely used open-source frameworks for LLM inference and serving. It is designed to make LLM serving fast, memory-efficient, and easy to use. The main idea behind vLLM is that serving performance is not only limited by model execution. It is also heavily affected by how requests are scheduled and how KV cache memory is managed during generation, every request needs KV cache memory to store key and value tensors from previous tokens. If this memory is managed poorly, GPU memory gets wasted, fragmentation increases, and the system can serve fewer concurrent requests.

vLLM addresses this problem with PagedAttention. Instead of storing KV cache as one large contiguous memory region per request, vLLM divides KV cache into smaller blocks. This makes memory allocation more flexible and allows the runtime to serve many requests with different sequence lengths more efficiently.

Another important feature is continuous batching. In traditional batching, the system waits for a group of requests, processes them together, and only then moves to the next batch. But LLM requests do not finish at the same time. Some generate short outputs, while others continue decoding for many more tokens.

With continuous batching, vLLM can add new requests into the running batch as older requests finish. This helps keep the GPU busy and improves throughput under concurrent workloads.

vLLM provides:

• PagedAttention for efficient KV cache memory management
• Continuous batching for better GPU utilization
• Prefix caching to reuse shared prompt prefixes
• Chunked prefill to avoid long prompts blocking decode
• CUDA graph support for reducing decode overhead
• Quantization support such as GPTQ, AWQ, INT8, and FP8
• Distributed inference using tensor, pipeline, data, and expert parallelism
• OpenAI-compatible API server for easy deployment

We have already covered many of these vLLM techniques in detail in previous blogs, including speculative decoding, quantization, distributed inference, expert parallelism, and practical optimization methods.

For engineers, vLLM is usually the easiest framework to start with. It has strong Hugging Face model support, simple serving commands, good production features, and a large community around it. In this benchmark, vLLM acts as a strong general-purpose baseline. It represents the serving-first approach: easy deployment, efficient memory management, and strong performance across a wide range of workloads.

SGLang

SGLang is a high-performance framework for serving large language models and vision-language models. Like vLLM, it is designed for fast inference, high throughput, and production-style serving. But SGLang puts a lot of focus on runtime efficiency, prefix reuse, and structured generation.

One of the most important ideas in SGLang is RadixAttention. In many real workloads, requests often share common prefixes. For example, multiple requests may use the same system prompt, same few-shot examples, same RAG template, or same long instruction prefix.

Instead of recomputing the KV cache for the same prefix again and again, SGLang stores and reuses shared prefixes using a radix-tree-based cache. This can reduce prefill cost when many requests have overlapping prompt prefixes.

SGLang provides:

• RadixAttention for efficient prefix caching
• Continuous batching for high-throughput serving
• Chunked prefill for better scheduling of long prompts
• Speculative decoding support
• Quantization support
• Tensor parallelism and other distributed inference options
• Structured generation and constrained decoding support
• OpenAI-compatible serving API

The key difference is that SGLang is not only trying to run model forward passes faster. It also tries to optimize the full serving runtime around how requests are structured and how much computation can be reused.

This makes SGLang especially interesting for workloads where prompts share common structure. Examples include chat templates, RAG pipelines, agent workflows, code assistants, and workloads with large repeated system prompts. In this benchmark, SGLang is compared against vLLM and TensorRT-LLM to understand how its runtime design performs across general chat-style workloads and long-context workloads.

TensorRT-LLM

TensorRT-LLM is NVIDIAs framework for optimizing and serving large language models on NVIDIA GPUs. Compared to vLLM and SGLang, TensorRT-LLM follows a more engine-centric approach. With vLLM and SGLang, we can usually start serving a Hugging Face model directly with fewer steps. TensorRT-LLM is different. For best performance, the model is usually converted and built into an optimized TensorRT engine before serving.

A simplified workflow looks like this:

Hugging Face model
    |
Checkpoint conversion
    |
TensorRT-LLM engine build
    |
TensorRT-LLM runtime / trtllm-serve
    |
Optimized inference on NVIDIA GPUs

This engine-building step allows TensorRT-LLM to apply NVIDIA-specific optimizations around kernels, memory layout, attention, batching, and execution planning. It can optimize the model for a specific dtype, maximum batch size, maximum input length, maximum sequence length, tensor parallel size, and target GPU architecture.

TensorRT-LLM provides:

• TensorRT engine-based inference
• Optimized NVIDIA GPU kernels
• In-flight batching
• KV cache management
• Quantization support such as FP8, INT8, and weight-only quantization
• Tensor parallelism and pipeline parallelism
• Speculative decoding support
• Multi-GPU and multi-node deployment options
• Python and C++ runtime support
• trtllm-serve for OpenAI-compatible serving

The main advantage of TensorRT-LLM is that it is deeply optimized for NVIDIA hardware. When the model, engine configuration, and workload shape are well aligned, it can deliver strong performance.

The tradeoff is operational complexity. TensorRT-LLM usually requires more setup than vLLM or SGLang. Engine build parameters matter a lot. If max_input_len, max_seq_len, max_batch_size, or max_num_tokens are not selected carefully, the engine may not support the target workload efficiently.

This is especially important for benchmarking. TensorRT-LLM should not be treated as a simple drop-in replacement where default settings always give the best result. It needs careful engine building, correct runtime configuration, and workload-aware tuning.

In this benchmark, TensorRT-LLM represents the NVIDIA-optimized path. The goal is to compare how much benefit we get from this engine-based approach against the simpler serving-first approach of vLLM and SGLang.

Difference Between vLLM, SGLang, and TensorRT-LLM

vLLM, SGLang, and TensorRT-LLM are all built for high-performance LLM inference, but they make different design choices.

vLLM is a serving-first framework. It focuses on making LLM serving easy, scalable, and memory-efficient. Its biggest strengths are PagedAttention, continuous batching, broad model support, and simple OpenAI-compatible deployment.

SGLang is also serving-first, but it puts stronger focus on runtime-level optimizations such as RadixAttention and efficient prefix reuse. This makes it especially interesting for workloads where many requests share common prefixes, such as chat templates, RAG pipelines, and agent workflows.

TensorRT-LLM follows an engine-first approach. Instead of directly serving a Hugging Face model, the model is usually converted and compiled into an optimized TensorRT engine. This gives more room for NVIDIA-specific optimizations, but also makes setup and tuning more complex.

vLLM is usually the easiest strong baseline, SGLang is very interesting when prefix reuse and runtime flexibility matter, and TensorRT-LLM is useful when we want deeper NVIDIA-specific optimization and are ready to spend more time on engine building and tuning.

Benchmarking Setup

To compare vLLM, SGLang, and TensorRT-LLM, we benchmarked all three frameworks using the same models, datasets, hardware, and concurrency sweep.

The goal of this benchmark is not only to check peak throughput. We also want to understand how each framework behaves as concurrency increases and how stable the latency-throughput curve looks under pressure.

Hardware

To run the experiments in this blog post you can rent a GPU at JarvisLabs.

Here's how to set up your own instance:

1. Log in: JarvisLabs and navigate to the dashboard.
2. Create an instance: click Create and select your desired GPU configuration.
3. Select your GPU: choose H100 80GB to match the hardware used in this article.
4. Choose the framework: select PyTorch from the available frameworks.
5. Launch: click Launch. Your instance will be ready in a few minutes.

For Qwen/Qwen2.5-7B-Instruct, we used 1xH100. For the other two models, we used 2xH100 GPUs. Within each model, all frameworks were compared on the same GPU count.

Models

We selected three Qwen models to cover a small dense model, a larger MoE model, and a larger dense model:

Datasets

We used two datasets:

ShareGPT helps measure normal serving behavior, while RULER 16K puts more pressure on long prompt processing and KV cache management.

Dataset Preparation

Download the ShareGPT file used by the benchmark client commands:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

For the long-context benchmark, we used lighteval/RULER-16384-Qwen2.5-Instruct, a RULER 16K dataset variant. In the rest of the blog, this workload is referred to as RULER 16K. We converted it into the custom JSONL format expected by vllm bench serve:

from datasets import load_dataset, concatenate_datasets
from transformers import AutoTokenizer
import json
from pathlib import Path

DATASET = "lighteval/RULER-16384-Qwen2.5-Instruct"
MODEL = "Qwen/Qwen2.5-7B-Instruct"  # Replace the respective model or keep it as is. all the models in this benchmarks are from same family

OUT = Path("ruler_16384_1000_vllm.jsonl")
N_SAMPLES = 1000

tokenizer = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True)

splits = [
    "vt",
    "cwe",
    "fwe",
    "niah_single_1",
    "niah_single_2",
    "niah_single_3",
    "niah_multikey_1",
    "niah_multikey_2",
    "niah_multivalue",
    "niah_multiquery",
    "qa_1",
    "qa_2",
]

datasets = []

for split in splits:
    ds = load_dataset(DATASET, split=split)
    datasets.append(ds)

full_ds = concatenate_datasets(datasets)
full_ds = full_ds.shuffle(seed=42)
subset = full_ds.select(range(N_SAMPLES))

with OUT.open("w", encoding="utf-8") as f:
    for row in subset:
        answer = row["outputs"]

        # outputs may be string or list depending on task
        if isinstance(answer, list):
            answer_text = " ".join(map(str, answer))
        else:
            answer_text = str(answer)

        output_tokens = len(
            tokenizer.encode(answer_text, add_special_tokens=False)
        )

        item = {
            "prompt": row["input"],
            "output_tokens": max(output_tokens, 1),
        }

        f.write(json.dumps(item, ensure_ascii=False) + "\n")

print(f"Saved {len(subset)} samples to {OUT}")

Concurrency Sweep

For each model and framework, we used the following concurrency sweep pattern:

60, 120, 180, 240, 300, 360, 420, 480, 540, 600, no_cap

In the tables, no_cap means the benchmark client sends requests as fast as possible and lets each server queue and schedule them in its own way. This shows how each framework behaves when it is pushed to saturation.

Benchmark Methodology

For ShareGPT, all runs used --ignore-eos --output-len 256 to fix the output length at 256 tokens per request. This removes output-length variance as a variable and isolates scheduling, prefill, and decode behavior across frameworks.

For RULER 16K, the benchmark used the dataset's long-context prompts (~16K input tokens) with the default output length (256 tokens). This tests long-context prefill and scheduling behavior with a uniform decode phase across all frameworks.

We start with Qwen2.5-7B-Instruct.

Qwen2.5-7B-Instruct

ShareGPT

Server Commands

vLLM

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.90

SGLang

python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 \
  --port 30000 \
  --dtype bfloat16

TensorRT-LLM

Step 1: create the BF16 config.

cat > trtllm_bf16_config.yaml <<EOF
dtype: bfloat16
EOF

Step 2: convert the Hugging Face checkpoint.

python /home/TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py \
  --model_dir /home/Qwen/Qwen2.5-7B-Instruct \
  --output_dir /home/qwen25_7b_checkpoint \
  --dtype bfloat16 \
  --tp_size 1

Step 3: build the ShareGPT engine.

trtllm-build \
  --checkpoint_dir ./qwen25_7b_checkpoint \
  --output_dir ./qwen25_7b_trt_engine_sharegpt \
  --gemm_plugin bfloat16 \
  --gpt_attention_plugin bfloat16 \
  --max_batch_size 256 \
  --max_input_len 4096 \
  --max_seq_len 8192 \
  --max_num_tokens 32768

Step 4: serve the ShareGPT engine.

trtllm-serve serve ./qwen25_7b_trt_engine_sharegpt \
  --backend tensorrt \
  --tokenizer /home/Qwen/Qwen2.5-7B-Instruct \
  --max_batch_size 256 \
  --host 0.0.0.0 \
  --port 8000

Benchmark Client Command

Note: For SGLang, use --base-url http://127.0.0.1:30000. Adjust --max-concurrency to 120, 180, 240, 300, 360, 420, 480, 540, 600, or remove it for no cap.

vllm bench serve \
  --model Qwen/Qwen2.5-7B-Instruct \
  --base-url http://127.0.0.1:8000 \
  --dataset-name sharegpt \
  --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 1000 \
  --max-concurrency 60 \
  --ignore-eos \
  --output-len 256 \
  --seed 42 \
  --save-result \
  --result-dir ./qwen25_7b_sharegpt \
  --result-filename qwen25_7b_sharegpt_c60.json

TTFT Mean (ms)

TPOT Mean (ms)

ITL Mean (ms)

Output Throughput (tok/s)

Insights

• TTFT: vLLM keeps first-token latency under control as concurrency increases, while TensorRT-LLM becomes much slower from concurrency 300 onward and reaches 7.8 s in the no-cap run. SGLang sits between the two, with some non-monotonic movement but lower high-concurrency TTFT than TensorRT-LLM.
• TPOT and ITL: TensorRT-LLM has the most stable decode latency in the capped runs, staying around 9-12 ms. vLLM decode latency rises gradually with concurrency. SGLang starts close to vLLM, but its ITL grows much faster at high concurrency and reaches 100.92 ms in the no-cap run.
• Output throughput: vLLM gives the strongest throughput curve overall, scaling from 6.9K tok/s at concurrency 60 to 23.5K tok/s in the no-cap run. SGLang remains competitive through most capped runs, while TensorRT-LLM stays lower than vLLM across the sweep.
• Takeaway: For this chat-style workload, vLLM gives the best latency-throughput balance. SGLang is still competitive on throughput, but decode latency becomes expensive at high concurrency. TensorRT-LLM decodes efficiently, but high TTFT makes it less attractive for interactive serving under pressure.

RULER 16K

Server Commands

vLLM

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 131072 \
  --hf-overrides '{"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'

SGLang

python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 \
  --port 30000 \
  --dtype bfloat16

TensorRT-LLM

Step 1: create the BF16 config.

cat > trtllm_bf16_config.yaml <<EOF
dtype: bfloat16
EOF

Step 2: convert the Hugging Face checkpoint.

python /home/TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py \
  --model_dir /home/Qwen/Qwen2.5-7B-Instruct \
  --output_dir /home/qwen25_7b_checkpoint \
  --dtype bfloat16 \
  --tp_size 1

Step 3: build the RULER 16K engine.

trtllm-build \
  --checkpoint_dir ./qwen25_7b_checkpoint \
  --output_dir ./qwen25_7b_trt_engine_ruler_16k \
  --gemm_plugin bfloat16 \
  --gpt_attention_plugin bfloat16 \
  --max_batch_size 64 \
  --max_input_len 16384 \
  --max_seq_len 16896 \
  --max_num_tokens 32768

Step 4: serve the RULER 16K engine.

trtllm-serve serve ./qwen25_7b_trt_engine_ruler_16k \
  --backend tensorrt \
  --tokenizer /home/Qwen/Qwen2.5-7B-Instruct \
  --max_batch_size 64 \
  --host 0.0.0.0 \
  --port 8000

Benchmark Client Command

Note: For SGLang, use --base-url http://127.0.0.1:30000. Adjust --max-concurrency to 120, 180, 240, 300, 360, 420, 480, 540, 600, or remove it for no cap.

vllm bench serve \
  --model Qwen/Qwen2.5-7B-Instruct \
  --base-url http://127.0.0.1:8000 \
  --dataset-name custom \
  --dataset-path ./ruler_16384_1000_vllm.jsonl \
  --num-prompts 1000 \
  --max-concurrency 60 \
  --seed 42 \
  --save-result \
  --result-dir ./qwen25_7b_ruler_16k \
  --result-filename qwen25_7b_ruler_16k_c60.json

TTFT Mean (ms)

TPOT Mean (ms)

ITL Mean (ms)

Output Throughput (tok/s)

Insights

• TTFT: RULER 16K is prefill-heavy, so first-token latency dominates the user experience. vLLM has the lowest TTFT at most concurrency levels. SGLang is consistently higher than vLLM, while TensorRT-LLM is competitive at low concurrency but rises more at medium and high concurrency.
• TPOT and ITL: SGLang has the best decode latency, with TPOT and ITL mostly around 77-81 ms after concurrency 120. vLLM stays around 119 ms for most runs, and TensorRT-LLM stays close to 108 ms.
• Output throughput: All three frameworks are close because the long-context workload is dominated by prompt processing. vLLM stays around 578-582 tok/s, TensorRT-LLM stays around 558-563 tok/s, and SGLang stays around 553-562 tok/s.
• Takeaway: vLLM is the best choice when TTFT matters most. SGLang is strongest for decode latency, but its higher TTFT makes it less attractive for long-prompt interactive workloads. TensorRT-LLM is stable, but does not clearly beat vLLM on this dataset.

Qwen3-30B-A3B

ShareGPT

Server Commands

vLLM

vllm serve /home/Qwen/Qwen3-30B-A3B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --dtype bfloat16

SGLang

python3 -m sglang.launch_server \
  --model-path /home/Qwen/Qwen3-30B-A3B \
  --host 0.0.0.0 \
  --port 30000 \
  --tp-size 2 \
  --dtype bfloat16

TensorRT-LLM

Step 1: create the BF16 config and download the model.

cat > trtllm_bf16_config.yaml <<EOF
dtype: bfloat16
EOF

hf download Qwen/Qwen3-30B-A3B \
  --local-dir /home/Qwen/Qwen3-30B-A3B

Step 2: convert the Hugging Face checkpoint.

python /home/TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py \
  --model_dir /home/Qwen/Qwen3-30B-A3B \
  --output_dir /home/qwen3_30b_checkpoint \
  --dtype bfloat16 \
  --tp_size 2

Step 3: build the ShareGPT engine.

trtllm-build \
  --checkpoint_dir ./qwen3_30b_checkpoint \
  --output_dir ./qwen3_30b_trt_engine \
  --gemm_plugin bfloat16 \
  --gpt_attention_plugin bfloat16 \
  --max_batch_size 128 \
  --max_input_len 4096 \
  --max_seq_len 8192

Step 4: serve the ShareGPT engine.

trtllm-serve serve ./qwen3_30b_trt_engine \
  --backend tensorrt \
  --tokenizer /home/Qwen/Qwen3-30B-A3B \
  --max_batch_size 128 \
  --max_input_len 4096 \
  --max_seq_len 8192 \
  --host 0.0.0.0 \
  --port 8000 \
  --tp_size 2

Benchmark Client Command

Note: For SGLang, use --base-url http://127.0.0.1:30000. Adjust --max-concurrency to 120, 180, 240, 300, 360, 420, 480, 540, 600, or remove it for no cap.

vllm bench serve \
  --model Qwen/Qwen3-30B-A3B \
  --base-url http://127.0.0.1:8000 \
  --dataset-name sharegpt \
  --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 1000 \
  --max-concurrency 60 \
  --ignore-eos \
  --output-len 256 \
  --seed 42 \
  --save-result \
  --result-dir ./qwen3_30b_sharegpt \
  --result-filename qwen3_30b_sharegpt_c60.json

TTFT Mean (ms)

TPOT Mean (ms)

ITL Mean (ms)

Output Throughput (tok/s)

Insights

• TTFT: vLLM and SGLang keep first-token latency much lower than TensorRT-LLM. TensorRT-LLM TTFT under this engine configuration grows aggressively with concurrency, crossing 8.9 s at concurrency 600 and reaching 13.6 s in the no-cap run.
• TPOT and ITL: TensorRT-LLM has the most stable decode path, staying near 17 ms TPOT/ITL across capped runs. vLLM increases gradually as concurrency grows. SGLang is competitive at low concurrency, but decode latency jumps after concurrency 240 and moves into the 40-50 ms range.
• Output throughput: vLLM gives the strongest throughput profile, peaking around 9.36K tok/s at concurrency 360 and staying above 8.4K tok/s at higher concurrency. SGLang is strong early but dips around concurrency 300-360 before recovering. TensorRT-LLM remains lower and flatter.
• Takeaway: For this MoE chat workload, vLLM gives the strongest end-to-end behavior. SGLang is useful at lower concurrency, but its decode latency becomes unstable under pressure. TensorRT-LLM under this engine configuration has efficient decode kernels, but TTFT is the limiting factor.

RULER 16K

Server Commands

vLLM

vllm serve /home/Qwen/Qwen3-30B-A3B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --dtype bfloat16

SGLang

python3 -m sglang.launch_server \
  --model-path /home/Qwen/Qwen3-30B-A3B \
  --host 0.0.0.0 \
  --port 30000 \
  --tp-size 2 \
  --dtype bfloat16

TensorRT-LLM

Step 1: create the BF16 config and download the model.

cat > trtllm_bf16_config.yaml <<EOF
dtype: bfloat16
EOF

hf download Qwen/Qwen3-30B-A3B \
  --local-dir /home/Qwen/Qwen3-30B-A3B

Step 2: convert the Hugging Face checkpoint.

python /home/TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py \
  --model_dir /home/Qwen/Qwen3-30B-A3B \
  --output_dir /home/qwen3_30b_checkpoint \
  --dtype bfloat16 \
  --tp_size 2

Step 3: build the RULER 16K engine.

trtllm-build \
  --checkpoint_dir ./qwen3_30b_checkpoint \
  --output_dir ./qwen3_30b_trt_engine_ruler_16k \
  --gemm_plugin bfloat16 \
  --gpt_attention_plugin bfloat16 \
  --max_batch_size 128 \
  --max_input_len 16384 \
  --max_seq_len 16896 \
  --max_num_tokens 16384

Step 4: serve the RULER 16K engine.

trtllm-serve serve ./qwen3_30b_trt_engine_ruler_16k \
  --backend tensorrt \
  --tokenizer /home/Qwen/Qwen3-30B-A3B \
  --max_batch_size 124 \
  --max_input_len 16384 \
  --max_num_tokens 16384 \
  --max_seq_len 16896 \
  --host 0.0.0.0 \
  --port 8000

Benchmark Client Command

Note: For SGLang, use --base-url http://127.0.0.1:30000. Adjust --max-concurrency to 120, 180, 240, 300, 360, 420, 480, 540, 600, or remove it for no cap.

vllm bench serve \
  --model Qwen/Qwen3-30B-A3B \
  --base-url http://127.0.0.1:8000 \
  --dataset-name custom \
  --dataset-path ./ruler_16384_1000_vllm.jsonl \
  --num-prompts 1000 \
  --max-concurrency 60 \
  --seed 42 \
  --save-result \
  --result-dir ./qwen3_30b_ruler_16k \
  --result-filename qwen3_30b_ruler_16k_c60.json

TTFT Mean (ms)

TPOT Mean (ms)

ITL Mean (ms)

Output Throughput (tok/s)

Insights

• TTFT: vLLM has the lowest TTFT at most concurrency levels, which is important because RULER 16K stresses prefill and long-context scheduling. SGLang is consistently higher, and TensorRT-LLM rises the most at high concurrency.
• TPOT and ITL: SGLang has the lowest decode latency, mostly around 61-63 ms. vLLM is steady around 70 ms. TensorRT-LLM is higher, usually in the 84-87 ms range after concurrency 120.
• Output throughput: vLLM leads throughput and stays around 804-807 tok/s for most of the sweep. SGLang follows closely around 776-781 tok/s. TensorRT-LLM trails at roughly 701-715 tok/s.
• Takeaway: vLLM handles this long-context MoE workload best overall because it combines the best TTFT and highest throughput. SGLang is attractive when decode latency is the main priority. TensorRT-LLM is stable but not the best choice for this run.

Qwen3-32B

ShareGPT

Server Commands

vLLM

vllm serve Qwen/Qwen3-32B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --dtype bfloat16

SGLang

python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-32B \
  --host 0.0.0.0 \
  --port 30000 \
  --tp-size 2 \
  --dtype bfloat16

TensorRT-LLM

Step 1: create the BF16 config and download the model.

cat > trtllm_bf16_config.yaml <<EOF
dtype: bfloat16
EOF

hf download Qwen/Qwen3-32B \
  --local-dir /home/Qwen/Qwen3-32B

Step 2: convert the Hugging Face checkpoint.

python /home/TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py \
  --model_dir /home/Qwen/Qwen3-32B \
  --output_dir /home/qwen3_32b_checkpoint \
  --dtype bfloat16 \
  --tp_size 2

Step 3: build the ShareGPT engine.

trtllm-build \
  --checkpoint_dir ./qwen3_32b_checkpoint \
  --output_dir ./qwen3_32b_trt_engine_sharegpt \
  --gemm_plugin bfloat16 \
  --gpt_attention_plugin bfloat16 \
  --max_batch_size 128 \
  --max_input_len 4096 \
  --max_seq_len 8192

Step 4: serve the ShareGPT engine.

trtllm-serve serve ./qwen3_32b_trt_engine_sharegpt \
  --backend tensorrt \
  --tokenizer /home/Qwen/Qwen3-32B \
  --max_batch_size 128 \
  --max_input_len 4096 \
  --max_seq_len 8192 \
  --host 0.0.0.0 \
  --port 8000 \
  --tp_size 2

Benchmark Client Command

Note: For SGLang, use --base-url http://127.0.0.1:30000. Adjust --max-concurrency to 120, 180, 240, 300, 360, 420, 480, 540, 600, or remove it for no cap.

vllm bench serve \
  --model Qwen/Qwen3-32B \
  --base-url http://127.0.0.1:8000 \
  --dataset-name sharegpt \
  --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 1000 \
  --max-concurrency 60 \
  --ignore-eos \
  --output-len 256 \
  --seed 42 \
  --save-result \
  --result-dir ./qwen3_32b_sharegpt \
  --result-filename qwen3_32b_sharegpt_c60.json

TTFT Mean (ms)

TPOT Mean (ms)

ITL Mean (ms)

Output Throughput (tok/s)

Insights

• TTFT: vLLM and SGLang have similar first-token latency patterns, with both increasing gradually as concurrency grows. TensorRT-LLM under this engine configuration starts reasonably at low concurrency, but TTFT becomes much higher from concurrency 180 onward and reaches 19.3 s in the no-cap run.
• TPOT and ITL: TensorRT-LLM has the most stable decode latency, with TPOT around 30 ms and ITL around 28 ms across most of the sweep. vLLM and SGLang decode latency both increase with concurrency, with SGLang generally higher than vLLM.
• Output throughput: vLLM reaches the highest throughput in several capped runs and peaks above 4.2K tok/s at concurrency 600. SGLang becomes close to vLLM at higher concurrency. TensorRT-LLM is stable but lower, staying around 3.2K tok/s after concurrency 240.
• Takeaway: vLLM gives the best throughput profile for this larger dense chat workload. SGLang is competitive, especially at higher concurrency, but has higher decode latency. TensorRT-LLM under this engine configuration is decode-stable, but its high TTFT makes it less suitable for interactive chat serving.

RULER 16K

Server Commands

vLLM

vllm serve Qwen/Qwen3-32B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.90

SGLang

python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-32B \
  --host 0.0.0.0 \
  --port 30000 \
  --tp-size 2 \
  --dtype bfloat16

TensorRT-LLM

Step 1: create the BF16 config and download the model.

cat > trtllm_bf16_config.yaml <<EOF
dtype: bfloat16
EOF

hf download Qwen/Qwen3-32B \
  --local-dir /home/Qwen/Qwen3-32B

Step 2: convert the Hugging Face checkpoint.

python /home/TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py \
  --model_dir /home/Qwen/Qwen3-32B \
  --output_dir /home/qwen3_32b_checkpoint \
  --dtype bfloat16 \
  --tp_size 2

Step 3: build the RULER 16K engine.

trtllm-build \
  --checkpoint_dir ./qwen3_32b_checkpoint \
  --output_dir ./qwen3_32b_trt_engine_ruler_16k \
  --gemm_plugin bfloat16 \
  --gpt_attention_plugin bfloat16 \
  --max_batch_size 128 \
  --max_input_len 16384 \
  --max_seq_len 16896 \
  --max_num_tokens 16384

Step 4: serve the RULER 16K engine.

trtllm-serve serve ./qwen3_32b_trt_engine_ruler_16k \
  --backend tensorrt \
  --tokenizer /home/Qwen/Qwen3-32B \
  --max_batch_size 124 \
  --max_input_len 16384 \
  --max_num_tokens 16384 \
  --max_seq_len 16896 \
  --host 0.0.0.0 \
  --port 8000

Benchmark Client Command

Note: For SGLang, use --base-url http://127.0.0.1:30000. Adjust --max-concurrency to 120, 180, 240, 300, 360, 420, 480, 540, 600, or remove it for no cap.

vllm bench serve \
  --model Qwen/Qwen3-32B \
  --base-url http://127.0.0.1:8000 \
  --dataset-name custom \
  --dataset-path ./ruler_16384_1000_vllm.jsonl \
  --num-prompts 1000 \
  --max-concurrency 60 \
  --seed 42 \
  --save-result \
  --result-dir ./qwen3_32b_ruler_16k \
  --result-filename qwen3_32b_ruler_16k_c60.json

TTFT Mean (ms)

TPOT Mean (ms)

ITL Mean (ms)

Output Throughput (tok/s)

Insights

• TTFT: All frameworks show very large TTFT because long-context prefill dominates this workload. vLLM and TensorRT-LLM are close through most capped runs, while SGLang has the highest TTFT across the full sweep.
• TPOT and ITL: vLLM has the best decode latency, with TPOT and ITL around 98 ms. TensorRT-LLM is stable but higher at around 107 ms. SGLang is more variable and reaches the highest decode latency in several high-concurrency runs.
• Output throughput: vLLM leads at about 191 tok/s. TensorRT-LLM is close behind at about 188 tok/s. SGLang trails at around 166-168 tok/s, which is a meaningful gap for this larger long-context workload.
• Takeaway: vLLM is the strongest overall for Qwen/Qwen3-32B on RULER 16K because it combines the best decode latency and highest throughput. TensorRT-LLM is a close throughput second, while SGLang is less competitive on both TTFT and throughput.

Verdict

Based on these specific H100 runs, vLLM is the strongest default choice. It gives the best overall balance of TTFT, decode latency, and output throughput across both chat-style and long-context workloads. It is especially strong when the workload needs predictable scaling across concurrency levels without heavy framework-specific tuning. These results are workload- and configuration-specific, not a universal ranking of the frameworks.

• Best general-purpose serving choice: vLLM vLLM is the most consistent framework across the benchmark. On ShareGPT, it usually gives the best throughput while keeping TTFT reasonable. On RULER 16K, it handles long-context prefill well and often gives the best TTFT or best overall throughput. This makes it the safest baseline for production-style serving where the workload can include both short chat requests and long prompts.

• Best decode-latency specialist in some RULER 16K runs: SGLang SGLang performs well when decode latency is the main focus. For example, on Qwen/Qwen2.5-7B-Instruct and Qwen/Qwen3-30B-A3B RULER 16K runs, SGLang has the lowest TPOT and ITL. However, this does not always translate into the best end-to-end serving behavior because TTFT is often higher and throughput is not always the best. SGLang looks most attractive when the application can benefit from runtime-level optimizations such as prefix reuse or when decode latency matters more than first-token latency.

• Best decode stability but not best interactive latency: TensorRT-LLM TensorRT-LLM under this engine configuration often shows very stable TPOT and ITL, especially on ShareGPT. This suggests that once decoding starts, the optimized TensorRT engine can produce tokens very consistently. The main issue is TTFT. In several ShareGPT runs, TensorRT-LLM first-token latency increases sharply as concurrency grows, partly because several engine builds use max_batch_size values lower than the tested concurrency sweep, making high-concurrency TTFT configuration-sensitive. This makes it less attractive for interactive chat workloads unless the engine and runtime configuration are tuned specifically for the target workload.

• For ShareGPT like workloads ShareGPT behaves more like normal chat serving, where both TTFT and throughput matter. vLLM is usually the best fit because it combines high output throughput with manageable first-token latency. SGLang is competitive, but can show higher decode latency at high concurrency. TensorRT-LLM has stable decode behavior, but TTFT can become the bottleneck.

• For RULER 16K like workloads RULER 16K is dominated by long-context prefill and KV cache pressure. In this setting, throughput differences are often smaller, and TTFT becomes very important. vLLM handles this balance best overall. SGLang can win on decode latency, but its higher TTFT reduces the benefit for interactive long-context use cases. TensorRT-LLM can be close to vLLM on throughput for larger models, but it does not consistently beat vLLM end to end.

Conclusion

The learning from this benchmark is that LLM serving performance cannot be judged from a single throughput number. A framework can have strong decode latency but poor TTFT, or high throughput but unstable latency under concurrency. TTFT, TPOT, ITL, and output throughput need to be read together.

For these Qwen workloads, vLLM is the best overall baseline. It is easy to run, scales well across concurrency levels, and gives strong results on both ShareGPT and RULER 16K. SGLang is promising when prefix reuse and decode latency are important, but its advantage depends more on workload shape. TensorRT-LLM is powerful but tuning-sensitive. It can provide stable decode performance, but the engine configuration and serving setup need to be carefully matched to the workload to avoid high first-token latency.

For practical deployment, we recommend starting with vLLM as the baseline, testing SGLang if the workload has repeated prefixes or structured generation patterns, and using TensorRT-LLM when the deployment is NVIDIA-specific and there is enough time to tune engine parameters for the exact model, prompt length, batch size, and concurrency target.