nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
256 → 1024
Runs
31 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)0316.903run_Af4d_mGCJygrOmHW: 1.43 req/s, p95 TTFT 130.8 msrun_b8gbY3QEmZgTnfm0: 0.93 req/s, p95 TTFT 85.9 msrun_PN16NPIdrZDfyvrX: 0.36 req/s, p95 TTFT 46.9 msrun_Ro9uIWtcgjiXiuYF: 2.25 req/s, p95 TTFT 212.2 msrun_c67Ej2RQWJDt2z9e: 2.24 req/s, p95 TTFT 238.6 msrun_3RXkhDsCK872xzMf: 2.26 req/s, p95 TTFT 214.6 msrun_qEFCrG6KI7tZgQ0U: 2.1 req/s, p95 TTFT 132.4 msrun_6L9AtKI64SQbmUoj: 1.35 req/s, p95 TTFT 59.2 msrun_ZPFWcgrcHzYZhFZW: 0.5 req/s, p95 TTFT 48.4 msrun_Tcx9VyYqx3FSU1cu: 3.36 req/s, p95 TTFT 216.5 msrun_A5jI35snsEfN0uB1: 3.26 req/s, p95 TTFT 316.9 msrun_L0DMVrEYnAJupHMI: 3.26 req/s, p95 TTFT 294.8 msrun_Dl5j1sG7KbhSzE-l: 1.97 req/s, p95 TTFT 206 msrun_6AslH1dWkrn61A71: 2.02 req/s, p95 TTFT 180.2 msrun_O-9BGbR6fSTJLzM2: 2.03 req/s, p95 TTFT 157.7 msrun_-GXlt-CY-CFtDPSs: 2.19 req/s, p95 TTFT 180.5 msrun_ooVBmyrpZ9OZd8a7: 2.18 req/s, p95 TTFT 230.7 msrun_LAUmeIc6To6n8bT3: 2.19 req/s, p95 TTFT 247.3 msrun_HR_uyNsH6xRwXOjY: 2.25 req/s, p95 TTFT 234.1 msrun_APmGBIAv0Col8Qxz: 1.99 req/s, p95 TTFT 189.5 msrun_w_Wcjq8ywHuLcok8: 2.19 req/s, p95 TTFT 205.1 msrun_Qh0JayXSKLD8bCuE: 2.17 req/s, p95 TTFT 206.1 msrun_7ss30ADFzStkAdol: 2.18 req/s, p95 TTFT 203.4 msrun_A-lyFQHmHcWj3iZj: 1.41 req/s, p95 TTFT 99.2 msrun_wmXE1uvK25ubnZBr: 0.92 req/s, p95 TTFT 56.9 msrun_U3WlTjqNAbfIK9qr: 3.38 req/s, p95 TTFT 299.8 msrun_uruQ7_VyXFpA2KYR: 3.41 req/s, p95 TTFT 298.4 msrun_cCuNskjvCjBOA2-A: 3.38 req/s, p95 TTFT 314.3 msrun_Sahw-4h0xpkjvwSD: 0.55 req/s, p95 TTFT 56 msrun_p80xuoGWwvaS4gcs: 0.36 req/s, p95 TTFT 30.6 msrun_LRkck5a8qkT9b7jp: 2.19 req/s, p95 TTFT 180.3 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=81,468.1130.8 ms5.56 ms5,789.2 ms0run_Af4d_mGCJygrOmHW
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4950.785.9 ms4.27 ms4,447.7 ms0run_b8gbY3QEmZgTnfm0
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1368.346.9 ms2.68 ms2,785.2 ms0run_PN16NPIdrZDfyvrX
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,308212.2 ms6.97 ms7,308.6 ms0run_Ro9uIWtcgjiXiuYF
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,296.7238.6 ms6.94 ms7,339.1 ms0run_c67Ej2RQWJDt2z9e
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,314.8214.6 ms6.97 ms7,349.6 ms0run_3RXkhDsCK872xzMf
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=82,155.5132.4 ms3.86 ms4,005.8 ms0run_qEFCrG6KI7tZgQ0U
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=41,384.759.2 ms3.03 ms3,152.4 ms0run_6L9AtKI64SQbmUoj
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1512.448.4 ms2.07 ms2,163.1 ms0run_ZPFWcgrcHzYZhFZW
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=163,445.4216.5 ms4.57 ms4,748.3 ms0run_Tcx9VyYqx3FSU1cu
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=163,339.6316.9 ms4.66 ms4,932.7 ms0run_A5jI35snsEfN0uB1
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=163,335.1294.8 ms4.7 ms4,935.8 ms0run_L0DMVrEYnAJupHMI
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162,018.2206 ms7.9 ms8,276.5 ms0run_Dl5j1sG7KbhSzE-l
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162,073.4180.2 ms7.88 ms8,246.4 ms0run_6AslH1dWkrn61A71
vllm 0.28.0dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162,077.6157.7 ms7.86 ms8,192.5 ms0run_O-9BGbR6fSTJLzM2
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162,241.3180.5 ms7.13 ms7,458.3 ms0run_-GXlt-CY-CFtDPSs
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162,229.1230.7 ms7.18 ms7,572.6 ms0run_ooVBmyrpZ9OZd8a7
vllm 0.28.0dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162,246.6247.3 ms7.15 ms7,566.1 ms0run_LAUmeIc6To6n8bT3
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162,303.4234.1 ms7.13 ms7,523.7 ms0run_HR_uyNsH6xRwXOjY
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162,037189.5 ms7.9 ms8,250.8 ms0run_APmGBIAv0Col8Qxz
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192c=162,247.4205.1 ms7.27 ms7,587.2 ms0run_w_Wcjq8ywHuLcok8
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048c=162,219.5206.1 ms7.28 ms7,592.5 ms0run_Qh0JayXSKLD8bCuE
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162,232.7203.4 ms7.18 ms7,546.4 ms0run_7ss30ADFzStkAdol
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=81,442.399.2 ms5.76 ms5,990.5 ms0run_A-lyFQHmHcWj3iZj
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4944.856.9 ms4.3 ms4,461.5 ms0run_wmXE1uvK25ubnZBr
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=323,457.1299.8 ms9.1 ms9,588.6 ms0run_U3WlTjqNAbfIK9qr
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=323,493.4298.4 ms9.03 ms9,432.8 ms0run_uruQ7_VyXFpA2KYR
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=323,458.4314.3 ms9.14 ms9,661.7 ms0run_cCuNskjvCjBOA2-A
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=2566.956 ms3.56 ms3,673.7 ms0run_Sahw-4h0xpkjvwSD
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=1364.730.6 ms2.72 ms2,812.5 ms0run_p80xuoGWwvaS4gcs
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162,238180.3 ms7.13 ms7,474.7 ms0run_LRkck5a8qkT9b7jp
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,468.1
p95 TTFT / TPOT
130.8 ms / 5.56 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
950.7
p95 TTFT / TPOT
85.9 ms / 4.27 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
368.3
p95 TTFT / TPOT
46.9 ms / 2.68 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,308
p95 TTFT / TPOT
212.2 ms / 6.97 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,296.7
p95 TTFT / TPOT
238.6 ms / 6.94 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,314.8
p95 TTFT / TPOT
214.6 ms / 6.97 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
2,155.5
p95 TTFT / TPOT
132.4 ms / 3.86 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
1,384.7
p95 TTFT / TPOT
59.2 ms / 3.03 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
512.4
p95 TTFT / TPOT
48.4 ms / 2.07 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
3,445.4
p95 TTFT / TPOT
216.5 ms / 4.57 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
3,339.6
p95 TTFT / TPOT
316.9 ms / 4.66 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
3,335.1
p95 TTFT / TPOT
294.8 ms / 4.7 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,018.2
p95 TTFT / TPOT
206 ms / 7.9 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,073.4
p95 TTFT / TPOT
180.2 ms / 7.88 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,077.6
p95 TTFT / TPOT
157.7 ms / 7.86 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,241.3
p95 TTFT / TPOT
180.5 ms / 7.13 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,229.1
p95 TTFT / TPOT
230.7 ms / 7.18 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,246.6
p95 TTFT / TPOT
247.3 ms / 7.15 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,303.4
p95 TTFT / TPOT
234.1 ms / 7.13 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,037
p95 TTFT / TPOT
189.5 ms / 7.9 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192
Test load
c=16
Req/s
Output tok/s
2,247.4
p95 TTFT / TPOT
205.1 ms / 7.27 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048
Test load
c=16
Req/s
Output tok/s
2,219.5
p95 TTFT / TPOT
206.1 ms / 7.28 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,232.7
p95 TTFT / TPOT
203.4 ms / 7.18 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,442.3
p95 TTFT / TPOT
99.2 ms / 5.76 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
944.8
p95 TTFT / TPOT
56.9 ms / 4.3 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
3,457.1
p95 TTFT / TPOT
299.8 ms / 9.1 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
3,493.4
p95 TTFT / TPOT
298.4 ms / 9.03 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
3,458.4
p95 TTFT / TPOT
314.3 ms / 9.14 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
566.9
p95 TTFT / TPOT
56 ms / 3.56 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
364.7
p95 TTFT / TPOT
30.6 ms / 2.72 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,238
p95 TTFT / TPOT
180.3 ms / 7.13 ms