nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
8192 → 256
Runs
35 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)07266.503run_2FLIC-6vnIULZkw0: 2.36 req/s, p95 TTFT 2,061.6 msrun_AHKS717Z228rrvoH: 1.92 req/s, p95 TTFT 1,032.1 msrun_Cj7-zuFhan8KOfQx: 1.94 req/s, p95 TTFT 1,037 msrun_Q6779AvfAerIfSsi: 1.1 req/s, p95 TTFT 277.6 msrun_-BOXt8JQCKaj7JfO: 1.1 req/s, p95 TTFT 277.2 msrun__Da6g0Rlzf9GN_Di: 1.1 req/s, p95 TTFT 276.5 msrun_617vimGjp9zy5Ls5: 2.71 req/s, p95 TTFT 2,841.5 msrun_8V44xmDUoVj2GzMc: 2.61 req/s, p95 TTFT 1,667.5 msrun_Z2KhZQG_T_xOhuRS: 2.22 req/s, p95 TTFT 851.1 msrun_65BuldXkrrSfNeNh: 2.26 req/s, p95 TTFT 1,091.6 msrun_AzgSbfULR4EvA3RC: 1.33 req/s, p95 TTFT 289.8 msrun_qjfl6PnmhQWEf5Ne: 1.33 req/s, p95 TTFT 288.8 msrun_N8EXKzHJik_BDMqW: 1.33 req/s, p95 TTFT 289.7 msrun_ctbMXvAL-EKHsALa: 2.9 req/s, p95 TTFT 2,982.1 msrun_V6q2dINUh3fxlK7l: 2.23 req/s, p95 TTFT 1,548.2 msrun_vUpZCChcg0NrMHgW: 2.22 req/s, p95 TTFT 1,905.9 msrun_yNgBMwyLnRwhOg7F: 2.22 req/s, p95 TTFT 1,905.6 msrun_vXbPxdKmNNOFGWBe: 2.29 req/s, p95 TTFT 2,141.8 msrun_10nhaykiuVOpplTQ: 2.34 req/s, p95 TTFT 1,972.5 msrun_8xi9IBwje5x8byJ0: 2.34 req/s, p95 TTFT 2,053.4 msrun_JJbypLCsWOVVTA0B: 2.37 req/s, p95 TTFT 2,035.2 msrun_TN2-FbwQo3RhJKSe: 2.19 req/s, p95 TTFT 1,921.3 msrun_xMislh5naHT0nFIJ: 2.32 req/s, p95 TTFT 1,546 msrun_egM6U1suHKreDORg: 2.33 req/s, p95 TTFT 606.5 msrun_xqyTjs_S1SV2HZHo: 2.34 req/s, p95 TTFT 2,058 msrun_3WFfEa7h7hNC6W5s: 2.27 req/s, p95 TTFT 1,353.3 msrun_lfDFigiwWYu9eINq: 2.28 req/s, p95 TTFT 2,057.2 msrun_sGIiozpF6tfs_yPv: 1.89 req/s, p95 TTFT 1,081.1 msrun_JAo_wRAqESj4rFPL: 1.89 req/s, p95 TTFT 1,080 msrun_4HC28_Uot5fg3SCe: 2.94 req/s, p95 TTFT 7,266.5 msrun_2R6uZc_mlaEd3zP-: 1.43 req/s, p95 TTFT 549 msrun_Vkk2P7k1R4eUCfLY: 1.43 req/s, p95 TTFT 549.6 msrun_D01tOQ-BWyC2ePCg: 1.43 req/s, p95 TTFT 548.4 msrun_UbrhTV7jyZ3EQGwJ: 1.07 req/s, p95 TTFT 284 msrun_fHZ0BOgbaugzpuRF: 2.64 req/s, p95 TTFT 2,957.8 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8603.22,061.6 ms12.19 ms3,704.4 ms0run_2FLIC-6vnIULZkw0
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4491.21,032.1 ms7.19 ms2,116.4 ms0run_AHKS717Z228rrvoH
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4497.51,037 ms7.09 ms2,090 ms0run_Cj7-zuFhan8KOfQx
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1281.2277.6 ms2.52 ms916.1 ms0run_Q6779AvfAerIfSsi
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1281.1277.2 ms2.52 ms916.5 ms0run_-BOXt8JQCKaj7JfO
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1280.8276.5 ms2.53 ms916.8 ms0run__Da6g0Rlzf9GN_Di
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=16694.92,841.5 ms21.73 ms6,955.5 ms0run_617vimGjp9zy5Ls5
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8669.41,667.5 ms10.94 ms3,513.9 ms0run_8V44xmDUoVj2GzMc
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4569.1851.1 ms6.16 ms1,890.2 ms0run_Z2KhZQG_T_xOhuRS
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4579.51,091.6 ms5.92 ms2,059.6 ms0run_65BuldXkrrSfNeNh
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1340.9289.8 ms2 ms797 ms0run_AzgSbfULR4EvA3RC
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1341.3288.8 ms1.99 ms793.4 ms0run_qjfl6PnmhQWEf5Ne
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1341.3289.7 ms1.99 ms793 ms0run_N8EXKzHJik_BDMqW
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=16742.82,982.1 ms20.22 ms6,974.9 ms0run_ctbMXvAL-EKHsALa
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8569.81,548.2 ms13.04 ms3,691.7 ms0run_V6q2dINUh3fxlK7l
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8568.31,905.9 ms12.96 ms3,869.7 ms0run_vUpZCChcg0NrMHgW
vllm 0.28.0dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8568.31,905.6 ms13 ms3,873.8 ms0run_yNgBMwyLnRwhOg7F
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8586.22,141.8 ms12.56 ms3,795.1 ms0run_vXbPxdKmNNOFGWBe
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8599.91,972.5 ms12.3 ms3,712.6 ms0run_10nhaykiuVOpplTQ
vllm 0.28.0dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8599.42,053.4 ms12.32 ms3,712.2 ms0run_8xi9IBwje5x8byJ0
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8606.52,035.2 ms12.17 ms3,675.9 ms0run_JJbypLCsWOVVTA0B
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8561.71,921.3 ms13.11 ms3,912.7 ms0run_TN2-FbwQo3RhJKSe
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192c=8594.51,546 ms11.65 ms3,712.3 ms0run_xMislh5naHT0nFIJ
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048c=8596.5606.5 ms12.09 ms3,530.4 ms0run_egM6U1suHKreDORg
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8599.72,058 ms12.32 ms3,718.5 ms0run_xqyTjs_S1SV2HZHo
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8581.31,353.3 ms12.71 ms3,800.4 ms0run_3WFfEa7h7hNC6W5s
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8582.42,057.2 ms12.7 ms3,792.5 ms0run_lfDFigiwWYu9eINq
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4483.41,081.1 ms7.28 ms2,149.7 ms0run_sGIiozpF6tfs_yPv
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4483.11,080 ms7.29 ms2,151.4 ms0run_JAo_wRAqESj4rFPL
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=32751.87,266.5 ms39.29 ms15,728.7 ms0run_4HC28_Uot5fg3SCe
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=2366.3549 ms4.43 ms1,413.2 ms0run_2R6uZc_mlaEd3zP-
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=2366.2549.6 ms4.43 ms1,413.5 ms0run_Vkk2P7k1R4eUCfLY
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=2365.5548.4 ms4.45 ms1,415.9 ms0run_D01tOQ-BWyC2ePCg
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=1274.6284 ms2.57 ms935.7 ms0run_UbrhTV7jyZ3EQGwJ
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=16676.72,957.8 ms22.46 ms6,888.2 ms0run_fHZ0BOgbaugzpuRF
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
603.2
p95 TTFT / TPOT
2,061.6 ms / 12.19 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
491.2
p95 TTFT / TPOT
1,032.1 ms / 7.19 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
497.5
p95 TTFT / TPOT
1,037 ms / 7.09 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
281.2
p95 TTFT / TPOT
277.6 ms / 2.52 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
281.1
p95 TTFT / TPOT
277.2 ms / 2.52 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
280.8
p95 TTFT / TPOT
276.5 ms / 2.53 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
694.9
p95 TTFT / TPOT
2,841.5 ms / 21.73 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
669.4
p95 TTFT / TPOT
1,667.5 ms / 10.94 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
569.1
p95 TTFT / TPOT
851.1 ms / 6.16 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
579.5
p95 TTFT / TPOT
1,091.6 ms / 5.92 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
340.9
p95 TTFT / TPOT
289.8 ms / 2 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
341.3
p95 TTFT / TPOT
288.8 ms / 1.99 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
341.3
p95 TTFT / TPOT
289.7 ms / 1.99 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
742.8
p95 TTFT / TPOT
2,982.1 ms / 20.22 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
569.8
p95 TTFT / TPOT
1,548.2 ms / 13.04 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
568.3
p95 TTFT / TPOT
1,905.9 ms / 12.96 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
568.3
p95 TTFT / TPOT
1,905.6 ms / 13 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
586.2
p95 TTFT / TPOT
2,141.8 ms / 12.56 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
599.9
p95 TTFT / TPOT
1,972.5 ms / 12.3 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
599.4
p95 TTFT / TPOT
2,053.4 ms / 12.32 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
606.5
p95 TTFT / TPOT
2,035.2 ms / 12.17 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
561.7
p95 TTFT / TPOT
1,921.3 ms / 13.11 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192
Test load
c=8
Req/s
Output tok/s
594.5
p95 TTFT / TPOT
1,546 ms / 11.65 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048
Test load
c=8
Req/s
Output tok/s
596.5
p95 TTFT / TPOT
606.5 ms / 12.09 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
599.7
p95 TTFT / TPOT
2,058 ms / 12.32 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
581.3
p95 TTFT / TPOT
1,353.3 ms / 12.71 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
582.4
p95 TTFT / TPOT
2,057.2 ms / 12.7 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
483.4
p95 TTFT / TPOT
1,081.1 ms / 7.28 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
483.1
p95 TTFT / TPOT
1,080 ms / 7.29 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
751.8
p95 TTFT / TPOT
7,266.5 ms / 39.29 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
366.3
p95 TTFT / TPOT
549 ms / 4.43 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
366.2
p95 TTFT / TPOT
549.6 ms / 4.43 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
365.5
p95 TTFT / TPOT
548.4 ms / 4.45 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
274.6
p95 TTFT / TPOT
284 ms / 2.57 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
676.7
p95 TTFT / TPOT
2,957.8 ms / 22.46 ms