nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
4096 → 1
Runs
22 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)08304.108run_XzSeR7JZOHSUp-c2: 8.03 req/s, p95 TTFT 3,996.8 msrun_-yRr2XFiWJ5otgKg: 7.62 req/s, p95 TTFT 4,197.8 msrun_W4IYJAM69vFSGNVW: 7.63 req/s, p95 TTFT 4,191.7 msrun_5BMoWYpQNsoRhl4y: 7.71 req/s, p95 TTFT 4,155.9 msrun_2G6dgtDCLc90O28u: 8.08 req/s, p95 TTFT 3,961 msrun_7YCHuHSH4bODJOBr: 8.08 req/s, p95 TTFT 3,959.7 msrun_hlKaL_q2ojegjW5y: 8.12 req/s, p95 TTFT 3,951.5 msrun_wXYeL5edHNj9IM6z: 7.57 req/s, p95 TTFT 4,227 msrun_oeK94Cn5wF-ZAFQj: 8.04 req/s, p95 TTFT 3,999.7 msrun_qicxrkCpbwXMEt3z: 8.16 req/s, p95 TTFT 3,948.1 msrun_cO_2Sj4P8rTFtpgD: 8.06 req/s, p95 TTFT 3,979.4 msrun_pTe0ezQoWSt4vtg2: 7.71 req/s, p95 TTFT 4,160.9 msrun_LKdM-MYA3qsd4IPC: 7.7 req/s, p95 TTFT 1,042.4 msrun_MMxpWfjbVsa8cDtc: 7.69 req/s, p95 TTFT 1,044.2 msrun_Jy3Mf6OjYe0VrJYe: 7.71 req/s, p95 TTFT 8,304.1 msrun_LyrubmuR_q2T1VaU: 7.59 req/s, p95 TTFT 529.3 msrun_xQCZz2nG_1U52xNv: 7.6 req/s, p95 TTFT 528.6 msrun_k4rY3_EH2LsEEEfA: 7.6 req/s, p95 TTFT 528.9 msrun_Snh3BUTGR2VTUjXq: 7.72 req/s, p95 TTFT 4,159.4 msrun_IQZP65WPTojqyLqa: 7.51 req/s, p95 TTFT 267.5 msrun_yncapDvxslwomtfW: 7.2 req/s, p95 TTFT 140.7 msrun_4w1ZmmUeubCNSxai: 7.7 req/s, p95 TTFT 2,083.2 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=3283,996.8 ms0 ms3,996.8 ms0run_XzSeR7JZOHSUp-c2
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=327.64,197.8 ms0 ms4,197.8 ms0run_-yRr2XFiWJ5otgKg
vllm 0.28.0dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=327.64,191.7 ms0 ms4,191.7 ms0run_W4IYJAM69vFSGNVW
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=327.74,155.9 ms0 ms4,155.9 ms0run_5BMoWYpQNsoRhl4y
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=328.13,961 ms0 ms3,961 ms0run_2G6dgtDCLc90O28u
vllm 0.28.0dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=328.13,959.7 ms0 ms3,959.7 ms0run_7YCHuHSH4bODJOBr
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=328.13,951.5 ms0 ms3,951.5 ms0run_hlKaL_q2ojegjW5y
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=327.64,227 ms0 ms4,227 ms0run_wXYeL5edHNj9IM6z
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192c=3283,999.7 ms0 ms3,999.7 ms0run_oeK94Cn5wF-ZAFQj
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048c=328.23,948.1 ms0 ms3,948.1 ms0run_qicxrkCpbwXMEt3z
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=328.13,979.4 ms0 ms3,979.4 ms0run_cO_2Sj4P8rTFtpgD
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=327.74,160.9 ms0 ms4,160.9 ms0run_pTe0ezQoWSt4vtg2
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=87.71,042.4 ms0 ms1,042.4 ms0run_LKdM-MYA3qsd4IPC
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=87.71,044.2 ms0 ms1,044.2 ms0run_MMxpWfjbVsa8cDtc
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=647.78,304.1 ms0 ms8,304.1 ms0run_Jy3Mf6OjYe0VrJYe
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=47.6529.3 ms0 ms529.3 ms0run_LyrubmuR_q2T1VaU
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=47.6528.6 ms0 ms528.6 ms0run_xQCZz2nG_1U52xNv
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=47.6528.9 ms0 ms528.9 ms0run_k4rY3_EH2LsEEEfA
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=327.74,159.4 ms0 ms4,159.4 ms0run_Snh3BUTGR2VTUjXq
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=27.5267.5 ms0 ms267.5 ms0run_IQZP65WPTojqyLqa
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=17.2140.7 ms0 ms140.7 ms0run_yncapDvxslwomtfW
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=167.72,083.2 ms0 ms2,083.2 ms0run_4w1ZmmUeubCNSxai
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
8
p95 TTFT / TPOT
3,996.8 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
7.6
p95 TTFT / TPOT
4,197.8 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
7.6
p95 TTFT / TPOT
4,191.7 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
7.7
p95 TTFT / TPOT
4,155.9 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
8.1
p95 TTFT / TPOT
3,961 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
8.1
p95 TTFT / TPOT
3,959.7 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
8.1
p95 TTFT / TPOT
3,951.5 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
7.6
p95 TTFT / TPOT
4,227 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192
Test load
c=32
Req/s
Output tok/s
8
p95 TTFT / TPOT
3,999.7 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048
Test load
c=32
Req/s
Output tok/s
8.2
p95 TTFT / TPOT
3,948.1 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
8.1
p95 TTFT / TPOT
3,979.4 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
7.7
p95 TTFT / TPOT
4,160.9 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
7.7
p95 TTFT / TPOT
1,042.4 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
7.7
p95 TTFT / TPOT
1,044.2 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
7.7
p95 TTFT / TPOT
8,304.1 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
7.6
p95 TTFT / TPOT
529.3 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
7.6
p95 TTFT / TPOT
528.6 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
7.6
p95 TTFT / TPOT
528.9 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
7.7
p95 TTFT / TPOT
4,159.4 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
7.5
p95 TTFT / TPOT
267.5 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
7.2
p95 TTFT / TPOT
140.7 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
7.7
p95 TTFT / TPOT
2,083.2 ms / 0 ms