nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Draft model
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
Workload
1024 → 256
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)0550.1011run_YBadRmJytl7d1TgN: 8.48 req/s, p95 TTFT 253.9 msrun_n3TknEdAoRY8mvIb: 5.93 req/s, p95 TTFT 90.4 msrun_goIaFr_Qgj0kdTst: 2.62 req/s, p95 TTFT 43.8 msrun_vJqiTBJHe1YjkpoT: 10.64 req/s, p95 TTFT 517.1 msrun_xvVibzjOPHIsbYHd: 10.61 req/s, p95 TTFT 550.1 msrun_bemnDkPX5tAmQMs7: 10.54 req/s, p95 TTFT 513 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=82,172.1253.9 ms10.5 ms2,767.2 ms0run_YBadRmJytl7d1TgN
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=41,518.490.4 ms7.54 ms1,998.6 ms0run_n3TknEdAoRY8mvIb
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1670.243.8 ms4.45 ms1,176.3 ms0run_goIaFr_Qgj0kdTst
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,724517.1 ms13.74 ms3,572.3 ms0run_vJqiTBJHe1YjkpoT
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,717550.1 ms13.83 ms3,597.9 ms0run_xvVibzjOPHIsbYHd
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,698.4513 ms14.62 ms3,844.1 ms0run_bemnDkPX5tAmQMs7
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
2,172.1
p95 TTFT / TPOT
253.9 ms / 10.5 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
1,518.4
p95 TTFT / TPOT
90.4 ms / 7.54 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
670.2
p95 TTFT / TPOT
43.8 ms / 4.45 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,724
p95 TTFT / TPOT
517.1 ms / 13.74 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,717
p95 TTFT / TPOT
550.1 ms / 13.83 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,698.4
p95 TTFT / TPOT
513 ms / 14.62 ms