nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Draft model
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
Workload
8192 → 256
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)02968.803run_tfzfgS8n0uZ01NXT: 2.85 req/s, p95 TTFT 1,622.8 msrun_pRrYrjM4CPliX0dS: 2.53 req/s, p95 TTFT 797.7 msrun_ArG_oINdsMEcAyCh: 2.51 req/s, p95 TTFT 844.9 msrun_eKkjC3IZCcVw6z3e: 2.49 req/s, p95 TTFT 862.1 msrun_ZIFcC2fUOhkuc0jc: 1.59 req/s, p95 TTFT 276.7 msrun_qsDhVimKEbtMIrVS: 3.03 req/s, p95 TTFT 2,968.8 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=87301,622.8 ms26.98 ms7,429.2 ms0run_tfzfgS8n0uZ01NXT
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4646.5797.7 ms14.31 ms3,931.6 ms0run_pRrYrjM4CPliX0dS
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4641.6844.9 ms14.66 ms4,112.1 ms0run_ArG_oINdsMEcAyCh
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4638.6862.1 ms14.84 ms4,160.3 ms0run_eKkjC3IZCcVw6z3e
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1407.8276.7 ms4.68 ms1,464.4 ms0run_ZIFcC2fUOhkuc0jc
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=16775.12,968.8 ms37.81 ms10,347.8 ms0run_qsDhVimKEbtMIrVS
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
730
p95 TTFT / TPOT
1,622.8 ms / 26.98 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
646.5
p95 TTFT / TPOT
797.7 ms / 14.31 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
641.6
p95 TTFT / TPOT
844.9 ms / 14.66 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
638.6
p95 TTFT / TPOT
862.1 ms / 14.84 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
407.8
p95 TTFT / TPOT
276.7 ms / 4.68 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
775.1
p95 TTFT / TPOT
2,968.8 ms / 37.81 ms