nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Draft model
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
Workload
256 → 1024
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)0222.304run_aWm8AoNgodtYiUZ2: 2.98 req/s, p95 TTFT 127 msrun_Fswyxt0znFesC5BD: 1.92 req/s, p95 TTFT 54.2 msrun_I0YaAH4pTAsLLpj5: 0.74 req/s, p95 TTFT 17.6 msrun_ct4jMDkN4n0DjthI: 4.08 req/s, p95 TTFT 214.9 msrun_vb2CuK5K0ZpY4H88: 4.09 req/s, p95 TTFT 222.3 msrun_oNrvbSwfaMBWVrYY: 4.13 req/s, p95 TTFT 217.5 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=83,053.7127 ms6.01 ms6,200.7 ms0run_aWm8AoNgodtYiUZ2
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=41,968.154.2 ms4.87 ms5,035.4 ms0run_Fswyxt0znFesC5BD
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1760.817.6 ms3.34 ms3,436.4 ms0run_I0YaAH4pTAsLLpj5
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=164,182.4214.9 ms8.2 ms8,545.1 ms0run_ct4jMDkN4n0DjthI
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=164,190.8222.3 ms7.76 ms8,071.5 ms0run_vb2CuK5K0ZpY4H88
vllm 0.28.0Speculative · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=164,231.6217.5 ms8.11 ms8,406.1 ms0run_oNrvbSwfaMBWVrYY
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
3,053.7
p95 TTFT / TPOT
127 ms / 6.01 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
1,968.1
p95 TTFT / TPOT
54.2 ms / 4.87 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
760.8
p95 TTFT / TPOT
17.6 ms / 3.34 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
4,182.4
p95 TTFT / TPOT
214.9 ms / 8.2 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
4,190.8
p95 TTFT / TPOT
222.3 ms / 7.76 ms
Runtime
vllm 0.28.0Speculative · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
4,231.6
p95 TTFT / TPOT
217.5 ms / 8.11 ms