nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Draft model
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash
Workload
8192 → 256
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)02986.603run_8kimbI29D2VmipKS: 2.68 req/s, p95 TTFT 1,340.3 msrun_fCbdP8B8dG3_yExJ: 2.3 req/s, p95 TTFT 558.6 msrun_IFhiBPHhCWe0L9ll: 2.3 req/s, p95 TTFT 552.8 msrun_q5AE-teZfGbi4p7H: 2.28 req/s, p95 TTFT 560.9 msrun_5f9xt25U4ZcL88lj: 1.35 req/s, p95 TTFT 283.4 msrun_t50JJDa2r9nTYmA7: 2.85 req/s, p95 TTFT 2,986.6 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0DFlash · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8686.11,340.3 ms27.69 ms7,449.7 ms0run_8kimbI29D2VmipKS
vllm 0.28.0DFlash · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4589.8558.6 ms16 ms4,431.5 ms0run_fCbdP8B8dG3_yExJ
vllm 0.28.0DFlash · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4589.6552.8 ms15.18 ms4,386.3 ms0run_IFhiBPHhCWe0L9ll
vllm 0.28.0DFlash · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4583.8560.9 ms16.38 ms4,506.8 ms0run_q5AE-teZfGbi4p7H
vllm 0.28.0DFlash · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1345.2283.4 ms5.53 ms1,686.9 ms0run_5f9xt25U4ZcL88lj
vllm 0.28.0DFlash · NVFP4dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=16728.42,986.6 ms38.21 ms11,288 ms0run_t50JJDa2r9nTYmA7
Runtime
vllm 0.28.0DFlash · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
686.1
p95 TTFT / TPOT
1,340.3 ms / 27.69 ms
Runtime
vllm 0.28.0DFlash · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
589.8
p95 TTFT / TPOT
558.6 ms / 16 ms
Runtime
vllm 0.28.0DFlash · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
589.6
p95 TTFT / TPOT
552.8 ms / 15.18 ms
Runtime
vllm 0.28.0DFlash · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
583.8
p95 TTFT / TPOT
560.9 ms / 16.38 ms
Runtime
vllm 0.28.0DFlash · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
345.2
p95 TTFT / TPOT
283.4 ms / 5.53 ms
Runtime
vllm 0.28.0DFlash · NVFP4 draft
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
728.4
p95 TTFT / TPOT
2,986.6 ms / 38.21 ms