nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

completed
Request throughput
25.44 req/s
Output throughput
6,513.6 tok/s
p95 TTFT
617.3 ms
p95 TPOT
10.63 ms

Manifest warnings

Review these published caveats before interpreting or comparing the metrics.

  • vLLM reported uncalibrated FP8 attention q/prob scales of 1.0; serving performance is measured, but model quality was not evaluated.
  • vLLM auto-selected weight-only Marlin FP4 kernels and reported no native FP4 support for this ModelOpt W4A16 checkpoint on RTX 5090.

Server launch command

CUDA_VISIBLE_DEVICES=0 vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --revision e8f3c7c4de75ad84fe1bcef95d38eca76214480b \
  --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 87 \
  --no-enable-prefix-caching \
  --disable-uvicorn-access-log \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --port 9001 \
  --tensor-parallel-size 1
CUDA_VISIBLE_DEVICES=1 vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --revision e8f3c7c4de75ad84fe1bcef95d38eca76214480b \
  --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 87 \
  --no-enable-prefix-caching \
  --disable-uvicorn-access-log \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --port 9002 \
  --tensor-parallel-size 1
CUDA_VISIBLE_DEVICES=2 vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --revision e8f3c7c4de75ad84fe1bcef95d38eca76214480b \
  --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 87 \
  --no-enable-prefix-caching \
  --disable-uvicorn-access-log \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --port 9003 \
  --tensor-parallel-size 1
CUDA_VISIBLE_DEVICES=3 vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --revision e8f3c7c4de75ad84fe1bcef95d38eca76214480b \
  --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 87 \
  --no-enable-prefix-caching \
  --disable-uvicorn-access-log \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --port 9004 \
  --tensor-parallel-size 1
RUNPILE_BACKENDS=<replica-urls> RUNPILE_PROXY_PORT=9000 python lb_proxy.py

Run details

Workload
1024 → 256
Methodology
closed loop · streaming
Test load
64 concurrent requests
Endpoint
openai_completions
Requests
16 explicit requests before the curve; the harness also issues one unmeasured endpoint test immediately before each timed point.
Dataset
name=vllm-random range_ratio=0 random_prefix_tokens=0
Sampling
mode=greedy temperature=0
Stopping
ignore_eos=true max_tokens=256
Prefix and cache
shared_prefix_tokens=0 server_prefix_caching=false
Client
name=vllm bench serve backend=vllm version=0.27.1
Hardware
4× NVIDIA GeForce RTX 5090
Runtime
vllm 0.27.1
Runtime configuration
dtype=bfloat16 max_num_seqs=87 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4
Measurement
vllm bench serve 0.27.1
Timing
Client-observed, streaming online serving interval after the harness initial test request. · client overhead included
Repetitions
1
Failure handling
No client retries; failed requests are counted as failures.
Quantization
format=NVFP4 checkpoint=modelopt
Model revision
e8f3c7c4de75ad84fe1bcef95d38eca76214480b
Evidence
Raw benchmark evidence attached · eligible and ranked
Published
by anonymous-TWTejS

Benchmark command

vllm bench serve \
  --backend vllm \
  --host localhost \
  --port 9000 \
  --endpoint /v1/completions \
  --model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --tokenizer nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --random-range-ratio 0 \
  --num-prompts 256 \
  --request-rate inf \
  --max-concurrency 64 \
  --ignore-eos \
  --temperature 0 \
  --seed 20260823 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --disable-tqdm \
  --save-result \
  --result-dir results \
  --result-filename nemotron35-lightning-30b-a3b-nvfp4__r4__i1024__o256__c64.json