nvidia/Qwen3.6-35B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
8192 → 256
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)06854.603run_8Ced01m-zqw5EIM_: 1.95 req/s, p95 TTFT 1,796.4 msrun_MAB2jInmH7SirpY-: 2.81 req/s, p95 TTFT 6,854.6 msrun_uaXfxL9APnL55fCl: 0.91 req/s, p95 TTFT 529.7 msrun_1X4kpIOhVAK3GEF6: 2.44 req/s, p95 TTFT 3,029.4 msrun_UtKxcDEm_98K8GgM: 0.6 req/s, p95 TTFT 280.1 msrun_t6SjCZwezRLNVbXi: 1.43 req/s, p95 TTFT 1,025.1 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8498.41,796.4 ms15 ms4,355.5 ms0run_8Ced01m-zqw5EIM_
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=32720.16,854.6 ms41.37 ms15,925.8 ms0run_MAB2jInmH7SirpY-
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=2233.3529.7 ms7.55 ms2,204.9 ms0run_uaXfxL9APnL55fCl
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=166253,029.4 ms24.32 ms7,313.3 ms0run_1X4kpIOhVAK3GEF6
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=1154.4280.1 ms5.42 ms1,661.1 ms0run_UtKxcDEm_98K8GgM
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4365.91,025.1 ms9.93 ms2,820.1 ms0run_t6SjCZwezRLNVbXi
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
498.4
p95 TTFT / TPOT
1,796.4 ms / 15 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
720.1
p95 TTFT / TPOT
6,854.6 ms / 41.37 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
233.3
p95 TTFT / TPOT
529.7 ms / 7.55 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
625
p95 TTFT / TPOT
3,029.4 ms / 24.32 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
154.4
p95 TTFT / TPOT
280.1 ms / 5.42 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
365.9
p95 TTFT / TPOT
1,025.1 ms / 9.93 ms