nvidia/Qwen3.6-35B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
256 → 1024
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)0293.502run_4fkMNYgAXjSddy1T: 0.94 req/s, p95 TTFT 118.9 msrun_WIhzqB_NliEAiOps: 2.43 req/s, p95 TTFT 293.5 msrun_mka-eHLzhgXMSPWe: 0.29 req/s, p95 TTFT 91.6 msrun_TCnNykWxfJSBjFMg: 1.59 req/s, p95 TTFT 214.5 msrun_d1gYj6D6EGbmSFg2: 0.17 req/s, p95 TTFT 53.1 msrun_oO7HdBDMk7LnD_fG: 0.56 req/s, p95 TTFT 90.7 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8967.6118.9 ms8.38 ms8,676.9 ms0run_4fkMNYgAXjSddy1T
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=322,493293.5 ms12.74 ms13,300.2 ms0run_WIhzqB_NliEAiOps
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=2292.591.6 ms6.81 ms7,022.9 ms0run_mka-eHLzhgXMSPWe
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,628.7214.5 ms9.88 ms10,298.4 ms0run_TCnNykWxfJSBjFMg
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=1174.553.1 ms5.69 ms5,871.3 ms0run_d1gYj6D6EGbmSFg2
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4577.690.7 ms6.94 ms7,178.7 ms0run_oO7HdBDMk7LnD_fG
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
967.6
p95 TTFT / TPOT
118.9 ms / 8.38 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
2,493
p95 TTFT / TPOT
293.5 ms / 12.74 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
292.5
p95 TTFT / TPOT
91.6 ms / 6.81 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,628.7
p95 TTFT / TPOT
214.5 ms / 9.88 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
174.5
p95 TTFT / TPOT
53.1 ms / 5.69 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
577.6
p95 TTFT / TPOT
90.7 ms / 6.94 ms