nvidia/Qwen3.6-35B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
4096 → 1
Runs
7 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)07454.109run_HWIpu4tw-o0e37dL: 8.58 req/s, p95 TTFT 1,864.3 msrun_hWNIAajUCzgQN7C3: 7.92 req/s, p95 TTFT 253.2 msrun_TMhDCrPyxvS4RtOR: 8.58 req/s, p95 TTFT 7,454.1 msrun_4_ay2L3rtyuVzDAS: 8.25 req/s, p95 TTFT 486.1 msrun_v-ZrkQZ9QfSngYEY: 8.58 req/s, p95 TTFT 3,727.1 msrun_7gsqaDlPmnel2H88: 7.33 req/s, p95 TTFT 138.3 msrun_CVLHAjG-mRboXSqF: 8.48 req/s, p95 TTFT 944.8 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=168.61,864.3 ms0 ms1,864.3 ms0run_HWIpu4tw-o0e37dL
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=27.9253.2 ms0 ms253.2 ms0run_hWNIAajUCzgQN7C3
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=648.67,454.1 ms0 ms7,454.1 ms0run_TMhDCrPyxvS4RtOR
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=48.3486.1 ms0 ms486.1 ms0run_4_ay2L3rtyuVzDAS
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=328.63,727.1 ms0 ms3,727.1 ms0run_v-ZrkQZ9QfSngYEY
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=17.3138.3 ms0 ms138.3 ms0run_7gsqaDlPmnel2H88
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=88.5944.8 ms0 ms944.8 ms0run_CVLHAjG-mRboXSqF
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
8.6
p95 TTFT / TPOT
1,864.3 ms / 0 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
7.9
p95 TTFT / TPOT
253.2 ms / 0 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
8.6
p95 TTFT / TPOT
7,454.1 ms / 0 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
8.3
p95 TTFT / TPOT
486.1 ms / 0 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
8.6
p95 TTFT / TPOT
3,727.1 ms / 0 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
7.3
p95 TTFT / TPOT
138.3 ms / 0 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
8.5
p95 TTFT / TPOT
944.8 ms / 0 ms