unsloth/Qwen3.8-27B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
8192 → 256
Runs
15 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)036262.701run_G3h2UvDdN-VkaEfs: 0.3 req/s, p95 TTFT 1,938.3 msrun_Zr0ZfH1dmdqT_kDb: 0.3 req/s, p95 TTFT 1,937.6 msrun_uLqgfmu_6Hy5cn1t: 0.3 req/s, p95 TTFT 1,936.1 msrun_vLZufKb1K1zn-elA: 0.54 req/s, p95 TTFT 3,061.3 msrun_ZmPTn050k3QQME9i: 0.56 req/s, p95 TTFT 1,514.1 msrun_koIl7B9e_gXcr-EG: 0.5 req/s, p95 TTFT 3,111 msrun_Rp8ecJixIoeV5Bsa: 0.5 req/s, p95 TTFT 2,338.4 msrun_3bSQLP_-3auTpmvN: 0.72 req/s, p95 TTFT 4,685.7 msrun_FW-OGBWRqx7n-e74: 0.5 req/s, p95 TTFT 3,109.8 msrun_BISUbbE4IuZZX-_J: 0.73 req/s, p95 TTFT 36,262.7 msrun_ppah1fyumJuYwaIL: 0.32 req/s, p95 TTFT 1,550.9 msrun_aJyhC38O-J7nURZ3: 0.19 req/s, p95 TTFT 780.3 msrun_1qf9IfT3FUYv7Z16: 0.19 req/s, p95 TTFT 780.9 msrun_e1K9_IeZXtwyfNAG: 0.19 req/s, p95 TTFT 781.2 msrun_Bn_ziSK4wG_mrv1l: 0.73 req/s, p95 TTFT 14,950.3 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=476.91,938.3 ms48.47 ms13,403.4 ms0run_G3h2UvDdN-VkaEfs
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=476.71,937.6 ms48.62 ms13,419.6 ms0run_Zr0ZfH1dmdqT_kDb
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=477.31,936.1 ms48.3 ms13,352.5 ms0run_uLqgfmu_6Hy5cn1t
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192c=4139.33,061.3 ms25.64 ms7,366.8 ms0run_vLZufKb1K1zn-elA
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048c=4142.91,514.1 ms24.33 ms7,345.5 ms0run_ZmPTn050k3QQME9i
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4127.23,111 ms28.47 ms8,087 ms0run_koIl7B9e_gXcr-EG
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4127.32,338.4 ms28.36 ms8,066.8 ms0run_Rp8ecJixIoeV5Bsa
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8185.64,685.7 ms40.08 ms11,848.6 ms0run_3bSQLP_-3auTpmvN
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4127.23,109.8 ms28.45 ms8,085.3 ms0run_FW-OGBWRqx7n-e74
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=3218636,262.7 ms40.13 ms45,629.6 ms0run_BISUbbE4IuZZX-_J
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=282.41,550.9 ms21.24 ms6,213.3 ms0run_ppah1fyumJuYwaIL
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=149.6780.3 ms17.2 ms5,164.9 ms0run_aJyhC38O-J7nURZ3
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=149.6780.9 ms17.2 ms5,165 ms0run_1qf9IfT3FUYv7Z16
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=149.6781.2 ms17.2 ms5,166.2 ms0run_e1K9_IeZXtwyfNAG
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=1618614,950.3 ms40.12 ms22,881.6 ms0run_Bn_ziSK4wG_mrv1l
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
76.9
p95 TTFT / TPOT
1,938.3 ms / 48.47 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
76.7
p95 TTFT / TPOT
1,937.6 ms / 48.62 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
77.3
p95 TTFT / TPOT
1,936.1 ms / 48.3 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192
Test load
c=4
Req/s
Output tok/s
139.3
p95 TTFT / TPOT
3,061.3 ms / 25.64 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048
Test load
c=4
Req/s
Output tok/s
142.9
p95 TTFT / TPOT
1,514.1 ms / 24.33 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
127.2
p95 TTFT / TPOT
3,111 ms / 28.47 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
127.3
p95 TTFT / TPOT
2,338.4 ms / 28.36 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
185.6
p95 TTFT / TPOT
4,685.7 ms / 40.08 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
127.2
p95 TTFT / TPOT
3,109.8 ms / 28.45 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
186
p95 TTFT / TPOT
36,262.7 ms / 40.13 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
82.4
p95 TTFT / TPOT
1,550.9 ms / 21.24 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
49.6
p95 TTFT / TPOT
780.3 ms / 17.2 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
49.6
p95 TTFT / TPOT
780.9 ms / 17.2 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
49.6
p95 TTFT / TPOT
781.2 ms / 17.2 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
186
p95 TTFT / TPOT
14,950.3 ms / 40.12 ms