unsloth/Qwen3.8-27B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
4096 → 1
Runs
13 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)023600.203run_nhl0lq4M7zMog-5d: 2.76 req/s, p95 TTFT 5,801.2 msrun_kEzv-CyYzjwbFLr4: 2.84 req/s, p95 TTFT 5,647 msrun_YkKQ8BGwQb-TG2z3: 2.71 req/s, p95 TTFT 5,902.9 msrun_S2IwINCj6GI8Oyds: 2.72 req/s, p95 TTFT 5,905.3 msrun_Ci4yh0brPF5LNFR8: 2.74 req/s, p95 TTFT 2,926 msrun_E0yvte6DTqZAFfJO: 2.71 req/s, p95 TTFT 23,600.2 msrun_ia-q20D3XySMhbuf: 2.75 req/s, p95 TTFT 1,454.1 msrun_eSqh0cm1G0zWWv4P: 2.71 req/s, p95 TTFT 11,802.8 msrun_9r9bLOWZICk5HCh3: 2.78 req/s, p95 TTFT 721.5 msrun_ru3-pldXjwU9fK_7: 2.78 req/s, p95 TTFT 721.8 msrun_IXtxWD5jy-o_YLLk: 2.78 req/s, p95 TTFT 721.5 msrun_jdFEbq6F0d2MRgxf: 2.74 req/s, p95 TTFT 367.9 msrun_hkL6LXZoY3fZ7hL7: 2.72 req/s, p95 TTFT 5,900.6 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192c=162.85,801.2 ms0 ms5,801.2 ms0run_nhl0lq4M7zMog-5d
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048c=162.85,647 ms0 ms5,647 ms0run_kEzv-CyYzjwbFLr4
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162.75,902.9 ms0 ms5,902.9 ms0run_YkKQ8BGwQb-TG2z3
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162.75,905.3 ms0 ms5,905.3 ms0run_S2IwINCj6GI8Oyds
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=82.72,926 ms0 ms2,926 ms0run_Ci4yh0brPF5LNFR8
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=642.723,600.2 ms0 ms23,600.2 ms0run_E0yvte6DTqZAFfJO
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=42.81,454.1 ms0 ms1,454.1 ms0run_ia-q20D3XySMhbuf
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=322.711,802.8 ms0 ms11,802.8 ms0run_eSqh0cm1G0zWWv4P
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=22.8721.5 ms0 ms721.5 ms0run_9r9bLOWZICk5HCh3
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=22.8721.8 ms0 ms721.8 ms0run_ru3-pldXjwU9fK_7
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=22.8721.5 ms0 ms721.5 ms0run_IXtxWD5jy-o_YLLk
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=12.7367.9 ms0 ms367.9 ms0run_jdFEbq6F0d2MRgxf
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=162.75,900.6 ms0 ms5,900.6 ms0run_hkL6LXZoY3fZ7hL7
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192
Test load
c=16
Req/s
Output tok/s
2.8
p95 TTFT / TPOT
5,801.2 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048
Test load
c=16
Req/s
Output tok/s
2.8
p95 TTFT / TPOT
5,647 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2.7
p95 TTFT / TPOT
5,902.9 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2.7
p95 TTFT / TPOT
5,905.3 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
2.7
p95 TTFT / TPOT
2,926 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
2.7
p95 TTFT / TPOT
23,600.2 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
2.8
p95 TTFT / TPOT
1,454.1 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
2.7
p95 TTFT / TPOT
11,802.8 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
2.8
p95 TTFT / TPOT
721.5 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
2.8
p95 TTFT / TPOT
721.8 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
2.8
p95 TTFT / TPOT
721.5 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
2.7
p95 TTFT / TPOT
367.9 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2.7
p95 TTFT / TPOT
5,900.6 ms / 0 ms