unsloth/Qwen3.8-27B-NVFP4

Hardware
4× NVIDIA GeForce RTX 5090
Workload
1024 → 256
Runs
8 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)012221.0010run_EpUX0RC2nD27Favg: 9.51 req/s, p95 TTFT 5,698.2 msrun_ZijkPzwF5OpCpk96: 3.41 req/s, p95 TTFT 356.8 msrun_fnPS9SkKESxDA8RX: 1.05 req/s, p95 TTFT 95.7 msrun_arKAxEVDmDFnwMzM: 0.53 req/s, p95 TTFT 95 msrun_xlrn81sRTyav7HWD: 9.46 req/s, p95 TTFT 12,221 msrun_F5yp10AfL5aiaDng: 6.09 req/s, p95 TTFT 719.3 msrun_M-dDOg-ZxrerXm3Y: 1.88 req/s, p95 TTFT 178 msrun_xm_EmAr3djdWaetd: 0.26 req/s, p95 TTFT 95 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4c=642,434.65,698.2 ms20.12 ms10,957.2 ms0run_EpUX0RC2nD27Favg
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4c=16873.6356.8 ms17.19 ms4,611.9 ms0run_ZijkPzwF5OpCpk96
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4c=4269.595.7 ms14.54 ms3,803.2 ms0run_fnPS9SkKESxDA8RX
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4c=2134.895 ms14.53 ms3,799.4 ms0run_arKAxEVDmDFnwMzM
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4c=1282,421.712,221 ms19.32 ms16,634.3 ms0run_xlrn81sRTyav7HWD
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4c=321,559.1719.3 ms17.59 ms4,914.9 ms0run_F5yp10AfL5aiaDng
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4c=8480.4178 ms16.23 ms4,270.2 ms0run_M-dDOg-ZxrerXm3Y
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4c=167.495 ms14.53 ms3,799.3 ms0run_xm_EmAr3djdWaetd
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4
Test load
c=64
Req/s
Output tok/s
2,434.6
p95 TTFT / TPOT
5,698.2 ms / 20.12 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4
Test load
c=16
Req/s
Output tok/s
873.6
p95 TTFT / TPOT
356.8 ms / 17.19 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4
Test load
c=4
Req/s
Output tok/s
269.5
p95 TTFT / TPOT
95.7 ms / 14.54 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4
Test load
c=2
Req/s
Output tok/s
134.8
p95 TTFT / TPOT
95 ms / 14.53 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4
Test load
c=128
Req/s
Output tok/s
2,421.7
p95 TTFT / TPOT
12,221 ms / 19.32 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4
Test load
c=32
Req/s
Output tok/s
1,559.1
p95 TTFT / TPOT
719.3 ms / 17.59 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4
Test load
c=8
Req/s
Output tok/s
480.4
p95 TTFT / TPOT
178 ms / 16.23 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4
Test load
c=1
Req/s
Output tok/s
67.4
p95 TTFT / TPOT
95 ms / 14.53 ms