unsloth/Qwen3.8-27B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
1024 → 256
Runs
34 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)049461.203run_CLR-g8H9yC2C41by: 0.77 req/s, p95 TTFT 877.5 msrun_CwSw25tLowbEw2I8: 0.77 req/s, p95 TTFT 877.3 msrun_a0pk4mKBwUBHmaHn: 0.76 req/s, p95 TTFT 876.8 msrun_0evI2JocYd2B1oqK: 1.77 req/s, p95 TTFT 26,291.5 msrun_CZeGb8kvrTCmiIdE: 0.4 req/s, p95 TTFT 404.1 msrun_QqFEtloCGtxkSHLj: 0.11 req/s, p95 TTFT 101.8 msrun_e91MMwTnLumOaOYA: 1.4 req/s, p95 TTFT 1,802.1 msrun_nNx-l8nHa-I-E6yr: 0.78 req/s, p95 TTFT 869.2 msrun_4-PUEdtiWinXqShE: 0.78 req/s, p95 TTFT 869.6 msrun_FyK7qznFVYbBh7ZZ: 0.77 req/s, p95 TTFT 869.2 msrun_Pcp_AFebl4zE4IQo: 0.41 req/s, p95 TTFT 400.4 msrun_8pivK0NCzjf-OeyO: 0.11 req/s, p95 TTFT 98.1 msrun_vrU-xRDsORs1dvnZ: 1.59 req/s, p95 TTFT 704.1 msrun_wXAyOwjQQuNcHQD2: 1.64 req/s, p95 TTFT 714.5 msrun_i8fQVS0OoIxBExGA: 1.65 req/s, p95 TTFT 713.5 msrun_iJOT0dJ3x7zeMLRl: 1.64 req/s, p95 TTFT 446.5 msrun_XeUTL-h0ITb5FZok: 1.47 req/s, p95 TTFT 720.9 msrun_rF9JXzLs1PKkGtKM: 0.78 req/s, p95 TTFT 897.8 msrun_GIkxM5zWpKCn4Fga: 1.5 req/s, p95 TTFT 720 msrun_AbbjecJk3__Tqolp: 1.47 req/s, p95 TTFT 720 msrun_9jI5XKqrH6d-rSP_: 1.47 req/s, p95 TTFT 719.8 msrun_Gj4lEnJwN751esg0: 2.93 req/s, p95 TTFT 21,576.5 msrun_2p_taQ3E3OqiBJq-: 0.77 req/s, p95 TTFT 364.6 msrun_xbmz6Ej0JLYfn44k: 2.94 req/s, p95 TTFT 7,814.2 msrun_EAsnutWcYbIcaPlq: 0.22 req/s, p95 TTFT 90.8 msrun_pbO5otSeyWNSEja-: 2.55 req/s, p95 TTFT 1,450 msrun_OtKDYvoQb5pkSPzk: 2.39 req/s, p95 TTFT 22,564.4 msrun_GV5rcoyhkNI_f_kW: 2.37 req/s, p95 TTFT 5,337.6 msrun_bsnkPBboAyCxvfPA: 0.89 req/s, p95 TTFT 349.3 msrun_QtmoRzJUpBXMlUTm: 0.47 req/s, p95 TTFT 186.7 msrun_wVEeCEFKlTbhUYqQ: 2.39 req/s, p95 TTFT 49,461.2 msrun_ElGXRAFgfF3ZuQ8u: 2.36 req/s, p95 TTFT 11,836.8 msrun_i_PVnlCYJEEHRUUo: 1.68 req/s, p95 TTFT 702.4 msrun_GufrPFUj3gf9V8QH: 0.26 req/s, p95 TTFT 93.9 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8196.1877.5 ms40.55 ms10,500.2 ms0run_CLR-g8H9yC2C41by
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8196.9877.3 ms40.28 ms10,516.1 ms0run_CwSw25tLowbEw2I8
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8194.6876.8 ms40.91 ms10,602.4 ms0run_a0pk4mKBwUBHmaHn
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=64452.826,291.5 ms45.36 ms36,941.9 ms0run_0evI2JocYd2B1oqK
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4103.4404.1 ms38.6 ms10,027.7 ms0run_CZeGb8kvrTCmiIdE
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=127.5101.8 ms36.72 ms9,464.2 ms0run_QqFEtloCGtxkSHLj
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=16357.81,802.1 ms44.08 ms11,547.3 ms0run_e91MMwTnLumOaOYA
vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8200.7869.2 ms39.54 ms10,316.6 ms0run_nNx-l8nHa-I-E6yr
vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8199.7869.6 ms39.86 ms10,324.9 ms0run_4-PUEdtiWinXqShE
vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8198.1869.2 ms40.14 ms10,410.8 ms0run_FyK7qznFVYbBh7ZZ
vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4105400.4 ms37.96 ms9,837.3 ms0run_Pcp_AFebl4zE4IQo
vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=127.898.1 ms36.07 ms9,295.2 ms0run_8pivK0NCzjf-OeyO
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8407.5704.1 ms19.3 ms5,027.5 ms0run_vrU-xRDsORs1dvnZ
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192c=8420.9714.5 ms18.63 ms4,867.9 ms0run_wXAyOwjQQuNcHQD2
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=32768c=8423.2713.5 ms18.53 ms4,841.5 ms0run_i8fQVS0OoIxBExGA
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048c=8420.6446.5 ms18.59 ms4,926.6 ms0run_iJOT0dJ3x7zeMLRl
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8376.5720.9 ms20.86 ms5,441.4 ms0run_XeUTL-h0ITb5FZok
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8200.9897.8 ms39.37 ms10,296.2 ms0run_rF9JXzLs1PKkGtKM
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8384720 ms20.44 ms5,333.9 ms0run_GIkxM5zWpKCn4Fga
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8376.4720 ms20.86 ms5,441.5 ms0run_AbbjecJk3__Tqolp
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8376.4719.8 ms20.86 ms5,442.1 ms0run_9jI5XKqrH6d-rSP_
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=64751.321,576.5 ms21.89 ms26,964 ms0run_Gj4lEnJwN751esg0
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4197.1364.6 ms19.9 ms5,197.3 ms0run_2p_taQ3E3OqiBJq-
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=32752.47,814.2 ms22.65 ms13,469.3 ms0run_xbmz6Ej0JLYfn44k
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=156.390.8 ms17.49 ms4,551 ms0run_EAsnutWcYbIcaPlq
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=16652.51,450 ms24.13 ms6,283.9 ms0run_pbO5otSeyWNSEja-
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=64612.422,564.4 ms18.85 ms26,874.8 ms0run_OtKDYvoQb5pkSPzk
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=16605.75,337.6 ms19.98 ms10,232.2 ms0run_GV5rcoyhkNI_f_kW
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=4227.1349.3 ms16.8 ms4,510.7 ms0run_bsnkPBboAyCxvfPA
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=2120.9186.7 ms16.04 ms4,237.4 ms0run_QtmoRzJUpBXMlUTm
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=128612.249,461.2 ms20.62 ms53,773.5 ms0run_wVEeCEFKlTbhUYqQ
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=32603.311,836.8 ms18.71 ms16,101.3 ms0run_ElGXRAFgfF3ZuQ8u
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=8430.4702.4 ms17.12 ms4,759.5 ms0run_i_PVnlCYJEEHRUUo
vllm 0.27.1dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=167.493.9 ms14.54 ms3,799.2 ms0run_GufrPFUj3gf9V8QH
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
196.1
p95 TTFT / TPOT
877.5 ms / 40.55 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
196.9
p95 TTFT / TPOT
877.3 ms / 40.28 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
194.6
p95 TTFT / TPOT
876.8 ms / 40.91 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
452.8
p95 TTFT / TPOT
26,291.5 ms / 45.36 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
103.4
p95 TTFT / TPOT
404.1 ms / 38.6 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
27.5
p95 TTFT / TPOT
101.8 ms / 36.72 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
357.8
p95 TTFT / TPOT
1,802.1 ms / 44.08 ms
Runtime
vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
200.7
p95 TTFT / TPOT
869.2 ms / 39.54 ms
Runtime
vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
199.7
p95 TTFT / TPOT
869.6 ms / 39.86 ms
Runtime
vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
198.1
p95 TTFT / TPOT
869.2 ms / 40.14 ms
Runtime
vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
105
p95 TTFT / TPOT
400.4 ms / 37.96 ms
Runtime
vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
27.8
p95 TTFT / TPOT
98.1 ms / 36.07 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
407.5
p95 TTFT / TPOT
704.1 ms / 19.3 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192
Test load
c=8
Req/s
Output tok/s
420.9
p95 TTFT / TPOT
714.5 ms / 18.63 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=32768
Test load
c=8
Req/s
Output tok/s
423.2
p95 TTFT / TPOT
713.5 ms / 18.53 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048
Test load
c=8
Req/s
Output tok/s
420.6
p95 TTFT / TPOT
446.5 ms / 18.59 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
376.5
p95 TTFT / TPOT
720.9 ms / 20.86 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
200.9
p95 TTFT / TPOT
897.8 ms / 39.37 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
384
p95 TTFT / TPOT
720 ms / 20.44 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
376.4
p95 TTFT / TPOT
720 ms / 20.86 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
376.4
p95 TTFT / TPOT
719.8 ms / 20.86 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
751.3
p95 TTFT / TPOT
21,576.5 ms / 21.89 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
197.1
p95 TTFT / TPOT
364.6 ms / 19.9 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
752.4
p95 TTFT / TPOT
7,814.2 ms / 22.65 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
56.3
p95 TTFT / TPOT
90.8 ms / 17.49 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
652.5
p95 TTFT / TPOT
1,450 ms / 24.13 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
612.4
p95 TTFT / TPOT
22,564.4 ms / 18.85 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
605.7
p95 TTFT / TPOT
5,337.6 ms / 19.98 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
227.1
p95 TTFT / TPOT
349.3 ms / 16.8 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
120.9
p95 TTFT / TPOT
186.7 ms / 16.04 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=128
Req/s
Output tok/s
612.2
p95 TTFT / TPOT
49,461.2 ms / 20.62 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
603.3
p95 TTFT / TPOT
11,836.8 ms / 18.71 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
430.4
p95 TTFT / TPOT
702.4 ms / 17.12 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
67.4
p95 TTFT / TPOT
93.9 ms / 14.54 ms