nvidia/Gemma-4-26B-A4B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
8192 → 256
Runs
22 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)08045.303run_0ev7P4GeOWLs_m62: 2.02 req/s, p95 TTFT 1,404.8 msrun_KTK-Amc3Aiaikyo0: 2.04 req/s, p95 TTFT 1,418.2 msrun_1oFE7AvKzrY44oL9: 1.39 req/s, p95 TTFT 942.2 msrun_XzGffOQR7Nq50OXh: 1.4 req/s, p95 TTFT 942.3 msrun_ZFEhJpeNeX446vm2: 1.39 req/s, p95 TTFT 946.8 msrun_BiRk0mNJhABdt3UH: 0.57 req/s, p95 TTFT 257.2 msrun_jzNnA40CG_fZeX_W: 2.59 req/s, p95 TTFT 2,593.5 msrun_n2sOjqer3hI_4sPo: 1.67 req/s, p95 TTFT 1,973.4 msrun_K5x47IoCipiOQYpS: 1.73 req/s, p95 TTFT 2,136.4 msrun_h2k5qLlXM38D_bJE: 1.73 req/s, p95 TTFT 2,070.8 msrun_SzSzrf6m4zCyxSvZ: 1.73 req/s, p95 TTFT 2,071.4 msrun_fzNlvr7gkcxeaBNA: 1.73 req/s, p95 TTFT 2,072.5 msrun_M84cqVLu1unP9nQ7: 1.16 req/s, p95 TTFT 1,879.5 msrun_Orr1njfPfCR-2rmQ: 1.72 req/s, p95 TTFT 1,500.2 msrun_fNYUxNJdPqizEmAh: 1.72 req/s, p95 TTFT 2,073.4 msrun_JSiA-ksRrXJoHBot: 1.2 req/s, p95 TTFT 1,200.7 msrun_Bi4oD5zDVQQrhQtS: 2.49 req/s, p95 TTFT 8,045.3 msrun_KhFf16YWKJMPHReT: 0.78 req/s, p95 TTFT 612.7 msrun_gQFNxnoIVQ3fCvid: 0.78 req/s, p95 TTFT 613.2 msrun_31EZhVxCkfKQwAzl: 0.78 req/s, p95 TTFT 613.8 msrun_5oSPUMap5ntW9ZaR: 0.49 req/s, p95 TTFT 324 msrun_rZgOPHFTHsvaPA4n: 2.16 req/s, p95 TTFT 3,309.1 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8517.21,404.8 ms14.52 ms4,228.9 ms0run_0ev7P4GeOWLs_m62
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8521.11,418.2 ms14.42 ms4,195.1 ms0run_KTK-Amc3Aiaikyo0
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4356.8942.2 ms10.32 ms3,073 ms0run_1oFE7AvKzrY44oL9
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4357.2942.3 ms10.3 ms3,072 ms0run_XzGffOQR7Nq50OXh
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4356.8946.8 ms10.33 ms2,920.6 ms0run_ZFEhJpeNeX446vm2
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1145.8257.2 ms5.93 ms1,765.6 ms0run_BiRk0mNJhABdt3UH
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=16661.92,593.5 ms22.43 ms7,055.9 ms0run_jzNnA40CG_fZeX_W
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8427.41,973.4 ms17.5 ms5,066.8 ms0run_n2sOjqer3hI_4sPo
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8441.92,136.4 ms16.85 ms4,950.5 ms0run_K5x47IoCipiOQYpS
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8441.92,070.8 ms16.84 ms4,696.3 ms0run_h2k5qLlXM38D_bJE
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=84422,071.4 ms16.85 ms4,919.5 ms0run_SzSzrf6m4zCyxSvZ
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8441.62,072.5 ms16.92 ms4,962.8 ms0run_fzNlvr7gkcxeaBNA
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8295.91,879.5 ms25.74 ms7,291.9 ms0run_M84cqVLu1unP9nQ7
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8439.71,500.2 ms16.92 ms4,930.3 ms0run_Orr1njfPfCR-2rmQ
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8441.52,073.4 ms16.88 ms4,902.5 ms0run_fNYUxNJdPqizEmAh
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4306.51,200.7 ms11.89 ms3,405 ms0run_JSiA-ksRrXJoHBot
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=32637.18,045.3 ms47.57 ms18,126.9 ms0run_Bi4oD5zDVQQrhQtS
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=2200.8612.7 ms8.82 ms2,586.8 ms0run_KhFf16YWKJMPHReT
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=2200.9613.2 ms8.82 ms2,585.9 ms0run_gQFNxnoIVQ3fCvid
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=2200.7613.8 ms8.83 ms2,589.9 ms0run_31EZhVxCkfKQwAzl
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=1126.1324 ms6.73 ms2,037.5 ms0run_5oSPUMap5ntW9ZaR
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=16553.63,309.1 ms27.46 ms7,996.3 ms0run_rZgOPHFTHsvaPA4n
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
517.2
p95 TTFT / TPOT
1,404.8 ms / 14.52 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
521.1
p95 TTFT / TPOT
1,418.2 ms / 14.42 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
356.8
p95 TTFT / TPOT
942.2 ms / 10.32 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
357.2
p95 TTFT / TPOT
942.3 ms / 10.3 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
356.8
p95 TTFT / TPOT
946.8 ms / 10.33 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
145.8
p95 TTFT / TPOT
257.2 ms / 5.93 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
661.9
p95 TTFT / TPOT
2,593.5 ms / 22.43 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
427.4
p95 TTFT / TPOT
1,973.4 ms / 17.5 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
441.9
p95 TTFT / TPOT
2,136.4 ms / 16.85 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
441.9
p95 TTFT / TPOT
2,070.8 ms / 16.84 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
442
p95 TTFT / TPOT
2,071.4 ms / 16.85 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
441.6
p95 TTFT / TPOT
2,072.5 ms / 16.92 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
295.9
p95 TTFT / TPOT
1,879.5 ms / 25.74 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
439.7
p95 TTFT / TPOT
1,500.2 ms / 16.92 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
441.5
p95 TTFT / TPOT
2,073.4 ms / 16.88 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
306.5
p95 TTFT / TPOT
1,200.7 ms / 11.89 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
637.1
p95 TTFT / TPOT
8,045.3 ms / 47.57 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
200.8
p95 TTFT / TPOT
612.7 ms / 8.82 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
200.9
p95 TTFT / TPOT
613.2 ms / 8.82 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
200.7
p95 TTFT / TPOT
613.8 ms / 8.83 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
126.1
p95 TTFT / TPOT
324 ms / 6.73 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
553.6
p95 TTFT / TPOT
3,309.1 ms / 27.46 ms