nvidia/Gemma-4-26B-A4B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
256 → 1024
Runs
20 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)0277.103run_t7Iu5TOPsXPnDikS: 1.06 req/s, p95 TTFT 64.7 msrun_xz9uDlcsPlB2r13n: 0.56 req/s, p95 TTFT 44.8 msrun_b_MS9YNaDpi8t9WM: 0.18 req/s, p95 TTFT 21 msrun_RsRL9UjvCKalIGrL: 1.83 req/s, p95 TTFT 126.7 msrun_Yz3zGJ-BAf8eX5mn: 1.82 req/s, p95 TTFT 123.5 msrun_uKuCW_zsbNJ-GEc3: 1.82 req/s, p95 TTFT 119.4 msrun_-QtO8qbCMEtTECXD: 1.49 req/s, p95 TTFT 149.9 msrun_O0V8gHo250RvWWoA: 1.59 req/s, p95 TTFT 157.5 msrun_1-WOTb0gNIhROIjr: 1.59 req/s, p95 TTFT 159.6 msrun_hYyotGJkXUmQ3NN8: 1.59 req/s, p95 TTFT 152.3 msrun_YtyfoUjszok3d7KV: 1.59 req/s, p95 TTFT 161.4 msrun_Q2lyHP2CoVgcmshT: 0.87 req/s, p95 TTFT 157.8 msrun_9r7-urqFhBDfhOqD: 0.91 req/s, p95 TTFT 84.8 msrun_mWGI8mktIL1NM3BB: 0.48 req/s, p95 TTFT 49.6 msrun_Vx88OjthW9Ro4c6-: 2.73 req/s, p95 TTFT 277 msrun__ZFW4N9U4h9bQJBV: 2.74 req/s, p95 TTFT 277.1 msrun_4fAlQFToTIjmIWDU: 2.67 req/s, p95 TTFT 270.9 msrun_FtCm5hyLoma07WXR: 0.26 req/s, p95 TTFT 34.7 msrun_LJszFB4TL0FDzFAl: 0.15 req/s, p95 TTFT 21.5 msrun_-B3jOS_FlpSvJEVe: 1.59 req/s, p95 TTFT 160.5 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=81,084.264.7 ms7.55 ms7,786.2 ms0run_t7Iu5TOPsXPnDikS
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4573.544.8 ms7.05 ms7,254.8 ms0run_xz9uDlcsPlB2r13n
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1186.321 ms5.36 ms5,501.4 ms0run_b_MS9YNaDpi8t9WM
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=161,876.3126.7 ms8.54 ms8,798.6 ms0run_RsRL9UjvCKalIGrL
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=161,868.6123.5 ms8.65 ms8,905.9 ms0run_Yz3zGJ-BAf8eX5mn
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=161,861.8119.4 ms8.77 ms9,025.1 ms0run_uKuCW_zsbNJ-GEc3
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,528.3149.9 ms10.52 ms10,888.9 ms0run_-QtO8qbCMEtTECXD
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,626.6157.5 ms9.99 ms10,303.8 ms0run_O0V8gHo250RvWWoA
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,626.1159.6 ms9.98 ms10,357 ms0run_1-WOTb0gNIhROIjr
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,629.8152.3 ms9.97 ms10,335.4 ms0run_hYyotGJkXUmQ3NN8
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,629.1161.4 ms9.97 ms10,342.9 ms0run_YtyfoUjszok3d7KV
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=16890.1157.8 ms18.02 ms18,557.5 ms0run_Q2lyHP2CoVgcmshT
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8926.884.8 ms8.75 ms9,024.4 ms0run_9r7-urqFhBDfhOqD
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4489.849.6 ms8.35 ms8,590.7 ms0run_mWGI8mktIL1NM3BB
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=322,798.9277 ms11.39 ms11,906 ms0run_Vx88OjthW9Ro4c6-
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=322,805.9277.1 ms11.3 ms11,827.7 ms0run__ZFW4N9U4h9bQJBV
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=322,735.5270.9 ms11.7 ms12,096.4 ms0run_4fAlQFToTIjmIWDU
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=2269.834.7 ms7.48 ms7,677.3 ms0run_FtCm5hyLoma07WXR
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=1153.721.5 ms6.49 ms6,665 ms0run_LJszFB4TL0FDzFAl
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,630.6160.5 ms9.96 ms10,340.3 ms0run_-B3jOS_FlpSvJEVe
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,084.2
p95 TTFT / TPOT
64.7 ms / 7.55 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
573.5
p95 TTFT / TPOT
44.8 ms / 7.05 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
186.3
p95 TTFT / TPOT
21 ms / 5.36 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,876.3
p95 TTFT / TPOT
126.7 ms / 8.54 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,868.6
p95 TTFT / TPOT
123.5 ms / 8.65 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,861.8
p95 TTFT / TPOT
119.4 ms / 8.77 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,528.3
p95 TTFT / TPOT
149.9 ms / 10.52 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,626.6
p95 TTFT / TPOT
157.5 ms / 9.99 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,626.1
p95 TTFT / TPOT
159.6 ms / 9.98 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,629.8
p95 TTFT / TPOT
152.3 ms / 9.97 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,629.1
p95 TTFT / TPOT
161.4 ms / 9.97 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
890.1
p95 TTFT / TPOT
157.8 ms / 18.02 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
926.8
p95 TTFT / TPOT
84.8 ms / 8.75 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
489.8
p95 TTFT / TPOT
49.6 ms / 8.35 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
2,798.9
p95 TTFT / TPOT
277 ms / 11.39 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
2,805.9
p95 TTFT / TPOT
277.1 ms / 11.3 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
2,735.5
p95 TTFT / TPOT
270.9 ms / 11.7 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
269.8
p95 TTFT / TPOT
34.7 ms / 7.48 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
153.7
p95 TTFT / TPOT
21.5 ms / 6.49 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,630.6
p95 TTFT / TPOT
160.5 ms / 9.96 ms