RedHatAI/gemma-4-31B-it-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
4096 → 1
Runs
14 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)019631.203run_VN0xETXC1kVOZ8qn: 3.25 req/s, p95 TTFT 5,545.5 msrun_g0j3r4vmqYxwBfLU: 3.36 req/s, p95 TTFT 5,067.9 msrun_dalomyvrRTgqIlRB: 3.4 req/s, p95 TTFT 4,716.6 msrun_tDX_t01PcyKcJ7z9: 3.04 req/s, p95 TTFT 5,929 msrun_YyY8n5N3qwxPOb49: 3.3 req/s, p95 TTFT 5,458.9 msrun_QB6SzaoYIZUwUFUo: 3.3 req/s, p95 TTFT 2,733.6 msrun_mIhU283IVJkeEj1A: 3.31 req/s, p95 TTFT 19,631.2 msrun__s7_4a5nR_n0dPd_: 3.3 req/s, p95 TTFT 1,212.9 msrun_t7tfTw_MEdKFLjEK: 3.31 req/s, p95 TTFT 9,965.7 msrun_AiY05m03gpyQEgXJ: 3.33 req/s, p95 TTFT 601.1 msrun_pRsWn_-1PQ-fCYdu: 3.34 req/s, p95 TTFT 600.4 msrun_TeLiatk-OWllZ0sE: 3.33 req/s, p95 TTFT 601.2 msrun_BYQx3ycuSD1aURdl: 3.29 req/s, p95 TTFT 306.3 msrun_ec66TMbQybDTOvZ7: 3.32 req/s, p95 TTFT 5,134.8 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=163.25,545.5 ms0 ms5,545.5 ms0run_VN0xETXC1kVOZ8qn
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192c=163.45,067.9 ms0 ms5,067.9 ms0run_g0j3r4vmqYxwBfLU
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048c=163.44,716.6 ms0 ms4,716.6 ms0run_dalomyvrRTgqIlRB
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=1635,929 ms0 ms5,929 ms0run_tDX_t01PcyKcJ7z9
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=163.35,458.9 ms0 ms5,458.9 ms0run_YyY8n5N3qwxPOb49
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=83.32,733.6 ms0 ms2,733.6 ms0run_QB6SzaoYIZUwUFUo
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=643.319,631.2 ms0 ms19,631.2 ms0run_mIhU283IVJkeEj1A
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=43.31,212.9 ms0 ms1,212.9 ms0run__s7_4a5nR_n0dPd_
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=323.39,965.7 ms0 ms9,965.7 ms0run_t7tfTw_MEdKFLjEK
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=23.3601.1 ms0 ms601.1 ms0run_AiY05m03gpyQEgXJ
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=23.3600.4 ms0 ms600.4 ms0run_pRsWn_-1PQ-fCYdu
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=23.3601.2 ms0 ms601.2 ms0run_TeLiatk-OWllZ0sE
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=13.3306.3 ms0 ms306.3 ms0run_BYQx3ycuSD1aURdl
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=163.35,134.8 ms0 ms5,134.8 ms0run_ec66TMbQybDTOvZ7
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
3.2
p95 TTFT / TPOT
5,545.5 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192
Test load
c=16
Req/s
Output tok/s
3.4
p95 TTFT / TPOT
5,067.9 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048
Test load
c=16
Req/s
Output tok/s
3.4
p95 TTFT / TPOT
4,716.6 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
3
p95 TTFT / TPOT
5,929 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=false expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
3.3
p95 TTFT / TPOT
5,458.9 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
3.3
p95 TTFT / TPOT
2,733.6 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
3.3
p95 TTFT / TPOT
19,631.2 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
3.3
p95 TTFT / TPOT
1,212.9 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
3.3
p95 TTFT / TPOT
9,965.7 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
3.3
p95 TTFT / TPOT
601.1 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
3.3
p95 TTFT / TPOT
600.4 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
3.3
p95 TTFT / TPOT
601.2 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
3.3
p95 TTFT / TPOT
306.3 ms / 0 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
3.3
p95 TTFT / TPOT
5,134.8 ms / 0 ms