RedHatAI/gemma-4-31B-it-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Decoding method
Speculative decoding
Workload
1024 → 256
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)03503.902run_5u3G5xljYyHUsW3A: 2.28 req/s, p95 TTFT 233.7 msrun_lv3SFf3eDuiyzKbj: 2.26 req/s, p95 TTFT 173 msrun_A3x-PKoAIV6BLxE_: 2.29 req/s, p95 TTFT 232.3 msrun_fuWVMLGSo7C5zAke: 1.24 req/s, p95 TTFT 167.9 msrun_kGHHDR1xyuT-Wybv: 0.35 req/s, p95 TTFT 80.1 msrun_5vMWKLn4mcUSzGiI: 2.49 req/s, p95 TTFT 3,503.9 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8583.9233.7 ms20.52 ms5,364.7 ms0run_5u3G5xljYyHUsW3A
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8579.8173 ms20.83 ms5,445.9 ms0run_lv3SFf3eDuiyzKbj
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8585.4232.3 ms20.46 ms5,361.8 ms0run_A3x-PKoAIV6BLxE_
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4316.7167.9 ms19.03 ms4,961.9 ms0run_fuWVMLGSo7C5zAke
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=188.680.1 ms17.43 ms4,514.7 ms0run_kGHHDR1xyuT-Wybv
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=16637.73,503.9 ms21.27 ms8,447.2 ms0run_5vMWKLn4mcUSzGiI
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
583.9
p95 TTFT / TPOT
233.7 ms / 20.52 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
579.8
p95 TTFT / TPOT
173 ms / 20.83 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
585.4
p95 TTFT / TPOT
232.3 ms / 20.46 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
316.7
p95 TTFT / TPOT
167.9 ms / 19.03 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
88.6
p95 TTFT / TPOT
80.1 ms / 17.43 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
637.7
p95 TTFT / TPOT
3,503.9 ms / 21.27 ms