RedHatAI/gemma-4-31B-it-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Decoding method
Speculative decoding
Workload
8192 → 256
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)035898.800run_F8dA9WhR0aKxkSTr: 0.42 req/s, p95 TTFT 16,554.2 msrun_miuLewM6zZB00mBP: 0.42 req/s, p95 TTFT 7,587.2 msrun_nHWgEfbii12SwXf7: 0.25 req/s, p95 TTFT 702.5 msrun_MzxyZ1RYLi8O29jg: 0.25 req/s, p95 TTFT 702.5 msrun_3gCzHWXBIiRANi3T: 0.25 req/s, p95 TTFT 702.3 msrun_NOUWNxg5Gb-4NBJW: 0.42 req/s, p95 TTFT 35,898.8 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8108.216,554.2 ms26.68 ms22,242.3 ms0run_F8dA9WhR0aKxkSTr
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4108.37,587.2 ms26.98 ms11,853.3 ms0run_miuLewM6zZB00mBP
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=164.5702.5 ms21.34 ms6,139.9 ms0run_nHWgEfbii12SwXf7
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=164.5702.5 ms21.35 ms6,140.3 ms0run_MzxyZ1RYLi8O29jg
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=163.9702.3 ms21.36 ms6,140.9 ms0run_3gCzHWXBIiRANi3T
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=16108.235,898.8 ms27.01 ms40,422.9 ms0run_NOUWNxg5Gb-4NBJW
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
108.2
p95 TTFT / TPOT
16,554.2 ms / 26.68 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
108.3
p95 TTFT / TPOT
7,587.2 ms / 26.98 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
64.5
p95 TTFT / TPOT
702.5 ms / 21.34 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
64.5
p95 TTFT / TPOT
702.5 ms / 21.35 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
63.9
p95 TTFT / TPOT
702.3 ms / 21.36 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
108.2
p95 TTFT / TPOT
35,898.8 ms / 27.01 ms