nvidia/Gemma-4-26B-A4B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Decoding method
Speculative decoding
Workload
256 → 1024
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)0130.702run_esWUo9CST3MGMHZ8: 1.4 req/s, p95 TTFT 56.5 msrun_kwpRO_STjjE9Sz47: 0.81 req/s, p95 TTFT 37.6 msrun_w_FQNY70DvqZ6wRe: 0.23 req/s, p95 TTFT 25.5 msrun_PAFgf6AWVUOSsYML: 2.43 req/s, p95 TTFT 128.5 msrun_Gc86zz2D9nox5WCh: 2.47 req/s, p95 TTFT 130.7 msrun_50pjM5e6rynjy6Bx: 2.48 req/s, p95 TTFT 122.5 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=81,436.256.5 ms9.35 ms9,599.1 ms0run_esWUo9CST3MGMHZ8
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4828.237.6 ms8.6 ms8,828.6 ms0run_kwpRO_STjjE9Sz47
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1233.925.5 ms7.88 ms8,084.8 ms0run_w_FQNY70DvqZ6wRe
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,487.3128.5 ms10.25 ms10,546.7 ms0run_PAFgf6AWVUOSsYML
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,532.2130.7 ms10.09 ms10,431.1 ms0run_Gc86zz2D9nox5WCh
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,535.8122.5 ms10.14 ms10,415.2 ms0run_50pjM5e6rynjy6Bx
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,436.2
p95 TTFT / TPOT
56.5 ms / 9.35 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
828.2
p95 TTFT / TPOT
37.6 ms / 8.6 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
233.9
p95 TTFT / TPOT
25.5 ms / 7.88 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,487.3
p95 TTFT / TPOT
128.5 ms / 10.25 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,532.2
p95 TTFT / TPOT
130.7 ms / 10.09 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,535.8
p95 TTFT / TPOT
122.5 ms / 10.14 ms