nvidia/Gemma-4-26B-A4B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Decoding method
Speculative decoding
Workload
8192 → 256
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)02854.403run_3O11VjGGMdK1OGlI: 2.1 req/s, p95 TTFT 1,302.1 msrun_4DZyW3TYsj6RWqJs: 1.51 req/s, p95 TTFT 544.6 msrun_Vrt0oihhgg-zhx40: 1.53 req/s, p95 TTFT 532.4 msrun_5NjW6yh4JtubXD42: 1.52 req/s, p95 TTFT 525.5 msrun_tR8XEjIX4Em8gXTx: 0.57 req/s, p95 TTFT 280.7 msrun_ymGsK6WGK058DD9U: 2.65 req/s, p95 TTFT 2,854.4 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8538.61,302.1 ms23.19 ms6,260.7 ms0run_3O11VjGGMdK1OGlI
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4386.7544.6 ms15.73 ms4,307 ms0run_4DZyW3TYsj6RWqJs
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4390.5532.4 ms15.67 ms4,306.6 ms0run_Vrt0oihhgg-zhx40
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4389.7525.5 ms15.64 ms4,310.6 ms0run_5NjW6yh4JtubXD42
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1145.4280.7 ms10.45 ms2,934.9 ms0run_tR8XEjIX4Em8gXTx
vllm 0.28.0Speculativedtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=16678.62,854.4 ms32.84 ms9,870.5 ms0run_ymGsK6WGK058DD9U
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
538.6
p95 TTFT / TPOT
1,302.1 ms / 23.19 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
386.7
p95 TTFT / TPOT
544.6 ms / 15.73 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
390.5
p95 TTFT / TPOT
532.4 ms / 15.67 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
389.7
p95 TTFT / TPOT
525.5 ms / 15.64 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
145.4
p95 TTFT / TPOT
280.7 ms / 10.45 ms
Runtime
vllm 0.28.0Speculative decoding
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
678.6
p95 TTFT / TPOT
2,854.4 ms / 32.84 ms