google/gemma-4-12B-it

Hardware
1× NVIDIA GeForce RTX 5090
Workload
256 → 1024
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)0474.002run_vfoxggT7OGVrNnNz: 0.68 req/s, p95 TTFT 139.9 msrun_nHtJhvJi4NcGvPwh: 0.35 req/s, p95 TTFT 96.2 msrun_Nw1U2fDeSwJfLpPp: 2.22 req/s, p95 TTFT 474 msrun_JRoYK-1Zc-F0gKvw: 0.18 req/s, p95 TTFT 63 msrun_JoZqHLjbBj8U5Nft: 0.09 req/s, p95 TTFT 27.4 msrun_tFo55ZNrx3T150mL: 1.27 req/s, p95 TTFT 257.2 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=8698.3139.9 ms11.4 ms11,769.7 ms0run_vfoxggT7OGVrNnNz
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=4361.596.2 ms11.06 ms11,406.3 ms0run_nHtJhvJi4NcGvPwh
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=322,269.6474 ms14.02 ms14,446.5 ms0run_Nw1U2fDeSwJfLpPp
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=2185.463 ms10.77 ms11,060.5 ms0run_JRoYK-1Zc-F0gKvw
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=193.427.4 ms10.71 ms10,984 ms0run_JoZqHLjbBj8U5Nft
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=161,297.6257.2 ms12.26 ms12,641.2 ms0run_tFo55ZNrx3T150mL
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=8
Req/s
Output tok/s
698.3
p95 TTFT / TPOT
139.9 ms / 11.4 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=4
Req/s
Output tok/s
361.5
p95 TTFT / TPOT
96.2 ms / 11.06 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=32
Req/s
Output tok/s
2,269.6
p95 TTFT / TPOT
474 ms / 14.02 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=2
Req/s
Output tok/s
185.4
p95 TTFT / TPOT
63 ms / 10.77 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=1
Req/s
Output tok/s
93.4
p95 TTFT / TPOT
27.4 ms / 10.71 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=16
Req/s
Output tok/s
1,297.6
p95 TTFT / TPOT
257.2 ms / 12.26 ms