google/gemma-4-12B-it

Hardware
1× NVIDIA GeForce RTX 5090
Workload
256 → 1024
Runs
14 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)04427.401run_CHN_Jn1NhTAv0zEF: 0.39 req/s, p95 TTFT 229.7 msrun_5KsyNrZCLNRkscWL: 0.21 req/s, p95 TTFT 132.1 msrun_e7vu7j9UShdzmTVu: 0.06 req/s, p95 TTFT 38.1 msrun_eyyYptMeEV5Kh3n_: 0.69 req/s, p95 TTFT 448.7 msrun_Tkzn60vetB72X_es: 0.69 req/s, p95 TTFT 464.8 msrun_8iVvnEWqLoOAxWA8: 0.69 req/s, p95 TTFT 448 msrun_KdW7BMgsOMRxkx_b: 0.42 req/s, p95 TTFT 258.5 msrun_rlS_AgZrhCMwfevH: 0.22 req/s, p95 TTFT 159.2 msrun_jZI0edW-oQzK9Ob8: 1.07 req/s, p95 TTFT 4,427.4 msrun_USwUST0Ob2lsLnkH: 0.11 req/s, p95 TTFT 107.8 msrun_XpvqCfbcRzBUrZiE: 0.06 req/s, p95 TTFT 42.4 msrun_GvFyNFQGv5i7d5AE: 0.79 req/s, p95 TTFT 469.7 msrun_T2RySkBLC6ZGTt1u: 0.79 req/s, p95 TTFT 464.4 msrun_cS9sRomZez4h5e9V: 0.79 req/s, p95 TTFT 475.5 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8403.8229.7 ms19.76 ms20,309.6 ms0run_CHN_Jn1NhTAv0zEF
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4215.5132.1 ms18.49 ms19,023.8 ms0run_5KsyNrZCLNRkscWL
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=157.538.1 ms17.38 ms17,812.7 ms0run_e7vu7j9UShdzmTVu
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=16707.4448.7 ms21.11 ms21,691.3 ms0run_eyyYptMeEV5Kh3n_
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=16707.1464.8 ms21.11 ms21,691.5 ms0run_Tkzn60vetB72X_es
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=16705.6448 ms21.18 ms21,979.7 ms0run_8iVvnEWqLoOAxWA8
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8425.1258.5 ms18.68 ms19,304.8 ms0run_KdW7BMgsOMRxkx_b
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4226.5159.2 ms17.61 ms18,117.8 ms0run_rlS_AgZrhCMwfevH
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=321,0924,427.4 ms34.19 ms35,379 ms0run_jZI0edW-oQzK9Ob8
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=2115.4107.8 ms17.29 ms17,770.9 ms0run_USwUST0Ob2lsLnkH
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=157.242.4 ms17.49 ms17,930.7 ms0run_XpvqCfbcRzBUrZiE
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=16805.7469.7 ms19.79 ms20,343.7 ms0run_GvFyNFQGv5i7d5AE
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=16806.6464.4 ms19.77 ms20,346.2 ms0run_T2RySkBLC6ZGTt1u
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=16805.5475.5 ms19.8 ms20,353.5 ms0run_cS9sRomZez4h5e9V
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
403.8
p95 TTFT / TPOT
229.7 ms / 19.76 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
215.5
p95 TTFT / TPOT
132.1 ms / 18.49 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
57.5
p95 TTFT / TPOT
38.1 ms / 17.38 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
707.4
p95 TTFT / TPOT
448.7 ms / 21.11 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
707.1
p95 TTFT / TPOT
464.8 ms / 21.11 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
705.6
p95 TTFT / TPOT
448 ms / 21.18 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
425.1
p95 TTFT / TPOT
258.5 ms / 18.68 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
226.5
p95 TTFT / TPOT
159.2 ms / 17.61 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
1,092
p95 TTFT / TPOT
4,427.4 ms / 34.19 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
115.4
p95 TTFT / TPOT
107.8 ms / 17.29 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
57.2
p95 TTFT / TPOT
42.4 ms / 17.49 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
805.7
p95 TTFT / TPOT
469.7 ms / 19.79 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
806.6
p95 TTFT / TPOT
464.4 ms / 19.77 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
805.5
p95 TTFT / TPOT
475.5 ms / 19.8 ms