nvidia/Gemma-4-26B-A4B-NVFP4

completed

Request throughput: 7.27 req/s
Output throughput: 7.3 tok/s
p95 TTFT: 4399.0 ms
p95 TPOT: 0.00 ms

Manifest warnings

Review these published caveats before interpreting or comparing the metrics.

Review imported facts against the local model, hardware, launch command, and benchmark command before publication.

Server launch command

CUDA_VISIBLE_DEVICES=0 vllm serve nvidia/Gemma-4-26B-A4B-NVFP4 \
  --revision a19cfe00be84568a6867111c9a68c9c44fdcffe6 \
  --port 9000 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 9216 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 64 \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 1 \
  --kv-cache-dtype fp8 \
  --distributed-executor-backend mp \
  --no-enable-prefix-caching \
  --enable-chunked-prefill \
  --async-scheduling \
  --no-enforce-eager \
  --served-model-name nvidia/Gemma-4-26B-A4B-NVFP4 \
  --disable-uvicorn-access-log \
  --language-model-only

Run details

Workload: 4096 → 1
Methodology: closed loop · streaming
Test load: 32 concurrent requests
Endpoint: openai-compatible-completions
Requests: 16 unmeasured warmups per invocation after one readiness request.
Dataset: generator=runpile-sweeps-tokenizer-verified-jsonl synthetic=true shared_prefix_tokens=0
Sampling: mode=greedy temperature=0
Stopping: ignore_eos=true max_output_tokens=1
Prefix and cache: cache_state=warm-host cache_enabled=false shared_prefix_tokens=0
Client: name=vllm bench serve backend=vllm version=0.23.0+cu129 placement=same-host
Hardware: 1× NVIDIA GeForce RTX 5090
Runtime: vllm 0.23.0+cu129
Runtime configuration: dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Measurement: vllm bench serve 0.23.0+cu129
Timing: client-observed streaming request latency · client overhead included
Repetitions: 1
Failure handling: Failed requests counted; no client retries.
Quantization: format=NVFP4 checkpoint=modelopt/compressed-tensors
Model revision: a19cfe00be84568a6867111c9a68c9c44fdcffe6
Evidence: Raw benchmark evidence attached · eligible and ranked
Published: 8/27/2026, 8:16:35 AM by anonymous-EVjB2S

Benchmark command

vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:9000 \
  --endpoint /v1/completions \
  --model nvidia/Gemma-4-26B-A4B-NVFP4 \
  --tokenizer nvidia/Gemma-4-26B-A4B-NVFP4 \
  --dataset-name custom \
  --dataset-path <EXACT_SYNTHETIC_DATASET> \
  --custom-output-len 1 \
  --skip-chat-template \
  --disable-shuffle \
  --num-prompts 256 \
  --max-concurrency 32 \
  --request-rate inf \
  --seed 0 \
  --ignore-eos \
  --temperature 0 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --disable-tqdm \
  --save-result \
  --result-dir <local-path> \
  --result-filename gemma4-26b-a4b-nvfp4-tp1__baseline__serve-fixed-4k-1-categorization-v1__c32__r1__attempt1.json \
  --num-warmups 16 \
  --metadata campaign=runpile-rtx5090-param-sweeps-20260826-retry2 config_id=gemma4-26b-a4b-nvfp4-tp1 server_variant=baseline workload_key=serve-fixed-4k-1-categorization-v1 repetition=1 aggregate_only=true