nvidia/Qwen3.6-35B-A3B-NVFP4

completed
Request throughput
7.33 req/s
Output throughput
7.3 tok/s
p95 TTFT
138.3 ms
p95 TPOT
0.00 ms

Manifest warnings

Review these published caveats before interpreting or comparing the metrics.

  • Review imported facts against the local model, hardware, launch command, and benchmark command before publication.
  • FP8 KV-cache serving performance measured; no quality evaluation was performed.

Server launch command

CUDA_VISIBLE_DEVICES=0 vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --revision 491c2f1ea524c639598bf8fa787a93fed5a6fbce \
  --served-model-name nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --port 9000 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 9216 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 64 \
  --kv-cache-dtype fp8 \
  --no-enable-prefix-caching \
  --enable-chunked-prefill \
  --no-enforce-eager \
  --async-scheduling \
  --distributed-executor-backend mp \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 1 \
  --disable-uvicorn-access-log \
  --language-model-only \
  --no-enable-flashinfer-autotune

Run details

Workload
4096 → 1
Methodology
closed loop · streaming
Test load
1 concurrent requests
Endpoint
openai-compatible-completions
Requests
16 declared unmeasured warmup requests before 256 measured requests; one separate endpoint-readiness request precedes the declared warmups and is not counted.
Dataset
generator=harness-random-token-ids synthetic=true shared_prefix_tokens=0
Sampling
mode=greedy temperature=0
Stopping
ignore_eos=true max_output_tokens=1
Prefix and cache
cache_state=cold cache_enabled=false shared_prefix_tokens=0
Client
name=vllm bench serve backend=vllm version=0.27.1 placement=same-host
Hardware
1× NVIDIA GeForce RTX 5090
Runtime
vllm 0.27.1
Runtime configuration
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Measurement
vllm bench serve 0.27.1
Timing
client-observed streaming request latency · client overhead included
Repetitions
1
Failure handling
Failed requests counted; no client retries.
Quantization
format=W4A16 NVFP4 mixed precision checkpoint=modelopt
Model revision
491c2f1ea524c639598bf8fa787a93fed5a6fbce
Evidence
Raw benchmark evidence attached · eligible and ranked
Published
by anonymous-52J7ji

Benchmark command

vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:9000 \
  --endpoint /v1/completions \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --tokenizer nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --dataset-name random \
  --random-input-len 4096 \
  --random-output-len 1 \
  --random-range-ratio 0 \
  --random-prefix-len 0 \
  --num-prompts 256 \
  --num-warmups 16 \
  --request-rate inf \
  --max-concurrency 1 \
  --ignore-eos \
  --temperature 0 \
  --seed 0 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --disable-tqdm \
  --save-result \
  --result-dir results \
  --result-filename qwen36-35b-a3b-nvfp4-tp1__tp1__baseline__serve-fixed-4k-1-categorization-v1__c1__rep1.json \
  --metadata campaign=runpile-rtx5090-param-sweeps-20260826 config_id=qwen36-35b-a3b-nvfp4-tp1 server_variant=baseline workload_key=serve-fixed-4k-1-categorization-v1 repetition=1 aggregate_only=true