unsloth/Qwen3.8-27B-NVFP4

completed
Request throughput
1.05 req/s
Output throughput
269.5 tok/s
p95 TTFT
95.7 ms
p95 TPOT
14.54 ms

Server launch command

CUDA_VISIBLE_DEVICES=0 vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --revision 7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108 \
  --served-model-name unsloth/Qwen3.8-27B-NVFP4 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 26 \
  --no-enable-prefix-caching \
  --disable-uvicorn-access-log \
  --language-model-only \
  --port 9001 \
  --tensor-parallel-size 1
CUDA_VISIBLE_DEVICES=1 vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --revision 7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108 \
  --served-model-name unsloth/Qwen3.8-27B-NVFP4 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 26 \
  --no-enable-prefix-caching \
  --disable-uvicorn-access-log \
  --language-model-only \
  --port 9002 \
  --tensor-parallel-size 1
CUDA_VISIBLE_DEVICES=2 vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --revision 7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108 \
  --served-model-name unsloth/Qwen3.8-27B-NVFP4 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 26 \
  --no-enable-prefix-caching \
  --disable-uvicorn-access-log \
  --language-model-only \
  --port 9003 \
  --tensor-parallel-size 1
CUDA_VISIBLE_DEVICES=3 vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --revision 7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108 \
  --served-model-name unsloth/Qwen3.8-27B-NVFP4 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 26 \
  --no-enable-prefix-caching \
  --disable-uvicorn-access-log \
  --language-model-only \
  --port 9004 \
  --tensor-parallel-size 1
RUNPILE_BACKENDS=<replica-urls> RUNPILE_PROXY_PORT=9000 python lb_proxy.py

Run details

Workload
1024 → 256
Methodology
closed loop · streaming
Test load
4 concurrent requests
Endpoint
openai_completions
Requests
16 explicit requests before the curve; the harness also issues one unmeasured endpoint test immediately before each timed point.
Dataset
name=vllm-random range_ratio=0 random_prefix_tokens=0
Sampling
mode=greedy temperature=0
Stopping
ignore_eos=true max_tokens=256
Prefix and cache
shared_prefix_tokens=0 server_prefix_caching=false
Client
name=vllm bench serve backend=vllm version=0.27.1
Hardware
4× NVIDIA GeForce RTX 5090
Runtime
vllm 0.27.1
Runtime configuration
dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384 replicas=4
Measurement
vllm bench serve 0.27.1
Timing
Client-observed, streaming online serving interval after the harness initial test request. · client overhead included
Repetitions
1
Failure handling
No client retries; failed requests are counted as failures.
Quantization
format=NVFP4 recipe=Unsloth Dynamic V3.0 preview checkpoint=compressed-tensors
Model revision
7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108
Evidence
Raw benchmark evidence attached · eligible and ranked
Published
by anonymous-TWTejS

Benchmark command

vllm bench serve \
  --backend vllm \
  --host localhost \
  --port 9000 \
  --endpoint /v1/completions \
  --model unsloth/Qwen3.8-27B-NVFP4 \
  --tokenizer unsloth/Qwen3.8-27B-NVFP4 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --random-range-ratio 0 \
  --num-prompts 200 \
  --request-rate inf \
  --max-concurrency 4 \
  --ignore-eos \
  --temperature 0 \
  --seed 20260823 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --disable-tqdm \
  --save-result \
  --result-dir results \
  --result-filename qwen38-27b-nvfp4__r4__i1024__o256__c4.json