unsloth/Qwen3.8-27B-NVFP4

completed

Request throughput: 2.37 req/s
Output throughput: 605.7 tok/s
p95 TTFT: 5337.6 ms
p95 TPOT: 19.98 ms

Server launch command

CUDA_VISIBLE_DEVICES=0 vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --revision 7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108 \
  --served-model-name unsloth/Qwen3.8-27B-NVFP4 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 26 \
  --no-enable-prefix-caching \
  --disable-uvicorn-access-log \
  --language-model-only \
  --port 9000 \
  --tensor-parallel-size 1

Run details

Workload: 1024 → 256
Methodology: closed loop · streaming
Test load: 16 concurrent requests
Endpoint: openai_completions
Requests: 16 explicit requests before the curve; the harness also issues one unmeasured endpoint test immediately before each timed point.
Dataset: name=vllm-random range_ratio=0 random_prefix_tokens=0
Sampling: mode=greedy temperature=0
Stopping: ignore_eos=true max_tokens=256
Prefix and cache: shared_prefix_tokens=0 server_prefix_caching=false
Client: name=vllm bench serve backend=vllm version=0.27.1
Hardware: 1× NVIDIA GeForce RTX 5090
Runtime: vllm 0.27.1
Runtime configuration: dtype=bfloat16 max_num_seqs=26 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Measurement: vllm bench serve 0.27.1
Timing: Client-observed, streaming online serving interval after the harness initial test request. · client overhead included
Repetitions: 1
Failure handling: No client retries; failed requests are counted as failures.
Quantization: format=NVFP4 recipe=Unsloth Dynamic V3.0 preview checkpoint=compressed-tensors
Model revision: 7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108
Evidence: Raw benchmark evidence attached · eligible and ranked
Published: 8/24/2026, 12:00:01 AM by anonymous-TWTejS

Benchmark command

vllm bench serve \
  --backend vllm \
  --host localhost \
  --port 9000 \
  --endpoint /v1/completions \
  --model unsloth/Qwen3.8-27B-NVFP4 \
  --tokenizer unsloth/Qwen3.8-27B-NVFP4 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --random-range-ratio 0 \
  --num-prompts 200 \
  --request-rate inf \
  --max-concurrency 16 \
  --ignore-eos \
  --temperature 0 \
  --seed 20260823 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --disable-tqdm \
  --save-result \
  --result-dir results \
  --result-filename qwen38-27b-nvfp4__r1__i1024__o256__c16.json