malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated

completed
Request throughput
0.22 req/s
Output throughput
222.4 tok/s
p95 TTFT
1003.2 ms
p95 TPOT
35.84 ms

Manifest warnings

Review these published caveats before interpreting or comparing the metrics.

  • Review imported facts against the local model, hardware, launch command, and benchmark command before publication.

Server launch command

CUDA_VISIBLE_DEVICES=0 vllm serve malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated \
  --revision f324fd03f047a8fc3fcf392c7dac3106c05905be \
  --port 9000 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 9216 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 32 \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 1 \
  --kv-cache-dtype fp8 \
  --distributed-executor-backend mp \
  --no-enable-prefix-caching \
  --enable-chunked-prefill \
  --no-async-scheduling \
  --enforce-eager \
  --served-model-name malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated \
  --disable-uvicorn-access-log \
  --language-model-only \
  --quantization exl3 \
  --quantization-config '{"linear":{"weight":"mxfp8"},"ignore":["re:.*visual\\..*","re:.*in_proj_a$","re:.*in_proj_b$","re:.*in_proj_ba$","re:.*mtp\\..*","lm_head"]}'

Run details

Workload
256 → 1024
Methodology
closed loop · streaming
Test load
8 concurrent requests
Endpoint
openai-compatible-completions
Requests
8 unmeasured warmups per invocation after one readiness request.
Dataset
sha256=9e1873a44a42e2cc60e0aef3613fb134f3786e4c50b64cd016ccf7ee9e65c0cb generator=runpile-sweeps-tokenizer-verified-jsonl synthetic=true generator_revision=d93886d7ee93eb0634b5f426ff480ce0a5798ad447d080e9c42a0d330a8bd055 shared_prefix_tokens=0
Sampling
mode=greedy temperature=0
Stopping
ignore_eos=true max_output_tokens=1024
Prefix and cache
cache_state=warm-host cache_enabled=false shared_prefix_tokens=0
Client
name=vllm bench serve backend=vllm version=0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34 placement=same-host
Hardware
1× NVIDIA GeForce RTX 5090
Runtime
vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34
Runtime configuration
dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Measurement
vllm bench serve 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34
Timing
client-observed streaming request latency · client overhead included
Repetitions
1
Failure handling
Failed requests counted; no client retries.
Quantization
format=EXL3 mixed K5/K6 hydrated recipe=MLP gate/up K5; down/attention/lm_head K6; selected BF16/FP16 passthrough checkpoint=Trellis/EXL3 runtime_specific=true
Model revision
f324fd03f047a8fc3fcf392c7dac3106c05905be
Evidence
Raw benchmark evidence attached · eligible and ranked
Published
by anonymous-EVjB2S

Benchmark command

vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:9000 \
  --endpoint /v1/completions \
  --model malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated \
  --tokenizer malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated \
  --dataset-name custom \
  --dataset-path <EXACT_SYNTHETIC_DATASET> \
  --custom-output-len 1024 \
  --skip-chat-template \
  --disable-shuffle \
  --num-prompts 128 \
  --max-concurrency 8 \
  --request-rate inf \
  --seed 0 \
  --ignore-eos \
  --temperature 0 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --disable-tqdm \
  --save-result \
  --result-dir <local-path> \
  --result-filename qwen38-27b-exl3-hydrated-gg-r34-tp1__baseline__serve-fixed-256-1k-v1__c8__r2__attempt1.json \
  --num-warmups 8 \
  --metadata campaign=runpile-rtx5090-qwen38-exl3-pilot-20260828 config_id=qwen38-27b-exl3-hydrated-gg-r34-tp1 server_variant=baseline workload_key=serve-fixed-256-1k-v1 repetition=2 aggregate_only=true