malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated

completed

Request throughput: 0.22 req/s
Output throughput: 225.1 tok/s
p95 TTFT: 1007.1 ms
p95 TPOT: 35.41 ms

Manifest warnings

Review these published caveats before interpreting or comparing the metrics.

Review imported facts against the local model, hardware, launch command, and benchmark command before publication.

Server launch command

CUDA_VISIBLE_DEVICES=0 vllm serve malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated \
  --revision f324fd03f047a8fc3fcf392c7dac3106c05905be \
  --port 9000 \
  --host 127.0.0.1 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 9216 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 32 \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 1 \
  --kv-cache-dtype fp8 \
  --distributed-executor-backend mp \
  --no-enable-prefix-caching \
  --enable-chunked-prefill \
  --no-async-scheduling \
  --enforce-eager \
  --served-model-name malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated \
  --disable-uvicorn-access-log \
  --language-model-only \
  --quantization exl3 \
  --quantization-config '{"linear":{"weight":"mxfp8"},"ignore":["re:.*visual\\..*","re:.*in_proj_a$","re:.*in_proj_b$","re:.*in_proj_ba$","re:.*mtp\\..*","lm_head"]}'

Run details

Workload: 256 → 1024
Methodology: closed loop · streaming
Test load: 8 concurrent requests
Endpoint: openai-compatible-completions
Requests: 8 unmeasured warmups per invocation after one readiness request.
Dataset: sha256=9e1873a44a42e2cc60e0aef3613fb134f3786e4c50b64cd016ccf7ee9e65c0cb generator=runpile-sweeps-tokenizer-verified-jsonl synthetic=true generator_revision=d93886d7ee93eb0634b5f426ff480ce0a5798ad447d080e9c42a0d330a8bd055 shared_prefix_tokens=0
Sampling: mode=greedy temperature=0
Stopping: ignore_eos=true max_output_tokens=1024
Prefix and cache: cache_state=warm-host cache_enabled=false shared_prefix_tokens=0
Client: name=vllm bench serve backend=vllm version=0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34 placement=same-host
Hardware: 1× NVIDIA GeForce RTX 5090
Runtime: vllm 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34
Runtime configuration: dtype=bfloat16 max_num_seqs=32 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Measurement: vllm bench serve 0.11.2.dev280+gilded.gnosis.v20.vllm4d006a4.b12xcd3ce19.fi1ac6942.cu132.20260810.r34
Timing: client-observed streaming request latency · client overhead included
Repetitions: 1
Failure handling: Failed requests counted; no client retries.
Quantization: format=EXL3 mixed K5/K6 hydrated recipe=MLP gate/up K5; down/attention/lm_head K6; selected BF16/FP16 passthrough checkpoint=Trellis/EXL3 runtime_specific=true
Model revision: f324fd03f047a8fc3fcf392c7dac3106c05905be
Evidence: Raw benchmark evidence attached · eligible and ranked
Published: 8/29/2026, 8:46:15 AM by anonymous-EVjB2S

Benchmark command

vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:9000 \
  --endpoint /v1/completions \
  --model malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated \
  --tokenizer malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated \
  --dataset-name custom \
  --dataset-path <EXACT_SYNTHETIC_DATASET> \
  --custom-output-len 1024 \
  --skip-chat-template \
  --disable-shuffle \
  --num-prompts 128 \
  --max-concurrency 8 \
  --request-rate inf \
  --seed 0 \
  --ignore-eos \
  --temperature 0 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --disable-tqdm \
  --save-result \
  --result-dir <local-path> \
  --result-filename qwen38-27b-exl3-hydrated-gg-r34-tp1__baseline__serve-fixed-256-1k-v1__c8__r1__attempt1.json \
  --num-warmups 8 \
  --metadata campaign=runpile-rtx5090-qwen38-exl3-pilot-20260828 config_id=qwen38-27b-exl3-hydrated-gg-r34-tp1 server_variant=baseline workload_key=serve-fixed-256-1k-v1 repetition=1 aggregate_only=true