meta-models/Muse-Glimmer-30B-GGUF

completed
Request throughput
0.26 req/s
Output throughput
66.1 tok/s
p95 TTFT
288.7 ms
p95 TPOT
14.12 ms

Server launch command

CUDA_VISIBLE_DEVICES=0 llama-server -m Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf -a meta-models/Muse-Glimmer-30B-GGUF -ngl 99 -c 131072 -np 64 \
  --flash-attn on -ctk q8_0 -ctv q8_0 \
  --host 127.0.0.1 \
  --port 9001 \
  --jinja \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 -t 16 -tb 16
CUDA_VISIBLE_DEVICES=1 llama-server -m Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf -a meta-models/Muse-Glimmer-30B-GGUF -ngl 99 -c 131072 -np 64 \
  --flash-attn on -ctk q8_0 -ctv q8_0 \
  --host 127.0.0.1 \
  --port 9002 \
  --jinja \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 -t 16 -tb 16
CUDA_VISIBLE_DEVICES=2 llama-server -m Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf -a meta-models/Muse-Glimmer-30B-GGUF -ngl 99 -c 131072 -np 64 \
  --flash-attn on -ctk q8_0 -ctv q8_0 \
  --host 127.0.0.1 \
  --port 9003 \
  --jinja \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 -t 16 -tb 16
CUDA_VISIBLE_DEVICES=3 llama-server -m Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf -a meta-models/Muse-Glimmer-30B-GGUF -ngl 99 -c 131072 -np 64 \
  --flash-attn on -ctk q8_0 -ctv q8_0 \
  --host 127.0.0.1 \
  --port 9004 \
  --jinja \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 -t 16 -tb 16
RUNPILE_BACKENDS=<replica-urls> RUNPILE_PROXY_PORT=9000 python llama_compat_proxy.py

Run details

Workload
1024 → 256
Methodology
closed loop · streaming
Test load
1 concurrent requests
Endpoint
openai_completions
Requests
16 explicit requests before the curve; the harness also issues one unmeasured endpoint test immediately before each timed point.
Dataset
name=vllm-random range_ratio=0 random_prefix_tokens=0
Sampling
mode=greedy temperature=0
Stopping
ignore_eos=true max_tokens=256
Prefix and cache
shared_prefix_tokens=0 server_prefix_caching=false
Client
name=vllm bench serve backend=openai version=0.27.1
Hardware
4× NVIDIA GeForce RTX 5090
Runtime
llama.cpp b10353
Runtime configuration
dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4
Measurement
vllm bench serve 0.27.1
Timing
Client-observed, streaming online serving interval after the harness initial test request. · client overhead included
Repetitions
1
Failure handling
No client retries; failed requests are counted as failures.
Quantization
format=Dynamic Q4_K_XL checkpoint=GGUF
Model revision
70bf1b61ac09f91b24d39038091b41c582bc5d7a
Evidence
Raw benchmark evidence attached · eligible and ranked
Published
by anonymous-TWTejS

Benchmark command

vllm bench serve \
  --backend openai \
  --host localhost \
  --port 9000 \
  --endpoint /v1/completions \
  --model meta-models/Muse-Glimmer-30B-GGUF \
  --tokenizer meta-models/Muse-Glimmer-30B \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --random-range-ratio 0 \
  --num-prompts 200 \
  --request-rate inf \
  --max-concurrency 1 \
  --ignore-eos \
  --temperature 0 \
  --seed 20260823 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --disable-tqdm \
  --save-result \
  --result-dir results \
  --result-filename muse-glimmer-30b-dynamic-q4-k-xl__r4__i1024__o256__c1.json