meta-models/Muse-Glimmer-30B-GGUF

Hardware
4× NVIDIA GeForce RTX 5090
Workload
1024 → 256
Runs
8 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)09298.905run_EV4-PuRWmIsHaHcj: 2.6 req/s, p95 TTFT 4,510.7 msrun_8UpzMA2oIwtEKZYV: 2.38 req/s, p95 TTFT 1,095.7 msrun_KE4Ca-iTDdmTQkyf: 1.03 req/s, p95 TTFT 287.6 msrun_NA5YxygJeYnPY5jx: 0.52 req/s, p95 TTFT 279.2 msrun_fyUaMp-rKzwely1c: 4.75 req/s, p95 TTFT 9,298.9 msrun_9ToGPxDd_y_xlhhy: 2.75 req/s, p95 TTFT 2,319 msrun_z1yxO0Hao6s3A81j: 1.71 req/s, p95 TTFT 352.2 msrun_OVle3RPsxaN1HeV-: 0.26 req/s, p95 TTFT 288.7 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
llama.cpp b10353dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4c=64666.34,510.7 ms143.7 ms39,020 ms0run_EV4-PuRWmIsHaHcj
llama.cpp b10353dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4c=16609.61,095.7 ms48.1 ms13,346.7 ms0run_8UpzMA2oIwtEKZYV
llama.cpp b10353dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4c=4263.7287.6 ms14.14 ms3,884.6 ms0run_KE4Ca-iTDdmTQkyf
llama.cpp b10353dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4c=2132.3279.2 ms14.12 ms3,875.1 ms0run_NA5YxygJeYnPY5jx
llama.cpp b10353dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4c=1281,216.49,298.9 ms129.37 ms36,491.1 ms0run_fyUaMp-rKzwely1c
llama.cpp b10353dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4c=32704.12,319 ms86.49 ms23,468 ms0run_9ToGPxDd_y_xlhhy
llama.cpp b10353dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4c=8438.8352.2 ms16.7 ms4,586.4 ms0run_z1yxO0Hao6s3A81j
llama.cpp b10353dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4c=166.1288.7 ms14.12 ms3,885 ms0run_OVle3RPsxaN1HeV-
Runtime
llama.cpp b10353
Material parameters
dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4
Test load
c=64
Req/s
Output tok/s
666.3
p95 TTFT / TPOT
4,510.7 ms / 143.7 ms
Runtime
llama.cpp b10353
Material parameters
dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4
Test load
c=16
Req/s
Output tok/s
609.6
p95 TTFT / TPOT
1,095.7 ms / 48.1 ms
Runtime
llama.cpp b10353
Material parameters
dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4
Test load
c=4
Req/s
Output tok/s
263.7
p95 TTFT / TPOT
287.6 ms / 14.14 ms
Runtime
llama.cpp b10353
Material parameters
dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4
Test load
c=2
Req/s
Output tok/s
132.3
p95 TTFT / TPOT
279.2 ms / 14.12 ms
Runtime
llama.cpp b10353
Material parameters
dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4
Test load
c=128
Req/s
Output tok/s
1,216.4
p95 TTFT / TPOT
9,298.9 ms / 129.37 ms
Runtime
llama.cpp b10353
Material parameters
dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4
Test load
c=32
Req/s
Output tok/s
704.1
p95 TTFT / TPOT
2,319 ms / 86.49 ms
Runtime
llama.cpp b10353
Material parameters
dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4
Test load
c=8
Req/s
Output tok/s
438.8
p95 TTFT / TPOT
352.2 ms / 16.7 ms
Runtime
llama.cpp b10353
Material parameters
dflash=false kv_cache=Q8_0 slots_per_replica=64 context_tokens_per_slot=2048 context_tokens_per_replica=131072 replicas=4
Test load
c=1
Req/s
Output tok/s
66.1
p95 TTFT / TPOT
288.7 ms / 14.12 ms