meta-models/Muse-Glimmer-30B-GGUF

Hardware
1× NVIDIA GeForce RTX 5090
Workload
256 → 1024
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)072.800run_hr4H16loHXgxGwLI: 0.07 req/s, p95 TTFT 72.6 msrun_aXKgACdJMkTRt5E_: 0.07 req/s, p95 TTFT 72 msrun_amWkOyipTZnBZoRo: 0.07 req/s, p95 TTFT 72.8 msrun_rcqbIXuIcCjDkdsm: 0.07 req/s, p95 TTFT 72.7 msrun_KbpGDCzaT_pbiULj: 0.07 req/s, p95 TTFT 72.8 msrun_MzwfVBHrKH4WI4YH: 0.07 req/s, p95 TTFT 72.7 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=17272.6 ms13.87 ms14,256.5 ms0run_hr4H16loHXgxGwLI
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=17272 ms13.85 ms14,233.9 ms0run_aXKgACdJMkTRt5E_
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=171.972.8 ms13.9 ms14,286.3 ms0run_amWkOyipTZnBZoRo
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=171.972.7 ms13.88 ms14,264.4 ms0run_rcqbIXuIcCjDkdsm
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=171.972.8 ms13.86 ms14,251.6 ms0run_KbpGDCzaT_pbiULj
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=171.872.7 ms13.91 ms14,296.9 ms0run_MzwfVBHrKH4WI4YH
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
72
p95 TTFT / TPOT
72.6 ms / 13.87 ms
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
72
p95 TTFT / TPOT
72 ms / 13.85 ms
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
71.9
p95 TTFT / TPOT
72.8 ms / 13.9 ms
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
71.9
p95 TTFT / TPOT
72.7 ms / 13.88 ms
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
71.9
p95 TTFT / TPOT
72.8 ms / 13.86 ms
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
71.8
p95 TTFT / TPOT
72.7 ms / 13.91 ms