meta-models/Muse-Glimmer-30B-GGUF

Hardware
1× NVIDIA GeForce RTX 5090
Draft model
meta-models/Muse-Glimmer-30B-GGUF
Workload
256 → 1024
Runs
10 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)0803.501run_Detp7qWzpW8eTAVS: 1.45 req/s, p95 TTFT 803.5 msrun_4yIvCI26xEIl1q3_: 1.16 req/s, p95 TTFT 609.7 msrun_iqr-3lMPfKk1Cnsk: 1.46 req/s, p95 TTFT 621 msrun_ItnsZ7eQkCsSGj9i: 1.2 req/s, p95 TTFT 623.2 msrun_-xw7EW6aooOjOWuP: 0.68 req/s, p95 TTFT 83 msrun_yN1_MOPbeicaIh_R: 1.48 req/s, p95 TTFT 772.2 msrun_N2VYNWN25kzDcrVk: 1.15 req/s, p95 TTFT 613.1 msrun_RPfdGEok-Szn4e-C: 1.46 req/s, p95 TTFT 608 msrun_kF4pG6SlGbL9BMnd: 1.15 req/s, p95 TTFT 621.6 msrun_BNuNM-mIdZ6fq4ht: 0.68 req/s, p95 TTFT 82.9 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
llama.cpp b10689Speculative · Q4_K_Mspeculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=81,481.2803.5 ms5.57 ms6,127.4 ms0run_Detp7qWzpW8eTAVS
llama.cpp b10689Speculative · Q4_K_Mspeculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=81,184.4609.7 ms6.75 ms7,182.4 ms0run_4yIvCI26xEIl1q3_
llama.cpp b10689Speculative · Q4_K_Mspeculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=81,490.4621 ms5.54 ms6,001.3 ms0run_iqr-3lMPfKk1Cnsk
llama.cpp b10689Speculative · Q4_K_Mspeculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=81,224.1623.2 ms6.56 ms6,977.5 ms0run_ItnsZ7eQkCsSGj9i
llama.cpp b10689Speculative · Q4_K_Mspeculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=1698.583 ms1.37 ms1,477.9 ms0run_-xw7EW6aooOjOWuP
llama.cpp b10689Speculative · Q4_K_Mspeculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=81,512.8772.2 ms5.44 ms5,940.3 ms0run_yN1_MOPbeicaIh_R
llama.cpp b10689Speculative · Q4_K_Mspeculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=81,181.5613.1 ms6.82 ms7,175.7 ms0run_N2VYNWN25kzDcrVk
llama.cpp b10689Speculative · Q4_K_Mspeculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=81,500608 ms5.49 ms5,942.4 ms0run_RPfdGEok-Szn4e-C
llama.cpp b10689Speculative · Q4_K_Mspeculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=81,181.4621.6 ms7 ms7,435.9 ms0run_kF4pG6SlGbL9BMnd
llama.cpp b10689Speculative · Q4_K_Mspeculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=1695.182.9 ms1.38 ms1,488.2 ms0run_BNuNM-mIdZ6fq4ht
Runtime
llama.cpp b10689Speculative · Q4_K_M draft
Material parameters
speculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,481.2
p95 TTFT / TPOT
803.5 ms / 5.57 ms
Runtime
llama.cpp b10689Speculative · Q4_K_M draft
Material parameters
speculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,184.4
p95 TTFT / TPOT
609.7 ms / 6.75 ms
Runtime
llama.cpp b10689Speculative · Q4_K_M draft
Material parameters
speculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,490.4
p95 TTFT / TPOT
621 ms / 5.54 ms
Runtime
llama.cpp b10689Speculative · Q4_K_M draft
Material parameters
speculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,224.1
p95 TTFT / TPOT
623.2 ms / 6.56 ms
Runtime
llama.cpp b10689Speculative · Q4_K_M draft
Material parameters
speculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
698.5
p95 TTFT / TPOT
83 ms / 1.37 ms
Runtime
llama.cpp b10689Speculative · Q4_K_M draft
Material parameters
speculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,512.8
p95 TTFT / TPOT
772.2 ms / 5.44 ms
Runtime
llama.cpp b10689Speculative · Q4_K_M draft
Material parameters
speculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,181.5
p95 TTFT / TPOT
613.1 ms / 6.82 ms
Runtime
llama.cpp b10689Speculative · Q4_K_M draft
Material parameters
speculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,500
p95 TTFT / TPOT
608 ms / 5.49 ms
Runtime
llama.cpp b10689Speculative · Q4_K_M draft
Material parameters
speculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,181.4
p95 TTFT / TPOT
621.6 ms / 7 ms
Runtime
llama.cpp b10689Speculative · Q4_K_M draft
Material parameters
speculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
695.1
p95 TTFT / TPOT
82.9 ms / 1.38 ms