gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090
- Hardware
- 1× NVIDIA GeForce RTX 5090
- Draft model
- incoai/Qwen3.8-27B-DFlash2
- Workload
- 256 → 1024
- Runs
- 10 eligible
Efficient frontierOther eligible run
Runs
All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.
| Runtime | Material parameters | Test load | Req/s | Output tok/s | p95 TTFT | p95 TPOT | p95 E2E | Failures | Run |
|---|---|---|---|---|---|---|---|---|---|
sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 | dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4 | c=8 | 1,305.1 | 4,179.5 ms | 5.69 ms | — | 0 | run_ZdaS_isTyCbrlzky | |
sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 | dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4 | c=6 | 1,305 | 2,425.1 ms | 5.69 ms | — | 0 | run_YSoYEmpUtwFOGqIT | |
sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 | dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4 | c=4 | 1,305.7 | 86.6 ms | 5.68 ms | — | 0 | run_hqnqmcX5ovC9wOgL | |
sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 | dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4 | c=4 | 1,306 | 86.8 ms | 5.68 ms | — | 0 | run_zsR6FSLwBSv6O6Xy | |
sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 | dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4 | c=4 | 1,305 | 89.3 ms | 5.69 ms | — | 0 | run_2TUVAgymXwzhJUp2 | |
sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 | dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4 | c=2 | 761.5 | 60.5 ms | 3.85 ms | — | 0 | run_S_8hksOANJ-0LjMh | |
sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 | dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4 | c=1 | 414.9 | 44.4 ms | 3.02 ms | — | 0 | run_3gDbZGc0Xjt8ZzwE | |
sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 | dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=1 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=1 | c=1 | 418.8 | 44.5 ms | 2.99 ms | — | 0 | run_dkYZh2AwUTjESR_A | |
sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 | dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=1 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=1 | c=1 | 418.9 | 44.3 ms | 2.99 ms | — | 0 | run_xCjQi0gT6Va71S5g | |
sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 | dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=1 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=1 | c=1 | 418.9 | 44.4 ms | 2.99 ms | — | 0 | run_fHL70cqgxKpV-z06 |
- Runtime
- sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 draft
- Material parameters
dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4- Test load
- c=8
- Req/s
- Output tok/s
- 1,305.1
- p95 TTFT / TPOT
- 4,179.5 ms / 5.69 ms
- Runtime
- sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 draft
- Material parameters
dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4- Test load
- c=6
- Req/s
- Output tok/s
- 1,305
- p95 TTFT / TPOT
- 2,425.1 ms / 5.69 ms
- Runtime
- sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 draft
- Material parameters
dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4- Test load
- c=4
- Req/s
- Output tok/s
- 1,305.7
- p95 TTFT / TPOT
- 86.6 ms / 5.68 ms
- Runtime
- sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 draft
- Material parameters
dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4- Test load
- c=4
- Req/s
- Output tok/s
- 1,306
- p95 TTFT / TPOT
- 86.8 ms / 5.68 ms
- Runtime
- sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 draft
- Material parameters
dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4- Test load
- c=4
- Req/s
- Output tok/s
- 1,305
- p95 TTFT / TPOT
- 89.3 ms / 5.69 ms
- Runtime
- sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 draft
- Material parameters
dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4- Test load
- c=2
- Req/s
- Output tok/s
- 761.5
- p95 TTFT / TPOT
- 60.5 ms / 3.85 ms
- Runtime
- sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 draft
- Material parameters
dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=4 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=4- Test load
- c=1
- Req/s
- Output tok/s
- 414.9
- p95 TTFT / TPOT
- 44.4 ms / 3.02 ms
- Runtime
- sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 draft
- Material parameters
dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=1 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=1- Test load
- c=1
- Req/s
- Output tok/s
- 418.8
- p95 TTFT / TPOT
- 44.5 ms / 2.99 ms
- Runtime
- sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 draft
- Material parameters
dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=1 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=1- Test load
- c=1
- Req/s
- Output tok/s
- 418.9
- p95 TTFT / TPOT
- 44.3 ms / 2.99 ms
- Runtime
- sglang 0.0.0.dev1+g5f55db35eDFlash2 · BF16 draft
- Material parameters
dtype=bfloat16 context_length=9216 kv_cache_dtype=fp8_e4m3 attention_backend=flashinfer cuda_graph_max_bs=1 mem_fraction_static=0.9 chunked_prefill_size=1024 max_running_requests=1- Test load
- c=1
- Req/s
- Output tok/s
- 418.9
- p95 TTFT / TPOT
- 44.4 ms / 2.99 ms