EXP
001
001
publisheddemo-runpileschema 1.08/16/2026
Gemma 4 NVFP4 throughput optimization on RTX 5090
Checkpoint, concurrency, GPU rotation, KV-cache, and scheduler sweeps for a prefill-heavy binary-judge workload.
binary-judgegemma-4llm-inferenceprefill-heavyrtx-5090
01
Question
Maximize total throughput for a large fixed-input binary-judge batch while retaining enough evidence to reproduce and interpret every configuration.
Method. One model replica per RTX 5090. Synthetic fixed 6,000-token prompts and greedy one-token outputs. Eight warmup requests precede timed closed-loop runs. Scheduler candidates use short screenings followed by 500-request validation near the optimum.
02
Author conclusion
The Unsloth checkpoint with a 10,240 scheduler token budget produced the best measured single-replica result around 8.65 requests/s. Larger budgets did not improve throughput and 12,288 failed at startup. Rotation results do not indicate a material device outlier.
Author-supplied interpretation · not a Runpile endorsement
03
Scheduler sweep
Token budget vs. throughput
Raw measured points · no smoothing
requests/s
| Max batched tokens | Requests per second |
|---|---|
| 2048 | 7.43 |
| 4096 | 8.38 |
| 6144 | 8.5 |
| 8192 | 8.53 |
| 10240 | 8.65 |
| 10752 | 8.58 |
| 11264 | 8.52 |
| 11776 | 8.56 |
04
Runs
Complete run record
Completed, failed, interrupted, repeated, and derived runs remain visible.
| # | Run | Configuration | Result | Status | Evidence |
|---|---|---|---|---|---|
| 01 | NVIDIA checkpoint, concurrency 8 | c=8 · 10,240 | 8.07 req/s | completed | 35 |
| 02 | Unsloth checkpoint, concurrency 8 | c=8 · 10,240 | 8.64 req/s | completed | 35 |
| 03 | Red Hat AI checkpoint, concurrency 8 | c=8 · 10,240 | 8.21 req/s | completed | 35 |
| 04 | Concurrency sweep, c=1 | c=1 · 10,240 | 1.18 req/s | completed | 35 |
| 05 | Concurrency sweep, c=2 | c=2 · 10,240 | 2.31 req/s | completed | 35 |
| 06 | Concurrency sweep, c=4 | c=4 · 10,240 | 4.55 req/s | completed | 35 |
| 07 | Concurrency sweep, c=8 | c=8 · 10,240 | 8.64 req/s | completed | 35 |
| 08 | Scheduler token budget 2048 | c=8 · 2,048 | 7.43 req/s | completed | 35 |
| 09 | Scheduler token budget 4096 | c=8 · 4,096 | 8.38 req/s | completed | 35 |
| 10 | Scheduler token budget 6144 | c=8 · 6,144 | 8.50 req/s | completed | 35 |
| 11 | Scheduler token budget 8192 | c=8 · 8,192 | 8.53 req/s | completed | 35 |
| 12 | Scheduler token budget 10240 | c=8 · 10,240 | 8.65 req/s | completed | 35 |
| 13 | Scheduler token budget 10752 | c=8 · 10,752 | 8.58 req/s | completed | 35 |
| 14 | Scheduler token budget 11264 | c=8 · 11,264 | 8.52 req/s | completed | 35 |
| 15 | Scheduler token budget 11776 | c=8 · 11,776 | 8.56 req/s | completed | 35 |
| 16 | GPU rotation validation, device 0 | c=8 · 10,240 | 8.61 req/s | completed | 35 |
| 17 | GPU rotation validation, device 1 | c=8 · 10,240 | 8.58 req/s | completed | 35 |
| 18 | GPU rotation validation, device 2 | c=8 · 10,240 | 8.63 req/s | completed | 35 |
| 19 | GPU rotation validation, device 3 | c=8 · 10,240 | 8.60 req/s | completed | 35 |
| 20 | Full 500-request validation A | c=8 · 10,240 | 8.54 req/s | completed | 35 |
| 21 | Full 500-request validation B | c=8 · 10,240 | 8.59 req/s | completed | 35 |
| 22 | Full 500-request validation C | c=8 · 10,240 | 8.64 req/s | completed | 35 |
| 23 | Scheduler token budget 12288 (startup failure) | c=8 · 12,288 | — | failed | 35 |
| 24 | Interrupted preliminary screening | c=8 · 8,192 | — | interrupted | 35 |
| 25 | Derived four-replica throughput | c=8 · 10,240 | 34.48 req/s | completed | 35 |
05
Notes
Limitations
- Synthetic tokens do not represent semantic prompt distributions.
- The four-replica aggregate is derived from independent rates, not timed end-to-end.
- Power was not measured.
Complete experiment manifest
{
"kind": "experiment",
"runs": [
{
"name": "NVIDIA checkpoint, concurrency 8",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.07,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "96d622112c81fab027dc97c56d16c7715a97e4ec3f1c94f0f3148878da03390a",
"license": "CC-BY-4.0",
"filename": "checkpoint-nvidia-c8.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "checkpoint-nvidia-c8",
"client_artifact_id": "checkpoint-nvidia-c8-raw",
"raw_requests_included": false
}
],
"client_trial_id": "checkpoint-nvidia-c8-trial-1",
"duration_seconds": 61.95786864931846
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"checkpoint-nvidia-c8"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 61.95786864931846,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.07,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 48420,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.07,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "03f11a2ecf5d5c682c2720f1fb9bd46001bcf651",
"identifier": "nvidia/Gemma-4-26B-A4B-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "nvidia",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:01:01.957Z",
"relation": {
"type": "sweep_point",
"group": "checkpoint"
},
"warnings": [
"Trust-remote-code was required by the submitted checkpoint."
],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "03f11a2ecf5d5c682c2720f1fb9bd46001bcf651",
"identifier": "nvidia/Gemma-4-26B-A4B-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "96d622112c81fab027dc97c56d16c7715a97e4ec3f1c94f0f3148878da03390a",
"license": "CC-BY-4.0",
"filename": "checkpoint-nvidia-c8.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "checkpoint-nvidia-c8",
"client_artifact_id": "checkpoint-nvidia-c8-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "checkpoint-nvidia-c8",
"duration_seconds": 61.95786864931846
},
{
"name": "Unsloth checkpoint, concurrency 8",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.6446,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "959ec01dd0c84f493e5877882900ea0aba84a942ec1e655539c19a6fc51b08d8",
"license": "CC-BY-4.0",
"filename": "checkpoint-unsloth-c8.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "checkpoint-unsloth-c8",
"client_artifact_id": "checkpoint-unsloth-c8-raw",
"raw_requests_included": false
}
],
"client_trial_id": "checkpoint-unsloth-c8-trial-1",
"duration_seconds": 57.83957615158596
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"checkpoint-unsloth-c8"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 57.83957615158596,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.6446,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51867.600000000006,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.6446,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:57.839Z",
"relation": {
"type": "sweep_point",
"group": "checkpoint"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "959ec01dd0c84f493e5877882900ea0aba84a942ec1e655539c19a6fc51b08d8",
"license": "CC-BY-4.0",
"filename": "checkpoint-unsloth-c8.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "checkpoint-unsloth-c8",
"client_artifact_id": "checkpoint-unsloth-c8-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "checkpoint-unsloth-c8",
"duration_seconds": 57.83957615158596
},
{
"name": "Red Hat AI checkpoint, concurrency 8",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.21,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "192782b5bbc317bf7245b1de8311bf7a565069932e86b6a147e746d29bc9e1cf",
"license": "CC-BY-4.0",
"filename": "checkpoint-redhat-c8.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "checkpoint-redhat-c8",
"client_artifact_id": "checkpoint-redhat-c8-raw",
"raw_requests_included": false
}
],
"client_trial_id": "checkpoint-redhat-c8-trial-1",
"duration_seconds": 60.90133982947624
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"checkpoint-redhat-c8"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 60.90133982947624,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.21,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 49260.00000000001,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.21,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "fdd8f6ca3d63c0a1db4ae62d105fabcc788726b9",
"identifier": "RedHatAI/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "redhat",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:01:00.901Z",
"relation": {
"type": "sweep_point",
"group": "checkpoint"
},
"warnings": [
"Checkpoint emitted a compressed-tensors compatibility warning at startup."
],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "fdd8f6ca3d63c0a1db4ae62d105fabcc788726b9",
"identifier": "RedHatAI/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "192782b5bbc317bf7245b1de8311bf7a565069932e86b6a147e746d29bc9e1cf",
"license": "CC-BY-4.0",
"filename": "checkpoint-redhat-c8.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "checkpoint-redhat-c8",
"client_artifact_id": "checkpoint-redhat-c8-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "checkpoint-redhat-c8",
"duration_seconds": 60.90133982947624
},
{
"name": "Concurrency sweep, c=1",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 1.18,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "48e9dc2d6d7adc253e7171b1c4e459ecbdd35ec6f02acaf853208b28b2ac135e",
"license": "CC-BY-4.0",
"filename": "concurrency-1.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "concurrency-1",
"client_artifact_id": "concurrency-1-raw",
"raw_requests_included": false
}
],
"client_trial_id": "concurrency-1-trial-1",
"duration_seconds": 423.7288135593221
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"concurrency-1"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 423.7288135593221,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 1.18,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 7080,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 1.18,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:07:03.728Z",
"relation": {
"type": "sweep_point",
"group": "concurrency"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 1,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "48e9dc2d6d7adc253e7171b1c4e459ecbdd35ec6f02acaf853208b28b2ac135e",
"license": "CC-BY-4.0",
"filename": "concurrency-1.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "concurrency-1",
"client_artifact_id": "concurrency-1-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "concurrency-1",
"duration_seconds": 423.7288135593221
},
{
"name": "Concurrency sweep, c=2",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 2.31,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "2a08f16881b942d7242c7415e5d851ce8171434b0c9131f261430a61c3a1f3b8",
"license": "CC-BY-4.0",
"filename": "concurrency-2.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "concurrency-2",
"client_artifact_id": "concurrency-2-raw",
"raw_requests_included": false
}
],
"client_trial_id": "concurrency-2-trial-1",
"duration_seconds": 216.45021645021646
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"concurrency-2"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 216.45021645021646,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 2.31,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 13860,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 2.31,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:03:36.450Z",
"relation": {
"type": "sweep_point",
"group": "concurrency"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 2,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "2a08f16881b942d7242c7415e5d851ce8171434b0c9131f261430a61c3a1f3b8",
"license": "CC-BY-4.0",
"filename": "concurrency-2.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "concurrency-2",
"client_artifact_id": "concurrency-2-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "concurrency-2",
"duration_seconds": 216.45021645021646
},
{
"name": "Concurrency sweep, c=4",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 4.55,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "9a2e687a197735ab05807e7747e83ba83f579630839b358e3719990dbc3e4c52",
"license": "CC-BY-4.0",
"filename": "concurrency-4.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "concurrency-4",
"client_artifact_id": "concurrency-4-raw",
"raw_requests_included": false
}
],
"client_trial_id": "concurrency-4-trial-1",
"duration_seconds": 109.8901098901099
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"concurrency-4"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 109.8901098901099,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 4.55,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 27300,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 4.55,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:01:49.890Z",
"relation": {
"type": "sweep_point",
"group": "concurrency"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 4,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "9a2e687a197735ab05807e7747e83ba83f579630839b358e3719990dbc3e4c52",
"license": "CC-BY-4.0",
"filename": "concurrency-4.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "concurrency-4",
"client_artifact_id": "concurrency-4-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "concurrency-4",
"duration_seconds": 109.8901098901099
},
{
"name": "Concurrency sweep, c=8",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.6446,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "ce9a510753fdc8b695fe36d15b2698dfc1043c3fc2b90de750c5dd2af166bb84",
"license": "CC-BY-4.0",
"filename": "concurrency-8.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "concurrency-8",
"client_artifact_id": "concurrency-8-raw",
"raw_requests_included": false
}
],
"client_trial_id": "concurrency-8-trial-1",
"duration_seconds": 57.83957615158596
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"concurrency-8"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 57.83957615158596,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.6446,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51867.600000000006,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.6446,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:57.839Z",
"relation": {
"type": "sweep_point",
"group": "concurrency"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "ce9a510753fdc8b695fe36d15b2698dfc1043c3fc2b90de750c5dd2af166bb84",
"license": "CC-BY-4.0",
"filename": "concurrency-8.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "concurrency-8",
"client_artifact_id": "concurrency-8-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "concurrency-8",
"duration_seconds": 57.83957615158596
},
{
"name": "Scheduler token budget 2048",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 7.43,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "eda6e7673a06238b74d12e16d5a01c4f9b8acafb5f0317c508a21c2836040777",
"license": "CC-BY-4.0",
"filename": "scheduler-2048.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-2048",
"client_artifact_id": "scheduler-2048-raw",
"raw_requests_included": false
}
],
"client_trial_id": "scheduler-2048-trial-1",
"duration_seconds": 67.29475100942126
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"scheduler-2048"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 67.29475100942126,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 7.43,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 44580,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 7.43,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 2048
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:01:07.294Z",
"relation": {
"type": "sweep_point",
"group": "max_num_batched_tokens"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "eda6e7673a06238b74d12e16d5a01c4f9b8acafb5f0317c508a21c2836040777",
"license": "CC-BY-4.0",
"filename": "scheduler-2048.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-2048",
"client_artifact_id": "scheduler-2048-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "scheduler-2048",
"duration_seconds": 67.29475100942126
},
{
"name": "Scheduler token budget 4096",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.38,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "e1365c3b671792993b0211e49c30a8163596b698263cec24c1a5980d4e823c94",
"license": "CC-BY-4.0",
"filename": "scheduler-4096.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-4096",
"client_artifact_id": "scheduler-4096-raw",
"raw_requests_included": false
}
],
"client_trial_id": "scheduler-4096-trial-1",
"duration_seconds": 59.665871121718375
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"scheduler-4096"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 59.665871121718375,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.38,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 50280.00000000001,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.38,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 4096
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 4096 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:59.665Z",
"relation": {
"type": "sweep_point",
"group": "max_num_batched_tokens"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "e1365c3b671792993b0211e49c30a8163596b698263cec24c1a5980d4e823c94",
"license": "CC-BY-4.0",
"filename": "scheduler-4096.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-4096",
"client_artifact_id": "scheduler-4096-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "scheduler-4096",
"duration_seconds": 59.665871121718375
},
{
"name": "Scheduler token budget 6144",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.5,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "dd045ff5fbfff084caafa2eff08531aeb88dc4c3ffd170bc869da14506765b0b",
"license": "CC-BY-4.0",
"filename": "scheduler-6144.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-6144",
"client_artifact_id": "scheduler-6144-raw",
"raw_requests_included": false
}
],
"client_trial_id": "scheduler-6144-trial-1",
"duration_seconds": 58.8235294117647
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"scheduler-6144"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 58.8235294117647,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.5,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51000,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.5,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 6144
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 6144 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:58.823Z",
"relation": {
"type": "sweep_point",
"group": "max_num_batched_tokens"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "dd045ff5fbfff084caafa2eff08531aeb88dc4c3ffd170bc869da14506765b0b",
"license": "CC-BY-4.0",
"filename": "scheduler-6144.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-6144",
"client_artifact_id": "scheduler-6144-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "scheduler-6144",
"duration_seconds": 58.8235294117647
},
{
"name": "Scheduler token budget 8192",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.53,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "5f2578bef81269c3cd1f75cf57662f234da71a440c0fc60384f9befb8a03b185",
"license": "CC-BY-4.0",
"filename": "scheduler-8192.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-8192",
"client_artifact_id": "scheduler-8192-raw",
"raw_requests_included": false
}
],
"client_trial_id": "scheduler-8192-trial-1",
"duration_seconds": 58.6166471277843
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"scheduler-8192"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 58.6166471277843,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.53,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51179.99999999999,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.53,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 8192
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:58.616Z",
"relation": {
"type": "sweep_point",
"group": "max_num_batched_tokens"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "5f2578bef81269c3cd1f75cf57662f234da71a440c0fc60384f9befb8a03b185",
"license": "CC-BY-4.0",
"filename": "scheduler-8192.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-8192",
"client_artifact_id": "scheduler-8192-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "scheduler-8192",
"duration_seconds": 58.6166471277843
},
{
"name": "Scheduler token budget 10240",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.65,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "3202c76774de95899b36ddbb2f920b9f826cc022e25647844428c7865e799c03",
"license": "CC-BY-4.0",
"filename": "scheduler-10240.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-10240",
"client_artifact_id": "scheduler-10240-raw",
"raw_requests_included": false
}
],
"client_trial_id": "scheduler-10240-trial-1",
"duration_seconds": 57.80346820809248
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"scheduler-10240"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 57.80346820809248,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.65,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51900,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.65,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:57.803Z",
"relation": {
"type": "sweep_point",
"group": "max_num_batched_tokens"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "3202c76774de95899b36ddbb2f920b9f826cc022e25647844428c7865e799c03",
"license": "CC-BY-4.0",
"filename": "scheduler-10240.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-10240",
"client_artifact_id": "scheduler-10240-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "scheduler-10240",
"duration_seconds": 57.80346820809248
},
{
"name": "Scheduler token budget 10752",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.58,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "782b5e9940b819009c30237612eb05b522e98291a1825e59f44f9eb71322372f",
"license": "CC-BY-4.0",
"filename": "scheduler-10752.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-10752",
"client_artifact_id": "scheduler-10752-raw",
"raw_requests_included": false
}
],
"client_trial_id": "scheduler-10752-trial-1",
"duration_seconds": 58.27505827505828
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"scheduler-10752"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 58.27505827505828,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.58,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51480,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.58,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10752
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10752 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:58.275Z",
"relation": {
"type": "sweep_point",
"group": "max_num_batched_tokens"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "782b5e9940b819009c30237612eb05b522e98291a1825e59f44f9eb71322372f",
"license": "CC-BY-4.0",
"filename": "scheduler-10752.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-10752",
"client_artifact_id": "scheduler-10752-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "scheduler-10752",
"duration_seconds": 58.27505827505828
},
{
"name": "Scheduler token budget 11264",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.52,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "9ca84451584837616509201c58a0b1b9ee9ed44fec11a84c4ec6629f3cb568c4",
"license": "CC-BY-4.0",
"filename": "scheduler-11264.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-11264",
"client_artifact_id": "scheduler-11264-raw",
"raw_requests_included": false
}
],
"client_trial_id": "scheduler-11264-trial-1",
"duration_seconds": 58.68544600938967
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"scheduler-11264"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 58.68544600938967,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.52,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51120,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.52,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 11264
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 11264 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:58.685Z",
"relation": {
"type": "sweep_point",
"group": "max_num_batched_tokens"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "9ca84451584837616509201c58a0b1b9ee9ed44fec11a84c4ec6629f3cb568c4",
"license": "CC-BY-4.0",
"filename": "scheduler-11264.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-11264",
"client_artifact_id": "scheduler-11264-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "scheduler-11264",
"duration_seconds": 58.68544600938967
},
{
"name": "Scheduler token budget 11776",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.56,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "1263bf649b4c58f5dc79bb52c5f6d776e9a4196ed2fc9fe07d005569d5da3e30",
"license": "CC-BY-4.0",
"filename": "scheduler-11776.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-11776",
"client_artifact_id": "scheduler-11776-raw",
"raw_requests_included": false
}
],
"client_trial_id": "scheduler-11776-trial-1",
"duration_seconds": 58.41121495327103
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"scheduler-11776"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 58.41121495327103,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.56,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51360,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.56,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 11776
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 11776 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:58.411Z",
"relation": {
"type": "sweep_point",
"group": "max_num_batched_tokens"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "1263bf649b4c58f5dc79bb52c5f6d776e9a4196ed2fc9fe07d005569d5da3e30",
"license": "CC-BY-4.0",
"filename": "scheduler-11776.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-11776",
"client_artifact_id": "scheduler-11776-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "scheduler-11776",
"duration_seconds": 58.41121495327103
},
{
"name": "GPU rotation validation, device 0",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.61,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "973ae9f36fea7eb0e527a1430dc72b0c063b80a4c0850ae30d990822847689d4",
"license": "CC-BY-4.0",
"filename": "benchmark-20260815T160000.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "gpu-rotation-0",
"client_artifact_id": "gpu-rotation-0-raw",
"raw_requests_included": false
}
],
"client_trial_id": "gpu-rotation-0-trial-1",
"duration_seconds": 23.228803716608596
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 1200200,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"gpu-rotation-0"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 23.228803716608596,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.61,
"measurement_status": "measured"
},
"successful_requests": 200,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51660,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.61,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:23.228Z",
"relation": {
"type": "sweep_point",
"group": "gpu_device"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 200,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "973ae9f36fea7eb0e527a1430dc72b0c063b80a4c0850ae30d990822847689d4",
"license": "CC-BY-4.0",
"filename": "benchmark-20260815T160000.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "gpu-rotation-0",
"client_artifact_id": "gpu-rotation-0-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "gpu-rotation-0",
"duration_seconds": 23.228803716608596
},
{
"name": "GPU rotation validation, device 1",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.58,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "db9be6d742923bbfdfe15edc396516cd1bf68b53e1791f936aef7baa4e56b5cf",
"license": "CC-BY-4.0",
"filename": "benchmark-20260815T160000.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "gpu-rotation-1",
"client_artifact_id": "gpu-rotation-1-raw",
"raw_requests_included": false
}
],
"client_trial_id": "gpu-rotation-1-trial-1",
"duration_seconds": 23.31002331002331
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 1200200,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"gpu-rotation-1"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 23.31002331002331,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.58,
"measurement_status": "measured"
},
"successful_requests": 200,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51480,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.58,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:23.310Z",
"relation": {
"type": "sweep_point",
"group": "gpu_device"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 200,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "db9be6d742923bbfdfe15edc396516cd1bf68b53e1791f936aef7baa4e56b5cf",
"license": "CC-BY-4.0",
"filename": "benchmark-20260815T160000.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "gpu-rotation-1",
"client_artifact_id": "gpu-rotation-1-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 1
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
1
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "gpu-rotation-1",
"duration_seconds": 23.31002331002331
},
{
"name": "GPU rotation validation, device 2",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.63,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "9d1dd0b12cc4185915ab281d047e885d691f8a3cd21d19d5a423bffdb05efc92",
"license": "CC-BY-4.0",
"filename": "benchmark-20260815T160000.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "gpu-rotation-2",
"client_artifact_id": "gpu-rotation-2-raw",
"raw_requests_included": false
}
],
"client_trial_id": "gpu-rotation-2-trial-1",
"duration_seconds": 23.17497103128621
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 1200200,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"gpu-rotation-2"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 23.17497103128621,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.63,
"measurement_status": "measured"
},
"successful_requests": 200,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51780.00000000001,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.63,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:23.174Z",
"relation": {
"type": "sweep_point",
"group": "gpu_device"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 200,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "9d1dd0b12cc4185915ab281d047e885d691f8a3cd21d19d5a423bffdb05efc92",
"license": "CC-BY-4.0",
"filename": "benchmark-20260815T160000.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "gpu-rotation-2",
"client_artifact_id": "gpu-rotation-2-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 2
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
2
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "gpu-rotation-2",
"duration_seconds": 23.17497103128621
},
{
"name": "GPU rotation validation, device 3",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.6,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "28d1069bd4d13b2ec72a75a0bad436de638914511cee8b3cff6d98918e54e08f",
"license": "CC-BY-4.0",
"filename": "benchmark-20260815T160000.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "gpu-rotation-3",
"client_artifact_id": "gpu-rotation-3-raw",
"raw_requests_included": false
}
],
"client_trial_id": "gpu-rotation-3-trial-1",
"duration_seconds": 23.255813953488374
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 1200200,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"gpu-rotation-3"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 23.255813953488374,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.6,
"measurement_status": "measured"
},
"successful_requests": 200,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51600,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.6,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:23.255Z",
"relation": {
"type": "sweep_point",
"group": "gpu_device"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 200,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "28d1069bd4d13b2ec72a75a0bad436de638914511cee8b3cff6d98918e54e08f",
"license": "CC-BY-4.0",
"filename": "benchmark-20260815T160000.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "gpu-rotation-3",
"client_artifact_id": "gpu-rotation-3-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 3
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
3
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "gpu-rotation-3",
"duration_seconds": 23.255813953488374
},
{
"name": "Full 500-request validation A",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.54,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "1d9a21c83ecddfe9b773fca0b49b6e171d87a4a9eb77d987bb6620a972d98f16",
"license": "CC-BY-4.0",
"filename": "validation-500-a.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "validation-500-a",
"client_artifact_id": "validation-500-a-raw",
"raw_requests_included": false
}
],
"client_trial_id": "validation-500-a-trial-1",
"duration_seconds": 58.5480093676815
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"validation-500-a"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 58.5480093676815,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.54,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51239.99999999999,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.54,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:58.548Z",
"relation": {
"type": "sweep_point",
"group": "repetition"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "1d9a21c83ecddfe9b773fca0b49b6e171d87a4a9eb77d987bb6620a972d98f16",
"license": "CC-BY-4.0",
"filename": "validation-500-a.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "validation-500-a",
"client_artifact_id": "validation-500-a-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "validation-500-a",
"duration_seconds": 58.5480093676815
},
{
"name": "Full 500-request validation B",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.59,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "703b6b48dba8c0504795932829b2e275690e0c3c0ad1fb8d63548a8173119ce9",
"license": "CC-BY-4.0",
"filename": "validation-500-b.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "validation-500-b",
"client_artifact_id": "validation-500-b-raw",
"raw_requests_included": false
}
],
"client_trial_id": "validation-500-b-trial-1",
"duration_seconds": 58.20721769499418
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"validation-500-b"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 58.20721769499418,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.59,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51540,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.59,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:58.207Z",
"relation": {
"type": "sweep_point",
"group": "repetition"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "703b6b48dba8c0504795932829b2e275690e0c3c0ad1fb8d63548a8173119ce9",
"license": "CC-BY-4.0",
"filename": "validation-500-b.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "validation-500-b",
"client_artifact_id": "validation-500-b-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "validation-500-b",
"duration_seconds": 58.20721769499418
},
{
"name": "Full 500-request validation C",
"status": "completed",
"trials": [
{
"status": "completed",
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 8.64,
"measurement_status": "measured"
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "8a51bbb7c93cd36378e17860ee8513afd424d3e4b203f91e5145e1a86dc0bbbf",
"license": "CC-BY-4.0",
"filename": "validation-500-c.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "validation-500-c",
"client_artifact_id": "validation-500-c-raw",
"raw_requests_included": false
}
],
"client_trial_id": "validation-500-c-trial-1",
"duration_seconds": 57.87037037037037
}
],
"results": {
"total_tokens": {
"unit": "tokens",
"value": 3000500,
"derivation": "successful_requests × (fixed_input_tokens + fixed_output_tokens)",
"input_run_ids": [
"validation-500-c"
],
"measurement_status": "derived"
},
"failed_requests": 0,
"duration_seconds": {
"unit": "s",
"value": 57.87037037037037,
"measurement_status": "measured"
},
"requests_per_second": {
"unit": "req/s",
"value": 8.64,
"measurement_status": "measured"
},
"successful_requests": 500,
"input_tokens_per_second": {
"unit": "tok/s",
"value": 51840,
"measurement_status": "measured"
},
"output_tokens_per_second": {
"unit": "tok/s",
"value": 8.64,
"measurement_status": "measured"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype auto"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "auto",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:57.870Z",
"relation": {
"type": "sweep_point",
"group": "repetition"
},
"warnings": [],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "8a51bbb7c93cd36378e17860ee8513afd424d3e4b203f91e5145e1a86dc0bbbf",
"license": "CC-BY-4.0",
"filename": "validation-500-c.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "validation-500-c",
"client_artifact_id": "validation-500-c-raw",
"raw_requests_included": false
}
],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "validation-500-c",
"duration_seconds": 57.87037037037037
},
{
"name": "Scheduler token budget 12288 (startup failure)",
"status": "failed",
"failure": {
"stage": "runtime_startup",
"category": "kv_cache_allocation",
"exit_code": 1,
"reproducible": true,
"message_sanitized": "KV cache allocation failed at the requested scheduler token budget.",
"replacement_client_run_id": "scheduler-11776"
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"max_model_len": 6144,
"max_num_batched_tokens": 12288
}
}
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"revision_kind": "commit"
},
"relation": {
"type": "sweep_point",
"group": "max_num_batched_tokens"
},
"workload": {
"mode": "online",
"task": "text_generation",
"concurrency": 8,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"output_tokens": {
"kind": "fixed",
"value": 1
}
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "d6d09aecbaecfb556265244550b804c38093d029f545384a6e60148635efc7ac",
"license": "CC-BY-4.0",
"filename": "startup-sanitized.log",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "scheduler-12288-failed",
"client_artifact_id": "scheduler-12288-failed-raw",
"raw_requests_included": false
}
],
"client_run_id": "scheduler-12288-failed"
},
{
"name": "Interrupted preliminary screening",
"status": "interrupted",
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"max_num_batched_tokens": 8192
}
}
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"revision_kind": "commit"
},
"relation": {
"type": "other",
"group": "screening"
},
"warnings": [
"Interrupted after 73 of 500 requests; excluded from completed-run rankings."
],
"workload": {
"mode": "online",
"task": "text_generation",
"concurrency": 8,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500
},
"artifacts": [
{
"kind": "benchmark_result_json",
"sha256": "01541ded8566c149dd63e6cd577bef82790752293bc07cbc056e457c9c101686",
"license": "CC-BY-4.0",
"filename": "screening-interrupted.json",
"media_type": "application/json",
"size_bytes": 18432,
"visibility": "public",
"run_client_id": "screening-interrupted",
"client_artifact_id": "screening-interrupted-raw",
"raw_requests_included": false
}
],
"client_run_id": "screening-interrupted"
},
{
"name": "Derived four-replica throughput",
"status": "completed",
"trials": [],
"results": {
"requests_per_second": {
"unit": "req/s",
"value": 34.48,
"derivation": "Sum of four independently timed single-GPU replica rates; not a shared end-to-end workload.",
"input_run_ids": [
"gpu-rotation-0",
"gpu-rotation-1",
"gpu-rotation-2",
"gpu-rotation-3"
],
"measurement_status": "derived"
},
"input_tokens_per_second": {
"unit": "tok/s",
"value": 206880,
"derivation": "Derived requests/s × fixed 6000 input tokens.",
"input_run_ids": [
"derived-four-replica"
],
"measurement_status": "derived"
}
},
"runtime": {
"engine": "vllm",
"version": "0.27.1",
"parameters": {
"vllm": {
"moe_backend": "FLASHINFER_CUTLASS",
"max_model_len": 6144,
"attention_backend": "TRITON_ATTN",
"enable_prefix_caching": false,
"future_scheduler_knob": "preserved",
"gpu_memory_utilization": 0.95,
"max_num_batched_tokens": 10240
}
},
"parallelism": {
"data": 1,
"tensor": 1,
"pipeline": 1,
"replicas": 1
},
"command_sanitized": "vllm serve MODEL --max-model-len 6144 --max-num-batched-tokens 10240 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3"
},
"subject": {
"type": "model",
"source": "huggingface",
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4",
"modalities": [
"text"
],
"architecture": "Gemma4ForConditionalGeneration",
"quantization": {
"scope": "all_linears",
"format": "compressed-tensors",
"method": "nvfp4",
"provider": "unsloth",
"weight_dtype": "nvfp4",
"kv_cache_dtype": "fp8_e4m3",
"activation_dtype": "bfloat16"
},
"revision_kind": "commit",
"parameter_count_total": 26000000000,
"parameter_count_active": 4000000000
},
"ended_at": "2026-08-15T16:00:58.004Z",
"relation": {
"type": "aggregate",
"group": "replica_aggregate"
},
"warnings": [
"This aggregate is derived from independent replicas and was not measured end-to-end."
],
"workload": {
"mode": "online",
"task": "text_generation",
"client": {
"placement": "same-host",
"overhead_included": true
},
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"stopping": {
"exact_output_tokens": true
},
"streaming": false,
"tokenizer": {
"revision": "20df0542b1a86ce19f495ac2eca2c7c12bce82f9",
"identifier": "unsloth/gemma-4-26B-A4B-it-NVFP4"
},
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"endpoint_type": "openai-compatible-chat",
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 2000,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"reset_policy": "server_restart",
"cache_enabled": false,
"repetition_ratio": 0,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"artifacts": [],
"started_at": "2026-08-15T16:00:00.000Z",
"environment": {
"extensions": {
"runpile.demo.gpu_rotation": 0
},
"accelerators": [
{
"count": 1,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"device_indices": [
0
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
]
},
"client_run_id": "derived-four-replica",
"duration_seconds": 58.004640371229705
}
],
"tags": [
"binary-judge",
"gemma-4",
"llm-inference",
"prefill-heavy",
"rtx-5090"
],
"title": "Gemma 4 NVFP4 throughput optimization on RTX 5090",
"shared": {
"workload": {
"mode": "online",
"task": "text_generation",
"dataset": {
"license": "CC0-1.0",
"generator": "vllm-random",
"synthetic": true
},
"sampling": {
"temperature": 0,
"max_output_tokens": 1
},
"streaming": false,
"concurrency": 8,
"random_seed": 42,
"input_tokens": {
"kind": "fixed",
"value": 6000
},
"output_tokens": {
"kind": "fixed",
"value": 1
},
"request_count": 500,
"arrival_process": "closed_loop",
"prefix_behavior": {
"cache_state": "cold",
"cache_enabled": false,
"shared_prefix_tokens": 0
},
"warmup_request_count": 8,
"request_rate_per_second": null
},
"environment": {
"os": "Ubuntu 24.04.2 LTS",
"cpu": {
"model": "AMD Ryzen Threadripper PRO",
"logical_cores": 64,
"physical_cores": 32
},
"kernel": "6.11.0-26-generic",
"topology": "Four discrete PCIe GPUs, one serving process per GPU.",
"accelerators": [
{
"count": 4,
"model": "GeForce RTX 5090",
"driver": "575.64.03",
"vendor": "NVIDIA",
"architecture": "Blackwell",
"interconnect": "PCIe 5.0; no NVLink",
"device_indices": [
0,
1,
2,
3
],
"power_limit_watts": 575,
"memory_bytes_per_device": 34359738368
}
],
"driver_versions": {
"nvidia": "575.64.03",
"cuda_toolkit": "12.9"
},
"host_architecture": "x86_64",
"system_memory_bytes": 274877906944,
"virtual_environment": {
"python": "3.12.10",
"dependency_lock_hash": "074c2809a1a0ddf856a6b751e0b20773f8cc0828e7015e49d4daa7678dc38f54"
}
},
"measurement_method": {
"harness": "vllm bench serve",
"version": "0.27.1",
"clock_source": "CLOCK_MONOTONIC",
"warmup_semantics": "Eight requests completed before the timed batch.",
"duration_semantics": "First timed request dispatch through final response completion.",
"tokenization_included": true,
"model_loading_included": false,
"retry_failure_treatment": "Failed requests remain in the failure count and are not retried.",
"client_overhead_included": true,
"metric_definition_version": "runpile-generation-1"
}
},
"license": "CC-BY-4.0",
"summary": "Checkpoint, concurrency, GPU rotation, KV-cache, and scheduler sweeps for a prefill-heavy binary-judge workload.",
"artifacts": [
{
"kind": "methodology_markdown",
"sha256": "a62e7baea6125e3df45961e257f36b4518e2d301df21e137b7f2b706f5016329",
"license": "CC-BY-4.0",
"filename": "methodology.md",
"media_type": "text/markdown",
"size_bytes": 4200,
"visibility": "public",
"client_artifact_id": "experiment-notes",
"raw_requests_included": false
}
],
"objective": "Maximize total throughput for a large fixed-input binary-judge batch while retaining enough evidence to reproduce and interpret every configuration.",
"extensions": {
"runpile.demo.fixture": {
"generated": true,
"production_seed": true
}
},
"provenance": {
"tool": "Codex",
"notes": "Synthetic demonstration data modeled on a motivating local benchmark. Values are clearly labeled demo data outside tests.",
"source": "agent",
"generated_at": "2026-08-16T14:00:00.000Z",
"tool_version": "demo-fixture-generator-1"
},
"visibility": "public",
"conclusions": "The Unsloth checkpoint with a 10,240 scheduler token budget produced the best measured single-replica result around 8.65 requests/s. Larger budgets did not improve throughput and 12,288 failed at startup. Rotation results do not indicate a material device outlier.",
"limitations": [
"Synthetic tokens do not represent semantic prompt distributions.",
"The four-replica aggregate is derived from independent rates, not timed end-to-end.",
"Power was not measured."
],
"methodology": "One model replica per RTX 5090. Synthetic fixed 6,000-token prompts and greedy one-token outputs. Eight warmup requests precede timed closed-loop runs. Scheduler candidates use short screenings followed by 500-request validation near the optimum.",
"schema_version": "1.0",
"publisher_notes": {
"warnings": [
"Two GPU rotation artifacts intentionally share a display filename to demonstrate why IDs and hashes are authoritative."
],
"redactions": [
"Local username, hostname, absolute paths, and LAN addresses were removed."
],
"omitted_files": [
"unsanitized-server.log",
"raw-prompts.jsonl"
],
"public_data_license_affirmed": true
},
"client_submission_id": "0198b812-cde0-7a2d-8cd7-51d426080001"
}