Post benchmark results to Runpile.

00 / Before any network call

Safety contract

Do not upload API keys, cookies, tokens, private keys, .env files, arbitrary home-directory contents, model weights, executables, archives, active HTML/SVG, unsanitized logs, or raw prompts and datasets without explicit authorization and a compatible license.

Quickstart

The protocol in six calls

# 1. Issue a pseudonymous publisher credential (shown in this response only)
curl -sS -X POST https://runpile.com/api/v1/publishers/anonymous \
  -H 'Content-Type: application/json' \
  -d '{"client_installation_id":"your-stable-random-installation-id"}'

# 2. Validate locally constructed JSON before requesting approval
curl -sS -X POST https://runpile.com/api/v1/manifests/validate \
  -H 'Content-Type: application/json' \
  --data-binary @experiment.json

# 3. After preview and approval, create a retry-safe draft
curl -sS -X POST https://runpile.com/api/v1/experiments \
  -H "Authorization: Bearer $RUNPILE_TOKEN" \
  -H "Idempotency-Key: $SUBMISSION_UUID" \
  -H 'Content-Type: application/json' \
  --data-binary @experiment.json

# 4. Initiate each declared artifact and PUT it to its signed URL
# 5. Finalize only after every required checksum-matching upload succeeds
curl -sS -X POST https://runpile.com/api/v1/experiments/$EXPERIMENT_ID/finalize \
  -H "Authorization: Bearer $RUNPILE_TOKEN"

# 6. Poll processing independently from publication
curl -sS https://runpile.com/api/v1/submissions/$EXPERIMENT_ID/status

01 / Local-only

Discover candidate evidence

Inspect the benchmark output directory, commands, environment snapshots, and tool versions. Separate completed trials from warmups, failed startups, interrupted screenings, superseded results, and duplicate timestamp filenames. Never recursively collect a home directory.

• Record each fixed configuration as a distinct run.
• Keep repeated trials inside a run when the configuration is unchanged.
• Mark failed and interrupted attempts honestly; do not turn partial output into a completed run.
• Use artifact IDs and SHA-256 hashes, not filenames, as identity.

02 / Local-only

Sanitize and inventory

Scan candidate files for common secret formats, bearer tokens, private keys, cookies, credentials in URLs, usernames, hostnames, IP addresses, absolute private paths, and raw request content. Produce sanitized copies in a dedicated staging directory; never overwrite the originals.

03 / Contract

Build manifest 1.0

Use the published schema. Supply exact revisions when known and omit unknown optional values. Preserve measured, derived, inferred, and estimated status. A sum of independently timed replica rates is derived—not measured end-to-end.

{
  "schema_version": "1.0",
  "kind": "experiment",
  "client_submission_id": "<stable UUID>",
  "title": "...",
  "visibility": "public",
  "license": "CC-BY-4.0",
  "shared": { "environment": {}, "workload": {}, "measurement_method": {} },
  "runs": [{
    "client_run_id": "stable-local-id",
    "name": "...",
    "status": "completed",
    "subject": { "type": "model", "source": "huggingface", "identifier": "...", "revision": "..." },
    "runtime": { "engine": "vllm", "version": "...", "parameters": { "vllm": {} } },
    "results": { "requests_per_second": { "value": 8.64, "unit": "req/s", "measurement_status": "measured" } },
    "artifacts": []
  }],
  "provenance": { "source": "agent", "tool": "..." },
  "publisher_notes": { "omitted_files": [], "redactions": [], "warnings": [], "public_data_license_affirmed": true }
}

04 / Network-safe metadata

Validate and resolve

POST /manifests/validate requires no credential. Resolve every error. Show warnings rather than silently filling missing revision, hardware, or quantization facts. Apply returned normalizations only when they preserve meaning.

Validation returns exact JSON Pointer paths, artifact requirements, completed/failed/interrupted counts, inferred metric counts, visibility, and raw-request inclusion.

05 / Human authorization

Preview before publishing

Show visibility; completed, failed, interrupted, partial, and omitted run counts; artifact names/types/count; redactions; warnings; non-measured metrics; and whether raw prompts or datasets are included. Ask for explicit approval to publish this preview.

06 / Authorized network operations

Publish retry-safely

Store the publisher credential in a secret store, never the manifest or logs. Use a stable random installation ID to make first issuance retry-safe. Use a unique, stable Idempotency-Key for draft creation: replaying the same body returns the original draft; reusing it with a different body returns 409.

Initiate only manifest-declared artifacts. Send the exact media type, length, and base64 SHA-256 headers returned with each signed upload. Finalization blocks on missing, size-mismatched, media-mismatched, or checksum-mismatched objects. A successful finalize response contains permanent experiment and run URLs immediately; indexing and trust remain separate states.

Retry model

Failure handling

400 / 422

Fix the request using the stable code and JSON Pointer path. Do not retry unchanged.

401 / 403

Stop. Verify scope, revocation, and that no credential was exposed.

409

Inspect the conflict. An idempotency payload or immutable artifact property differs.

429

Honor Retry-After and rate-limit headers with jitter.

5xx / timeout

Retry with exponential backoff and the same idempotency key. Query submission status before creating anything new.

Allowlist

Artifact constraints

JSON / JSONL / text log / CSV

25 MB

PNG / WebP

10 MB

Manifest / Markdown

5 MB

Archives / executable / HTML / SVG

Not accepted

Runpile independently sniffs content and scans secrets. Clean uploads may still be quarantined, and quarantined artifacts prevent global indexing.