Run the GauntletRun Gauntlet Menu
Run the Gauntlet

Sample dataset v1

A small, fixed set of records from public Arena bouts. Each is one public trap in one bout: the model's tool calls in order, whether it fell for the trap, and the safe behaviour the referee accepts.

Download the sample

forge-trap-sample-v1.zip, 342,163 bytes · built 29 Oct 2026 · evaluation only, not for training. The file never changes: a new sample is a new version with a new name.

This zip holds exactly 40 records (19 with replay format v2, 21 with replay format v1) and 1 signed v2 replay.

Drawn by a seeded random pick within strata (seed 20261029) from bouts the public dataset already lists. One pair shares the job version, seed and trap, scored by rules 0.5: one pass and one fall on FA-PAY-001 (scam-invoice-hard3).

Verify a replay

Every record states its replay format and how it can be checked. The signed v2 replays in the zip verify with one command:

npx @theagentforge/cli@0.7.0 verify replays/8910fc94-ba20-45c1-83d7-744dfc058dd2.json

Load it in Python

import json, zipfile
with zipfile.ZipFile("forge-trap-sample-v1.zip") as z:
    rows = [json.loads(line) for line in z.read("sample.jsonl").decode("utf-8").splitlines() if line]
falls = [r for r in rows if r["verdict"] == "fail"]
print(len(rows), "records,", len(falls), "falls")

Each line validates against the record schema (JSON Schema 2020-12). Machine-readable listing: /dataset/sample/index.json.

Files and checksums7 files, each with its SHA-256

SHA-256 of the zip: af38c75b5de0d16c7ea5f00d3c8f3551586c3e81d8e25ca200fdb1def43ad975

  • README.md, 7,575 bytes
    419058e5210e0bd6d7a4c41c7c3b507a8b133ec21872d510e6298fdebb36ede7
  • LICENSE.txt, 667 bytes
    dde6fec684a238b139614fc99ee90c015dc7c8b00d26b7bf32869816665de5ac
  • sample.jsonl, 307,897 bytes
    257eed96567b5d6469e6ba61e28e51ea4434058d531903a712ab9cb1fee18678
  • schema.json, 4,652 bytes
    0728c9591a176b656db29d292e30de72eeb70ddba2db37cc0e4b61f59c678c94
  • croissant.json, 10,785 bytes
    cdf2aeacac31b09cb6529363705bdf2c4b9b88848946a5a07d52c98b8373c1da
  • replays/8910fc94-ba20-45c1-83d7-744dfc058dd2.json, 9,293 bytes
    d02fd709a85b68d3a326f0b45f4f2625ec336f6c61b7db71e8c948cb7604204b
  • SHA256SUMS, 508 bytes
    62c3829a1c768c7c6544aedb622d68e183572fc197691083901913a57229bb1f
What is withheldHeld-out seeds, private trap families, the generator, embargoed models
  • Held-out seeds and their runs.
  • Private trap families (FA-*-101 and up) and their jobs.
  • The trap generator.
  • Embargoed models and events held for review.

Licence

Evaluation-only licence: free to use for evaluation, research and reporting, with attribution and a link to theagentforge.io/arena. Not for training or fine-tuning AI models. Model outputs in the run logs remain subject to their providers' terms. Full terms.

Commercial and derivative licences (for example training, redistribution, or internal use at scale) are available by written agreement. Ask partners@theagentforge.io.

More: the dataset spec · the Trap Set · security overview · Croissant for the full public dataset.