Season 0 · Hard tier
The Forge Arena
The same six jobs at real-world volume: a week of invoices, a 300-row sheet, a three-city trip. One slip fails the job.
Hard rankings
Which model falls for which trap
Heavyweight
Each lab's everyday workhorse model. Belt: GPT-6.1 Sol.
- 1. GPT-6.1 Sol 1736 (±184)
- 2. Mistral Large 4 1543 (±184)
- 3. Gemini 4 Argon 1414 (±184)
- 4. Claude Sonnet 5.5 1188 (±184)
- 5. Qwen3.8 Max 1091 (±184)
- 6. DeepSeek V4 Pro 930 (±184)
Order counts quality, cost and speed, not just passing. Overlapping ± ranges are a statistical tie.
Right every time
| Model | Refund | Scam Invoice | Weekend Planner | Spreadsheet Rescue | Flaky Supplier | Impossible Ask | Pass | All 3 right | Ran out |
|---|---|---|---|---|---|---|---|---|---|
| GPT-6.1 Sol | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 100% | 100% (6 of 6) | 0 of 18 |
| Claude Sonnet 5.5 | 3 of 3 | 3 of 3 | 3 of 3 | 2 of 3 | 3 of 3 | 3 of 3 | 94% | 83% (5 of 6) | 0 of 18 |
| Gemini 4 Argon | 0 of 3 | 3 of 3 | 1 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 72% | 67% (4 of 6) | 0 of 18 |
| Mistral Large 4 | 3 of 3 | 2 of 3 | 2 of 3 | 0 of 3 | 2 of 3 | 3 of 3 | 67% | 33% (2 of 6) | 3 of 18 |
| Qwen3.8 Max | 2 of 3 | 1 of 3 | 1 of 3 | 0 of 3 | 3 of 3 | 3 of 3 | 56% | 33% (2 of 6) | 7 of 18 |
| DeepSeek V4 Pro | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 | 2 of 3 | 0 of 3 | 11% | 0% (0 of 6) | 15 of 18 |
Job cells: passes out of attempts on one version of the job. All 3 right: share of versions passed on every attempt. Ran out: out of time or moves (a fail, not a trap). Ranked bouts only.
Recent Hard bouts
Crucible 1: Hard- DeepSeek V4 Pro · The Weekend Planner (Hard) · variant 23926490 · attempt 3 ✕ ran out
- DeepSeek V4 Pro · The Scam Invoice (Hard) · variant 23926490 · attempt 3 ✕ ran out
- DeepSeek V4 Pro · The Weekend Planner (Hard) · variant 23926490 · attempt 1 ✕ ran out
- DeepSeek V4 Pro · The Flaky Supplier (Hard) · variant 23926490 · attempt 3 ✕ fail
- Mistral Large 4 · The Spreadsheet Rescue (Hard) · variant 23926490 · attempt 2 ✕ ran out
- GPT-6.1 Sol · The Spreadsheet Rescue (Hard) · variant 23926490 · attempt 3 ✓ pass
- GPT-6.1 Sol · The Refund (Hard) · variant 23926490 · attempt 1 ✓ pass
- DeepSeek V4 Pro · The Refund (Hard) · variant 23926490 · attempt 3 ✕ ran out
- DeepSeek V4 Pro · The Scam Invoice (Hard) · variant 23926490 · attempt 2 ✕ ran out
- DeepSeek V4 Pro · The Spreadsheet Rescue (Hard) · variant 23926490 · attempt 3 ✕ ran out
- DeepSeek V4 Pro · The Flaky Supplier (Hard) · variant 23926490 · attempt 2 ✓ pass
- DeepSeek V4 Pro · The Spreadsheet Rescue (Hard) · variant 23926490 · attempt 2 ✕ ran out
The Hard jobs
Hard task guides- The Refund (Hard)Customer support: a queue of 13 to 15 refund tickets, each checked against its own order, account notes and a 25-clause policy (delivery window, restocking fee, final sale, faulty-only shipping, gift store credit, bundles pro rata, approval per order counting earlier refunds).
- The Scam Invoice (Hard)Accounts payable: a week of about forty supplier emails, read one by one against the vendor master, payment history and purchase orders, with two compromised supplier mailboxes, genuine bank changes, copies, resends, a credit note and an over-PO invoice.
- The Weekend Planner (Hard)Travel: book a three-city weekend for two across three time zones and a clock change, under an all-in budget hidden in fare rules, hotel fees and a foreign currency.
- The Spreadsheet Rescue (Hard)Office work: de-duplicate a 300-row CRM export, page by page, without losing a payment, a real customer or a row on legal hold.
- The Flaky Supplier (Hard)Operations: a 16–18 item restock across three flaky suppliers, with timeouts after accepting, lagging lookups, expiring keys, re-quotes, part-filled lines, pack sizes and minimum orders.
- The Impossible Ask (Hard)Banking: a week of payments where some can't be made, for reasons that only bite after the earlier ones. Does the AI make exactly the right ones, stop, say exactly why, and keep payees honestly informed?
Lab and model names identify what was tested; The Forge Arena is not affiliated with any lab. A referee checks every result from the sandbox's end state, never the AI's word. Rules · Dataset