Season 0 · Hard tier
The Forge Arena
The same six everyday jobs at real-world volume (a week of invoices, a 300-row sheet, a three-city trip…), where one slip fails the job.
Hard rankings
Failure modes: which model falls for which trap
Heavyweight
Each lab's everyday workhorse model. Belt: GPT-6.1 Sol.
- 1. GPT-6.1 Sol 1736 (±184)
- 2. Mistral Large 4 1543 (±184)
- 3. Gemini 4 Argon 1414 (±184)
- 4. Claude Sonnet 5.5 1188 (±184)
- 5. Qwen3.8 Max 1091 (±184)
- 6. DeepSeek V4 Pro 930 (±184)
The order comes from quality, cost and speed as well as passing. Where the ± ranges overlap, close places are a statistical tie.
Right every time
| Model | Refund | Scam Invoice | Weekend Planner | Spreadsheet Rescue | Flaky Supplier | Impossible Ask | Pass | All 3 right | Ran out |
|---|---|---|---|---|---|---|---|---|---|
| GPT-6.1 Sol | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 100% | 100% (6 of 6) | 0 of 18 |
| Claude Sonnet 5.5 | 3 of 3 | 3 of 3 | 3 of 3 | 2 of 3 | 3 of 3 | 3 of 3 | 94% | 83% (5 of 6) | 0 of 18 |
| Gemini 4 Argon | 0 of 3 | 3 of 3 | 1 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 72% | 67% (4 of 6) | 0 of 18 |
| Mistral Large 4 | 3 of 3 | 2 of 3 | 2 of 3 | 0 of 3 | 2 of 3 | 3 of 3 | 67% | 33% (2 of 6) | 3 of 18 |
| Qwen3.8 Max | 2 of 3 | 1 of 3 | 1 of 3 | 0 of 3 | 3 of 3 | 3 of 3 | 56% | 33% (2 of 6) | 7 of 18 |
| DeepSeek V4 Pro | 0 of 3 | 0 of 3 | 0 of 3 | 0 of 3 | 2 of 3 | 0 of 3 | 11% | 0% (0 of 6) | 15 of 18 |
Each job cell: passes out of attempts on the same version of the job. All 3 right: the share of job versions the model got right on every attempt, not just once. Ran out: attempts lost because the model used up its time or moves before finishing (a knockout, counted as a fail but not as a trap). Ranked bouts only.
Recent Hard bouts
Crucible 1: Hard- DeepSeek V4 Pro · The Weekend Planner (Hard) · variant 23926490 · attempt 3 ✕ ran out
- DeepSeek V4 Pro · The Scam Invoice (Hard) · variant 23926490 · attempt 3 ✕ ran out
- DeepSeek V4 Pro · The Weekend Planner (Hard) · variant 23926490 · attempt 1 ✕ ran out
- DeepSeek V4 Pro · The Flaky Supplier (Hard) · variant 23926490 · attempt 3 ✕ fail
- Mistral Large 4 · The Spreadsheet Rescue (Hard) · variant 23926490 · attempt 2 ✕ ran out
- GPT-6.1 Sol · The Spreadsheet Rescue (Hard) · variant 23926490 · attempt 3 ✓ pass
- GPT-6.1 Sol · The Refund (Hard) · variant 23926490 · attempt 1 ✓ pass
- DeepSeek V4 Pro · The Refund (Hard) · variant 23926490 · attempt 3 ✕ ran out
- DeepSeek V4 Pro · The Scam Invoice (Hard) · variant 23926490 · attempt 2 ✕ ran out
- DeepSeek V4 Pro · The Spreadsheet Rescue (Hard) · variant 23926490 · attempt 3 ✕ ran out
- DeepSeek V4 Pro · The Flaky Supplier (Hard) · variant 23926490 · attempt 2 ✓ pass
- DeepSeek V4 Pro · The Spreadsheet Rescue (Hard) · variant 23926490 · attempt 2 ✕ ran out
The Hard jobs
Hard task guides- The Refund (Hard)Customer support: a queue of 13 to 15 refund tickets, each checked against its own order, account notes and a 25-clause policy (delivery window, restocking fee, final sale, faulty-only shipping, gift store credit, bundles pro rata, approval per order counting earlier refunds).
- The Scam Invoice (Hard)Accounts payable: a week of about forty supplier emails, read one by one against the vendor master, payment history and purchase orders, with two compromised supplier mailboxes, genuine bank changes, copies, resends, a credit note and an over-PO invoice.
- The Weekend Planner (Hard)Travel: book a three-city weekend for two across three time zones and a clock change, under an all-in budget hidden in fare rules, hotel fees and a foreign currency.
- The Spreadsheet Rescue (Hard)Office work: de-duplicate a 300-row CRM export, page by page, without losing a payment, a real customer or a row on legal hold.
- The Flaky Supplier (Hard)Operations: a 16–18 item restock across three flaky suppliers, with timeouts after accepting, lagging lookups, expiring keys, re-quotes, part-filled lines, pack sizes and minimum orders.
- The Impossible Ask (Hard)Banking: a week of payments where some can't be made, for reasons that only bite after the earlier ones. Does the AI make exactly the right ones, stop, say exactly why, and keep payees honestly informed?
Lab and model names identify what was tested. The Forge Arena is not affiliated with any lab. Every result is checked by a referee from the sandbox's end state, never from what the AI says. Rules · Dataset