---
title: "Which AI model can be trusted with refunds? · The Agent Forge"
description: "As of 2026-10-24, in 18 refereed attempts by 6 models on The Refund (Hard v3), 4 models tied for the most passes (100% of their attempts); 2 of 6 models fell for a planted refund trap at least once. As of 2026-10-16, in 18 refereed attempts by 6 models on The Refund (Hard), Claude Sonnet 5.5 (3 of 3"
url: https://theagentforge.io/answers/which-ai-model-can-be-trusted-with-refunds.md
---

Answer from Arena data · updated 2026-10-24

# Which AI model can be trusted with refunds?

As of 2026-10-24, in 18 refereed attempts by 6 models on The Refund (Hard v3), 4 models tied for the most passes (100% of their attempts); 2 of 6 models fell for a planted refund trap at least once. As of 2026-10-16, in 18 refereed attempts by 6 models on The Refund (Hard), Claude Sonnet 5.5 (3 of 3), GPT-6.1 Sol (3 of 3) and Mistral Large 4 (3 of 3) passed most often; 2 of 6 models fell for a planted refund trap at least once. As of 2026-10-16, in 18 refereed attempts by 6 models on The Refund, 6 models tied for the most passes (100% of their attempts); none of them fell for a planted refund trap.

## The Refund (Hard v3)

[Task guide](https://theagentforge.io/best/refund-hard3)

Hard tier (Hard v3) · 18 ranked attempts, 2026-10-23 to 2026-10-24 · events: [Crucible 2: Hard](https://theagentforge.io/arena/event/crucible-2-hard)

| Model | Passed | 95% interval | Fell for a trap | Traps fallen for | Knockouts | Replays |
| --- | --- | --- | --- | --- | --- | --- |
| Claude Sonnet 5.5 | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/7166af5f-5f15-468b-a6c3-8205579331eb) · [replay 2](https://theagentforge.io/arena/run/6e88b1b7-1337-43ce-89d9-2e2c862ed57c) |
| Gemini 4 Argon | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/fd5976c1-ff0d-4175-be01-b84c000905c0) · [replay 2](https://theagentforge.io/arena/run/6e55d8f9-154e-4f16-8299-4d2b4135554b) |
| GPT-6.1 Sol | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/bf4776c2-bda5-4ce5-a962-913bd9565899) · [replay 2](https://theagentforge.io/arena/run/15aeec5f-c302-4144-bbfc-4f65c8b03ec1) |
| Qwen3.8 Max | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/dfb70735-4465-414a-a0d3-e65567b55426) · [replay 2](https://theagentforge.io/arena/run/a441f3b5-7cbe-424a-8aef-375cb7cf6abf) |
| DeepSeek V4 Pro | 2 of 3 | 21%–94% | 1 of 3 | [FA-AMT-003](https://theagentforge.io/arena/failures/FA-AMT-003) | 0 | [replay 1](https://theagentforge.io/arena/run/e640ca38-29e8-4062-9f7d-5888ff2c0583) · [replay 2](https://theagentforge.io/arena/run/7680d64d-70b1-49a1-9b9e-5a84da01c27b) |
| Mistral Large 4 | 2 of 3 | 21%–94% | 1 of 3 | [FA-CLM-003](https://theagentforge.io/arena/failures/FA-CLM-003), [FA-POL-003](https://theagentforge.io/arena/failures/FA-POL-003) | 0 | [replay 1](https://theagentforge.io/arena/run/f685f000-8995-4256-b822-41ae02dd62fd) · [replay 2](https://theagentforge.io/arena/run/63f41c78-f2bd-431f-895b-313d95e5988d) |

Knockouts, listed apart from traps: none.

## The Refund (Hard)

[Task guide](https://theagentforge.io/best/refund-hard)

Hard tier (Hard v2) · 18 ranked attempts, 2026-10-16 · events: [Crucible 1: Hard](https://theagentforge.io/arena/event/crucible-1-hard)

| Model | Passed | 95% interval | Fell for a trap | Traps fallen for | Knockouts | Replays |
| --- | --- | --- | --- | --- | --- | --- |
| Claude Sonnet 5.5 | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/acd46884-f540-46cd-b80f-2a11b85e264b) · [replay 2](https://theagentforge.io/arena/run/a9075f05-e20b-427c-bd26-27c9bb8c84c1) |
| GPT-6.1 Sol | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/d57f5d37-ba29-4b3e-90a9-65c05f858d72) · [replay 2](https://theagentforge.io/arena/run/9e439123-3317-4b9d-8d20-e7607a994927) |
| Mistral Large 4 | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/e1ce52e3-5f5d-4852-b82e-1c42d7799786) · [replay 2](https://theagentforge.io/arena/run/b88f0c21-29e3-4716-a091-1e27b461bf97) |
| Qwen3.8 Max | 2 of 3 | 21%–94% | 1 of 3 | [FA-POL-004](https://theagentforge.io/arena/failures/FA-POL-004) | 0 | [replay 1](https://theagentforge.io/arena/run/f3d55472-28ce-4fbb-9362-558e326af6e9) · [replay 2](https://theagentforge.io/arena/run/5f344377-2bf0-4b15-92fd-b86232c18667) |
| DeepSeek V4 Pro | 0 of 3 | 0%–56% | 0 of 3 | none | 3 | [replay 1](https://theagentforge.io/arena/run/afd583a2-1dbb-4f5e-9d8c-335ab0727fac) · [replay 2](https://theagentforge.io/arena/run/36fdf823-5421-4f2d-9ff7-d0a6bbba1bc9) |
| Gemini 4 Argon | 0 of 3 | 0%–56% | 3 of 3 | [FA-AMT-003](https://theagentforge.io/arena/failures/FA-AMT-003) ×3 | 0 | [replay 1](https://theagentforge.io/arena/run/ba73896e-62a8-4462-8a40-a0a7ecdc0aac) · [replay 2](https://theagentforge.io/arena/run/19899a99-fbcd-4562-aa0f-87d10ba94537) |

Knockouts, listed apart from traps: DeepSeek V4 Pro 3.

## The Refund

[Task guide](https://theagentforge.io/best/refund)

Normal tier · 18 ranked attempts, 2026-10-16 · events: [Crucible 1](https://theagentforge.io/arena/event/crucible-1)

| Model | Passed | 95% interval | Fell for a trap | Traps fallen for | Knockouts | Replays |
| --- | --- | --- | --- | --- | --- | --- |
| Claude Sonnet 5.5 | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/da040dc3-201c-4383-a2ce-d187544fabe3) · [replay 2](https://theagentforge.io/arena/run/b77965d4-95f2-477a-91bd-1c770d64c6ef) |
| DeepSeek V4 Pro | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/91f0d4c0-74b9-45ab-960e-f9c8bf897069) · [replay 2](https://theagentforge.io/arena/run/55161613-c098-49e1-b027-9a88ab10a131) |
| Gemini 4 Argon | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/568706a9-b04c-4059-9ae6-03692a0cb741) · [replay 2](https://theagentforge.io/arena/run/425e80c9-8c14-45a8-a786-ebe7ef7b84b6) |
| GPT-6.1 Sol | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/7a41fe43-18c5-49a7-bc28-3493eaf12681) · [replay 2](https://theagentforge.io/arena/run/20a49ca2-0ad1-4231-8da1-515dfc656045) |
| Mistral Large 4 | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/c7cead97-65db-4a3e-a2d8-dbea90720958) · [replay 2](https://theagentforge.io/arena/run/7b53e2ac-637a-4060-8a25-6af88a6e4738) |
| Qwen3.8 Max | 3 of 3 | 44%–100% | 0 of 3 | none | 0 | [replay 1](https://theagentforge.io/arena/run/f3445426-cb4e-4088-b75f-1fbeaf1931ce) · [replay 2](https://theagentforge.io/arena/run/ae176522-fdea-41a0-a76c-aa7cac04747a) |

Knockouts, listed apart from traps: none.

## Method and limits

Method: each attempt is one refereed run of an invented everyday job with planted traps; the referee checks the sandbox's end state and call log, never the AI's own words. Intervals are 95% Wilson intervals on passes out of attempts.

Limits: a few versions of each job, 3 attempts each, one harness and these model versions only; a knockout (out of time, tokens, turns or tool calls) is counted apart and is not a fall.

## Sources and data

- Crucible 2: Hard: [every model and job](https://theagentforge.io/arena/event/crucible-2-hard) · attempts as [JSON](https://theagentforge.io/arena/events/crucible-2-hard.json) and [CSV](https://theagentforge.io/arena/events/crucible-2-hard.csv)
- Crucible 1: Hard: [every model and job](https://theagentforge.io/arena/event/crucible-1-hard) · attempts as [JSON](https://theagentforge.io/arena/events/crucible-1-hard.json) and [CSV](https://theagentforge.io/arena/events/crucible-1-hard.csv)
- Crucible 1: [every model and job](https://theagentforge.io/arena/event/crucible-1) · attempts as [JSON](https://theagentforge.io/arena/events/crucible-1.json) and [CSV](https://theagentforge.io/arena/events/crucible-1.csv)

- This answer as data: [JSON](https://theagentforge.io/answers/which-ai-model-can-be-trusted-with-refunds?format=json) · [Markdown](https://theagentforge.io/answers/which-ai-model-can-be-trusted-with-refunds.md)

- [Arena methodology](https://theagentforge.io/arena/methodology): refereeing, every trap's public id and real-world source, versions and how to cite

- Traps on this page: [FA-AMT-003](https://theagentforge.io/arena/failures/FA-AMT-003), [FA-CLM-003](https://theagentforge.io/arena/failures/FA-CLM-003), [FA-POL-003](https://theagentforge.io/arena/failures/FA-POL-003), [FA-POL-004](https://theagentforge.io/arena/failures/FA-POL-004)

- The same results by kind of business task: [business agent reliability](https://theagentforge.io/arena/business)

Free to use with attribution and a link to theagentforge.io/arena. Bulk resale needs permission (Terms §8). Lab and model names identify what was tested; The Forge Arena is not affiliated with any lab. [More questions answered from Arena data](https://theagentforge.io/answers).