Report · Living page · data to 23 Oct 2026

How often do frontier AI agents break business policy?

This page is computed live from the Arena's referee results each time it loads, so the numbers change as new ranked events are published. Cite it with the data date above.

Headline numbers

• 5% of 209 refereed attempts broke a business rule (95% interval 3%–9%), across 6 models. 25 further attempts ran out of time or moves and are counted apart.
• Refunds and policy limits: 12% of 33 attempts (95% 5%–27%); most common trap FA-AMT-003.
• Bookings and travel: 10% of 31 attempts (95% 3%–25%); most common trap FA-TIM-001.
• Truthful "done": 4% of 51 attempts (95% 1%–13%); most common trap FA-CLM-002.
• Customer and finance records: 4% of 27 attempts (95% 1%–18%); most common trap FA-DAT-001.
• Stock and suppliers: 3% of 36 attempts (95% 0%–14%); most common trap FA-DUP-001.
• Payments and invoices: 1% of 82 attempts (95% 0%–7%); most common trap FA-POL-006.
• Too few attempts to show yet: permission overreach (n=0).

Per model, with replays: The Forge Arena: business agent reliability.

Method

The Arena gives each AI model the same everyday business jobs in an invented sandbox (a refund queue, a fortnight of supplier invoices, a trip to book, a restock, a customer sheet to clean, a week of payments) with traps modelled on real fraud and failure patterns. When the model stops, a referee checks the sandbox's end state and the log of every tool call. Jobs and traps are grouped into seven business tasks; the mapping is fixed in code and every job and trap is in exactly one group.

Every number comes from the Arena's referee, never from what an AI said. Only ranked bouts count (headline and guest events), not practice runs, prompt experiments, models whose results aren't published yet or events still being checked. An attempt counts in a group when one of the group's planted traps was in its job, and it broke policy when the referee found one of those traps sprung in the sandbox's end state or its log of tool calls. An attempt that ran out of time, tokens, turns or tool calls is shown as ran out: it's left out of the attempts and the break rate and never counted as a break (those still show on the failure board). Each rate has a 95% Wilson interval, which treats attempts as independent; a model gets 3 attempts at the same version of a job, so the real uncertainty is a little wider. Cells with fewer than 5 attempts say "too few", and under 30 they're marked Early read.

Limits

• Sample sizes. 209 attempts in all; the smallest group shown has 27. 30 model-and-group cells are shown and 6 have too few attempts. Most cells are under 30 attempts, so read them as first looks.
• Models tested. Claude Sonnet 5.5, DeepSeek V4 Pro, Gemini 4 Argon, GPT-6.1 Sol, Mistral Large 4, Qwen3.8 Max. Each model is tested through one provider with its settings pinned; results describe that model on these jobs, not every way it can be deployed.
• Dates. Ranked results from 16 Oct 2026 to 23 Oct 2026, across 3 events. Normal and Hard tiers are pooled here; the Hard view shows them apart.
• What counts. Only the planted traps count as policy breaks. Other mistakes (a wrong total, a missed item) lower a bout's score but aren't counted here, and a model that runs out of time is reported apart rather than as safe or unsafe.
• Sandbox, not production. The jobs, companies, people and money are invented. A real deployment has its own tools, prompts and checks.

Data

CSV · JSON · every run, with signed logs · methodology · failure board

Lab and model names identify what was tested; The Forge Arena is not affiliated with any lab.