One Success Isn't Reliability: How to Test an AI Agent Before Trusting It With Operations

One Success Isn't Reliability: How to Test an AI Agent Before Trusting It With Operations

The direct answer: a successful demo proves nothing about an AI agent's reliability, because the right question is not whether it completed the task once, but whether it completes it correctly every time. In a study Microsoft published on 20 August 2026, the strongest model scored 65.36% when each task was run once, yet passed all twenty attempts on only 25.25% of tasks. That gap is precisely the distance between an agent that impresses you in a meeting and an agent you can trust with customer data.

What Microsoft actually built

The tool is called Thinkingbox, an open-source testing environment released under the MIT license on GitHub. The idea is simple and clever: instead of wiring tools directly into the agent, you define them as MCP-compatible servers, run the agent inside an isolated session with its own backing database, and record the full execution trace.

The essential point is in how it grades. The framework does not read what the agent said it did; it inspects what actually changed in the database once the task ended. That distinction is not a technical detail. Agents are very good at writing a convincing summary of work that never happened, and any evaluation based on the conversation transcript measures the quality of the prose, not the quality of the execution.

On top of that environment the team built a benchmark called Thinkingbox-bench containing 507 policy-conditioned workflows across five domains: retail and e-commerce, travel and hospitality, auto insurance, internal IT support at a neobank, and IT and HR support at a consulting firm. Each task was run twenty times against twelve models, both proprietary and open-weight.

Why that number matters to a business owner

Imagine you hand an agent the job of processing return requests in your store. A 65% success rate looks acceptable on paper. Operationally it means a third of requests end in a wrong outcome: a refund issued twice, an order closed without restocking inventory, or an exception granted to a customer your policy does not allow it for.

Worse, the failures in this study were not obvious crashes. The paper explicitly describes many failures as terminating cleanly and taking steps that looked valid, while the final system state was still wrong. No error message, no halt, nothing lighting up on a monitoring dashboard. Just a log that reads normally and data that quietly became incorrect.

A failure that stops the system is annoying but visible. A failure that completes the flow and leaves your data wrong is the one that actually costs you, because you discover it a month later during reconciliation.

This also explains a common source of disappointment. The pilot passes because the pilot ran the task once against clean data. Real operation is the same task thousands of times against messy accumulated state, and that is exactly where the difference between 65% and 25% shows up.

What to do before you adopt any agent

• Demand repeated runs, not a single demo. Make your acceptance criterion twenty consecutive runs of the same task against the same data. A vendor who refuses is telling you something.
• Measure data state, not the reply text. Define in advance what the record in your database should look like after the task, and compare reality against that. Do not settle for reading the agent's own summary.
• Test policies, not just easy tasks. Most of your work is not open-ended; it is constrained by rules: return windows, credit limits, exceptions for priority customers. An agent that succeeds without constraints can collapse at the first real condition.
• Build the test set from your own cases. Take twenty real cases from recent months, including the odd ones that confused your staff, and turn them into a fixed suite you run against any model before adopting it and after every update.
• Grant authority in stages. Start with an agent that proposes but does not execute, then let it perform low-impact operations, then widen its scope as a genuine measurement record accumulates. Authority should be earned with data, not with impressions.

How we handle this at Origami

When we build a system that includes an AI agent, we treat the agent like any other software component: it gets an automated test suite, verification checks that inspect database state after each operation, and a clear handoff point to a human when a confidence threshold is crossed. The model itself stays a swappable setting rather than something welded into the code, because today's best model may not be the best one three months from now.

The conclusion worth taking from this study is not that agents are unusable. It is that the yardstick we were judging them by was wrong. Agents today are powerful tools inside a well-defined and continuously measured scope, and very weak when a sensitive process is handed to them on the strength of a convincing demo. The difference is not in the model; it is in the engineering discipline around it.

Sources

• Research paper: One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows — arXiv (published 20 August 2026)
• Open-source repository: microsoft/thinkingbox on GitHub