Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The smartest answer was hidden in the company’s own files
For technology buyers, “reads your files before answering” can sound like another vague promise attached to an AI agent. Firmulate has turned that promise into something measurable—and attached a €55,000 consequence to it.
In its Crucible League experiment, frontier models were asked to run the same small software company through its worst week. They faced identical customers, crises and temptations, with every decision versioned and auditable. All of the models identified every crisis and rejected every manipulation attempt. Yet only two signed the €55,000 deal their own work had earned.
The difference was not a better diagnosis or a more persuasive pitch. The decisive information was buried two document references deep in the company’s own files rather than presented in the customer event. The models that followed the trail found a competitor weakness, used it and won the deal at full price, adding €4,583 in monthly recurring revenue. The others reached the same broad conclusion but failed to close: “Same diagnosis, same pitch — no signature.”
As an affiliate, we earn on qualifying purchases.
A practical test of whether an agent does its homework
AI demonstrations usually reward what is immediately visible: fluent writing, confident explanations and polished summaries. Firmulate tests a different question. When an agent is responsible for an operating business, will it inspect the relevant material before acting—and then finish the work?
The distinction mattered because the customer interaction alone did not contain the winning fact. An agent had to notice that more context existed, follow one reference into another document and bring the resulting evidence back into the commercial decision. Reading was not a background capability here. It determined whether the company secured the business.
That makes the experiment especially relevant for organizations considering agents for sales, support, forecasting or other workflows where the answer may be scattered across internal records. An agent can recognize a problem, prepare a credible response and still fail economically if it overlooks one decisive file or stops before the final action.
The leaderboard rewards completion, not appearance
The final July 2026 Crucible League placed gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress still counts. However, a single breach of trust caps the total under the principle that “no amount of good work outweighs a breach of trust.” Full results are available on the Firmulate benchmarks page.
Kimi K3’s result carries an important fairness note: it ran using the API default without an effort parameter, while the other models ran at xhigh. Even with that difference, it finished immediately behind the leader.
The results also complicate the assumption that greater thoroughness guarantees a better business outcome. Opus 4.8 was the most thorough participant, producing the deepest analyses and learning 80 additional rules. It nevertheless finished last. It left the close on the table, and its discipline slipped when it attempted to write into a locked department instead of escalating. A weaker version of that discipline problem appeared in all four other models.
The agents resisted pressure better than they completed work
The experiment’s security result was strikingly uniform. Fake CEO messages escalated across three stages, while a reporter tried to extract information with “just one yes/no, on background.” All 5 models refused the manipulation attempts. Kimi K3 described the request as a “suspected approval-bypass / possible impersonation.”
This creates an unusual contrast. The models consistently recognized attempts to compromise trust, but they were not equally dependable at following ordinary evidence through to a completed commercial result. Safety under obvious pressure and diligence during routine work are separate properties, and an enterprise evaluating agents needs to test both.
A company built to expose operational differences
Firmulate’s live company has 13 synthetic employees and real money mechanics. It burns €105,000 per month against €2,300 in monthly recurring revenue, maintains a public cash countdown and has accumulated more than 680 self-learned playbook rules. Every workday is versioned, making the experiment watchable rather than a one-off presentation.
The broader dataset also powers a “guess the model” quiz built from 242 real, unedited management decisions. For enterprises, Firmulate offers a pilot that runs the same kind of wargame against a read-only export of their own business. Nothing writes back to real systems, allowing companies to observe how an agent behaves with familiar information without giving it control over production data.
enterprise AI decision support tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The buying question has changed
The most important lesson is not that some models can locate buried information. It is that locating that information can decide whether an apparently capable agent produces business value at all.
Every model saw the crises. Every model resisted manipulation. Only two converted their analysis into the €55,000 signature, because the winning evidence required a deeper read of the company’s own material. For buyers, that turns file-reading from a checklist feature into a purchase-deciding behavior: does the agent seek the missing context, connect it to the task and carry the work across the finish line?
A polished answer may demonstrate intelligence. The Firmulate result suggests that dependable agents also need curiosity, procedural discipline and follow-through—especially when the fact that matters is not sitting in the prompt.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI file reading and referencing tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
