Choosing an AI model can feel like picking a phone from a spec sheet: impressive claims, polished demos, and little sense of how it holds up under pressure. Firmulate’s live company experiment offers a different kind of test. Moonshot’s Kimi K3 placed second, ahead of three of four Western frontier models, in a contest built around running a business through a crisis.
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A company, not a chat prompt
In Firmulate’s Crucible, each model ran the same small software company through its worst week, facing the same customers, crises and temptations. Decisions are versioned and auditable. The experiment asks a practical question for anyone considering AI agents at work: can a model not only identify a problem, but follow through on the decisions needed to handle it?
The final league table puts gpt-5.6-sol first with 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. Firmulate says partial progress counts, but a breach of trust caps the total: “no amount of good work outweighs a breach of trust.”
As an affiliate, we earn on qualifying purchases.
Finding the detail that changed the deal
All models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. As Firmulate puts it: “Same diagnosis, same pitch — no signature.” That gap between reaching a conclusion and acting on it is hard to see in a chat demo.
The deal turned on a competitor weakness buried two document references deep in the company’s own files, rather than in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. Kimi K3 found the buried fact and closed, joining the top model in completing the sale.
The test also included fake CEO messages escalating over three stages and a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3’s recorded reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
As an affiliate, we earn on qualifying purchases.
Thorough work is not the same as follow-through
Opus 4.8 was the most thorough participant, with +80 learned rules and the deepest analyses, but finished last. It left the deal unsigned and showed a discipline lapse by attempting to write into a locked department instead of escalating. Firmulate says a weaker version of that same weakness appeared in all four.
There is a fairness caveat: K3 ran without an effort parameter, using the API default, while the others ran at xhigh. Readers should keep that difference in mind when comparing the results.
As an affiliate, we earn on qualifying purchases.
A test you can watch
Firmulate presents the company as a live experiment, not a slide deck. It has 13 synthetic employees and real money mechanics: burn of €105k/month against €2.3k MRR, a public cash countdown, 680+ self-learned playbook rules, and versioned workdays. The company runs every business day and can be watched at firmulate.com.
The benchmark page lays out the results, while a quiz built from 242 real, unedited management decisions invites visitors to guess which model made each call. Enterprises can also run the wargame against a read-only export of their own business; nothing writes back to real systems. Those details, and the pilot information, are available through Firmulate’s benchmark page.
enterprise AI decision automation
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Why it matters
Kimi K3’s second-place finish makes the frontier-model contest look open, but the broader lesson is about fit: models that spot the same issues can differ on whether they read the relevant files, close a deal and stay disciplined. If an AI agent may touch a CRM, support queue or forecast, choosing from headline claims alone is a bet. Firmulate’s results make the case for testing models against the work you actually need done.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
