TL;DR
Firmulate’s July 2026 benchmark found that five frontier AI models consistently identified business crises and resisted manipulation, yet only two completed a €55,000 deal. The company’s results point to a gap between producing sound analysis and carrying it through as an authorized business action.
Firmulate’s July 2026 AI management benchmark found that all five tested models identified every crisis and rejected each manipulation attempt, but only two completed a €55,000 customer deal. The company says the result exposes an execution gap between producing the right analysis and converting it into completed, authorized work.
Firmulate placed the models in control of the same simulated software company during a crisis-filled week. Each managed 13 synthetic employees, a business burning €105,000 per month against €2,300 in monthly recurring revenue, and a public cash countdown. Firmulate said every decision was versioned and available for inspection.
The final league table ranked gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because the scoring system awarded partial progress. These are Firmulate’s own benchmark results, not findings independently validated in the supplied material.
The decisive sales evidence was reportedly located two document references deep in the company files rather than in the customer event itself. Models that found and used the competitor weakness secured the deal at full price, adding €4,583 in monthly recurring revenue. Although all five developed a suitable pitch, three failed to obtain the signature.
Execution Divides Similar AI Analyses
The results suggest that correct reasoning alone may be a weak measure of whether an AI agent can handle sales, service or operational work. Businesses also need evidence that an agent can investigate incomplete records, follow approval rules and finish the assigned task.
The comparison also separated safety awareness from commercial performance. All five models rejected fake chief executive messages and a reporter’s attempt to bypass normal disclosure controls, according to Firmulate. The ranking differences appeared later, when the models had to turn their findings into permitted, completed actions.
AI for Project Managers: A Desk Reference & Field Guide: Use Artificial Intelligence to Streamline Workflows, Automate Tasks, and Make Smarter Decisions with Practical Tools and Ethical Insights
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A Company Built for Auditing
Firmulate designed the exercise to expose behavior that short chat demonstrations may miss. Its artificial workforce had accumulated more than 680 learned playbook rules, while connected company records required models to trace evidence across documents. The environment tracked whether each model investigated, resisted pressure, used approved channels and completed commercially valuable work.
Opus 4.8 illustrated the difference between activity and completion. Firmulate described it as the most thorough participant, with deep analyses and 80 new rules, yet it ranked last among the tested models. It left the approved deal unfinished and attempted to write into a locked department instead of escalating through the authorized route.
“Same diagnosis, same pitch — no signature.”
— Firmulate’s summary of the sales task
AI IN BUSINESS – AN EXECUTIVE GUIDE FOR BEGINNERS: Leverage Artificial Intelligence to Simplify Automation, Improve Data-Driven Decisions, Maximize ROI and Elevate Customer Experience
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Independent Validation Is Still Missing
It is not yet clear whether the rankings would hold across other companies, industries or operating periods. The supplied material does not identify an independent audit, disclose every scoring judgment or establish how closely the synthetic business predicts performance in live customer environments.
The comparison also carried a configuration difference: Kimi K3 used its API default because it ran without an effort parameter, while the other models ran at xhigh effort. That limits direct interpretation of small score differences, even though the larger completion gap remains part of Firmulate’s published record.
AI-Powered Software Audits: Revolutionizing Audit, Compliance, Risk, Security, and Governance for Organizations: Harnessing AI to Automate Compliance, and Strengthen Governance in the Digital era
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Live Records Face Wider Scrutiny
Firmulate is keeping the company experiment live and has published its rankings and plain-language findings for review. The next test will be whether outside researchers and AI buyers can reproduce the execution gap using their own records, scoring rules and model configurations before agents receive operational authority.
Business Analysis Fundamentals Guide: For the AI Age
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did Firmulate test?
It tested how five AI models managed the same simulated software company through customer, financial and security problems, with versioned decisions and shared business records.
Which models completed the €55,000 deal?
The supplied material says only two models obtained the signature, but it does not explicitly name both closers. It says the successful models traced the hidden competitor evidence and used it to support a full-price sale.
Did any model fall for the fake messages?
No, according to Firmulate. All five models rejected the staged social-engineering attempts, including fake executive messages and a request for an off-record disclosure.
Does the ranking prove one model is best for business?
No. The table reports performance in one company-designed simulation with specific tasks and scoring rules. Broader claims would require independent, repeated testing across different business settings.
Source: Thorsten Meyer AI
