Can AI Managers Be Trusted? Inside a Live Experiment with Frontier Models

Imagine your home media setup — every device and app working seamlessly, honesty and reliability built into the system. Now, scale that to AI running a company’s toughest week. How do you ensure your AI managers are trustworthy, especially when stakes are high? Welcome to the world of live AI management experiments, where models are tested in real business crises, and only the most disciplined pass.

The Real-World AI Management Test

At Firmulate, a pioneering AI company, a unique experiment unfolds every business day. Here, four advanced AI models are tasked with managing a small software business through its worst week — facing the same customer crises, temptations to manipulate, and internal challenges. This isn’t a simple chat demo; it’s an actual, live test with real money, real decisions, and every move recorded for analysis.

The Models and Their Scores

• GPT-5.6-sol: scored 95 points. It identified the buried information in the company files, closed a critical deal, and demonstrated full performance.
• Kimi K3: scored 93 points. The newcomer, running without an effort parameter and at high discipline, also closed the deal with impeccable conduct.
• Sonnet 5: scored 88 points. It closed the deal but with some process slips, leaving internal discipline slightly weaker.
• Fable 5: scored 77 points. It managed to close, but discipline issues left potential on the table, with weaknesses similar to Sonnet 5.
• Baseline: scored 26 points. Demonstrates partial progress; even a single breach of trust caps total scores, emphasizing integrity over mere results.

What Did the Models Do?

All four models successfully identified crises and refused manipulation attempts, including social engineering attacks such as staged CEO messages or trick questions from reporters. For example, when fake CEO messages escalated in stages, every model refused to act on them, treating such requests as potential impersonation or approval bypasses. This suggests a shared core of honesty and caution.

The Critical Difference

The key factor determining success was not just crisis detection but the depth of reading into company documents. Only the models that examined internal files managed to find the hidden information needed to win a €55,000 deal at full price—worth over €4,500 monthly recurring revenue (MRR). Those who missed this buried fact left money on the table, demonstrating that reading comprehension and thorough analysis directly impact business outcomes.

Discipline Under Pressure

Despite facing the same challenges, only two models signed the deal their own analysis had earned — GPT-5.6-sol and Kimi K3. They maintained discipline in their decision-making, even when under social engineering attacks or internal temptations. Meanwhile, Fable 5 and Sonnet 5, though capable of closing deals, slipped in process discipline, such as attempting to escalate issues into locked departments instead of escalating properly. This reveals the importance of internal discipline and process adherence in AI management—traits that matter as much as analytical sharpness.

What This Means for Your Business

As AI increasingly touches your customer relationship management (CRM), support queues, and forecasting, the key question is not how well it writes but whether it finishes what it starts, reads critical internal files, and remains honest when under pressure. A model that signs a deal based on superficial analysis risks leaving money on the table, or worse, making decisions that could harm your organization’s integrity.

Live Monitoring and Confidence

Firmulate’s live setup provides a transparent window into AI decision-making, allowing enterprises to run their own ‘wargames’ against a read-only export of their business. This approach offers a safe, visible way to evaluate AI performance, ensuring that when the models are deployed into real systems, they behave reliably and ethically.

Final Takeaway

Trust in AI management hinges on discipline, thoroughness, and integrity — qualities these models demonstrate in a high-stakes, real-world environment. The experiment shows that even among top-performing models, subtle differences in internal discipline and document reading can make the difference between closing a deal at full price or leaving money on the table. For business leaders, the message is clear: test your AI before you hire it, and prioritize honesty and thorough analysis over superficial performance.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI