In a world where AI tools are increasingly integrated into our daily work, the real question isn’t just about how well these models can generate text or answer questions. It’s whether they can actually get the job done — especially when it matters most. For educators, researchers, and anyone involved in decision-making, understanding AI’s true capabilities can be a game-changer.
Testing AI’s Business Skills in a Live Environment
Recently, a pioneering experiment by Firmulate took four advanced AI models and tasked them with running a small software company through its most challenging week. The goal? To see if these models could not only identify crises but also follow through with decisive action — including closing a significant deal worth €55,000. The results shed light on a critical gap that chat-based demos often hide.
The Experiment Setup
All four models faced identical scenarios: the same customers, the same crises, and the same temptations to manipulate or cut corners. Every decision was versioned and auditable, meaning the process was transparent and could be scrutinized. The models were evaluated on two key metrics: whether they identified every crisis and whether they signed the deal they had analyzed and recommended.
Key Findings: Recognition vs. Action
Interestingly, all models excelled at recognizing crises and refused manipulation attempts, such as fake CEO messages or reporter tricks. This demonstrated strong situational awareness and integrity. However, a crucial difference emerged when it came to executing the final step: closing the deal. Only two models, gpt-5.6-sol and Kimi K3, followed through and signed the contract, earning their own analysis.
The Hidden Weakness: Reading Critical Files
Digging deeper, the decisive advantage for the successful models was their ability to read and understand complex internal documents—information buried two references deep in the company’s files. Those that read the files thoroughly managed to win the deal at full price, adding over €4,583 in monthly recurring revenue (MRR). Conversely, the models that skipped this step left the opportunity on the table, missing out on significant revenue.
What This Means for Business and Education
This experiment underscores a vital insight for anyone using AI: surface-level chat demos can be deceiving. Passing a chat-based test doesn’t guarantee that an AI will follow through with real-world tasks requiring diligence, discipline, and integrity. For educational institutions and researchers, this highlights the importance of testing AI in environments that mirror actual decision-making processes, not just conversational prowess.
Business Intelligence in the Age of AI: Modern Data Warehousing, Analytics and AI-driven Decision-Making
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Trust and Discipline Under Pressure
Beyond technical skills, the models’ responses to social engineering attempts demonstrated their integrity. All five models refused fake CEO messages escalating through multiple stages and a reporter’s subtle request for background approval. Kimi K3 justified this by treating such requests as potential impersonation or approval-bypass attempts.
Real-World Implications
The live experiment was conducted within a simulated company environment using 13 synthetic employees and real-money mechanics, burning €105k monthly against a revenue of just €2.3k. The company’s operations are transparent and publicly visible at firmulate.com/live. This setup exemplifies how AI-driven decision-making can be tested before deployment in actual business or educational settings.
Why This Matters for Education and Science
In academic and scientific contexts, the ability of AI to genuinely complete complex tasks — especially those involving understanding nuanced documents and making disciplined decisions — is crucial. The experiment reveals that AI’s real value lies not in superficial chat interactions but in its capacity to act decisively and ethically under pressure.
Final Takeaway: Look Beyond the Surface
While chat demos are useful for initial impressions, they hide the harder, more important question: can the AI finish what it starts? Can it read critical information, stay honest, and execute decisions that impact real money and real outcomes? The answer, as this experiment shows, is only apparent when tested in the wild.
For educators, researchers, and decision-makers, it’s a call to evaluate AI models in environments that mirror real-world complexity and responsibility. Because when it counts, only the models that can truly follow through — reading deeply, resisting temptation, and signing their own work — will make the difference.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
