Hi HN,

Can current AI systems actually run a business?

There’s a growing belief that LLM agents can already manage entire teams, replace the entire software stack or even act as an AI CEO.

So we built a controlled, measurable environment to evaluate this premise.

Why did we build this benchmark?

A modern enterprise operates in a dynamic environment with high uncertainty and incomplete information. The CEO has to deal with delayed consequences, staffing/resource tradeoffs and death by a thousand cuts of failure modes.

If we ever want AI systems that can meaningfully make operational or strategic decisions, say an AI CEO, then they must be able to handle these dynamics.

So we made one.

What did we build?

Mini Amusement Parks (MAPs) is a RollerCoaster Tycoon style business simulator with: - Stochastic events - Incomplete information - Staffing, restocking, maintenance - Long horizon planning - Compounding operational failures - Resource constraints - Spatial layout affecting outcomes

You can play it & make it to the leaderboard here: https://maps.skyfall.ai/play (it’s fun)

It looks like a simple game. But underneath, it’s a benchmark designed to answer one question:

Can an agent operate a business coherently over time?

What we tested

We evaluated: - Humans (internal and external testers) - Multiple GPT-5 agents - Variants with additional tools, documents, practice mode, planning scaffolds, etc.

We intentionally stacked in favour of the models - full documentation, step by step action interfaces, sandbox exploration mode, extra observations, multiple prompting strategies, etc.

What happened?

Humans destroyed the agents by FAR. Even the strongest model, with documentation, tool use, and sandbox “practice”, reached <10% of human performance. The failure modes were consistent: - chasing flashy upgrades instead of profitable ones - ignoring maintenance, staffing, restocking - overreacting to noise - zero long-term plan - sandbox training often made things worse

It became clear: LLMs can use tools, but they cannot run systems. They break when randomness, time, and spatial constraints matter.

Why does this matter?

There’s a growing narrative that: - LLMs will run entire companies - LLMs will take over the jobs of CEOs - LLMs can be autonomous agents - LLMs can manage workflows end-to-end

MAPs show the complete opposite.

Operating a business requires: foresight, risk modeling, temporal reasoning, causal understanding, prioritization under uncertainty, adaptive planning. These are the basics of what a functional and real AI CEO would need and this is exactly where the current models break.

If an LLM can’t run a toy business, how can you trust it with a real business?

This benchmark is our first step toward understanding what an AI system would actually need in order to exhibit enterprise level decision making and the basics of the AI CEO. AI CEO is not a chatbot, not chain of thought, definitely not an agent wrapper but a true demonstration of operational intelligence.

We’re sharing this because: - we want the community to try to beat the models - we want criticism of the benchmark - most importantly, we want an honest discussion about what “AI CEO” is and should do (surely it’s not LLMs)

If you want to try beating the agents (it’s fun!): https://maps.skyfall.ai/play

If you want the read more about it, you can do so here: https://skyfall.ai/blog/building-the-foundations-of-an-ai-ce...

Check our the launch video here: https://www.youtube.com/watch?v=7oqVAWw5Ii8

Happy to answer questions in the thread.

user:	sohilbhatia
created:	January 11, 2024
karma:	1