What’s the best way to test how smart an AI is? Some researchers think the answer lies in Super Mario Bros.—yes, the same game we all played growing up!
Not complex math problems, medical research, or even self-driving cars, but video games to test AI.
Why? Because video games provide a unique and challenging environment that requires real-time decision-making, strategy, and adaptability—just like real-world tasks.
Why Super Mario Bros.?
AI has been tested on games for years—think Chess, Go, and even Pokémon. But now, experts believe Super Mario Bros. offers a better challenge. Unlike board games, Mario requires fast decision-making, adapting to new obstacles, and learning from mistakes—just like in real life!
What Makes Mario a Good AI Test?
✔ It’s unpredictable – AI must react to random enemies, tricky jumps, and unexpected obstacles.
✔ It’s a side-scroller – Unlike Pokémon battles, Mario doesn’t pause. AI needs to keep up in real time.
✔ It rewards problem-solving – Just like us, AI has to figure out the best way to beat each level.
How Does AI Play Super Mario?
Researchers at the Hao AI Lab at the University of California San Diego recently used Super Mario Bros. as an AI benchmark. They designed an experimental framework called GamingAgent, which allows AI models to interact with the game.
Here’s how it works:
The AI receives game instructions and screenshots—just like a human player seeing the screen.
It analyzes the situation—identifying obstacles, enemies, and the best moves to take.
It generates Python code that controls Mario’s movements, telling him when to run, jump, or dodge.
Surprisingly, reasoning-based models like OpenAI’s GPT-4o struggled more than expected, possibly because they overcomplicated their decision-making process. Meanwhile, Anthropic’s Claude 3.7 performed the best, navigating the game with efficient decision-making and faster responses.
But… Is This a Good Idea?
Not everyone agrees that testing AI with Mario is useful. Some argue it’s just a game and doesn’t prove AI can solve real-world problems. Others believe video games are a fun and effective way to push AI’s limits in a safe, controlled environment.
Final Thoughts
Whether or not Mario is the future of AI testing, one thing is clear—AI is getting better at games, and who knows? Maybe one day, AI will speedrun Mario better than any human!
Would you trust an AI to beat your favorite game? 🎮🤖
