EVOLVING HUMAN-LIKE MICROMANAGEMENT IN STARCRAFT II
WITH NEURO-EVOLUTION AND REINFORCEMENT LEARNING
Video games have always been a popular proving ground for artificial intelligence techniques.
Traditional AI agents have relied on scripted, rule-based approaches that have several flaws
such as the inability to handle massive state spaces and the specificity of logic to a game that can
be exploited. Recent breakthroughs in this domain have come through the application of novel
approaches in machine learning such as reinforcement learning and neuro-evolution. After
preliminary testing in the Arcade and Real-Time Strategy genre, we chose to further explore the
applications of machine learning in StarCraft II which is claimed to be the next “grand
challenge” for AI research.
In our project, we implement neuro-evolution using NEAT and reinforcement learning using
Sarsa(λ) on micromanagement scenarios in StarCraft II involving the small-scale precise control
of combat units. Using our developed training framework for applying NEAT to StarCraft II, we
evolved neuroevolutionary agents that learned to demonstrate precise hit-and-run strategies to
beat the in-game AI in ranged vs melee matchups. Our reinforcement learning agents using
Sarsa(λ) learned to be successful in more complex micromanagement scenarios involving enemy
engagement selection and timing. Our results serve as a proof-of-concept of the benefits and
potential of the applications of these techniques in video games and represent meaningful
contributions to the wider video gaming and artificial intelligence communities.
• Explore various classes of games and gaming environments. Given the diversity in video games and the time constraint, we decided to choose StarCraft II as our chosen game environment for further study. • Develop a framework for interaction between our machine learning agents and the StarCraft II game environment. • Experiment with various traditional forms of machine learning approaches to identify the most promising approaches • Focus on the most promising approach and fine-tune it to increase its performance to a reasonable level aiming to develop agents that can outperform conventional scripted AI and possibly beat humans without any prior knowledge of the rules of the game. • Report on the degree of success of our various approaches after extensive testing and experimentation
Machine Learning Implementation
Applied Neuroevolution and Reinforcement Learning to StarCraft II micromanagement
Frameworks built for machine learning agents to interact with SC2
Machine Learning Agents
Generic training agents with helper functions to speed up development
Trained agents that successfully learnt strategies for micro-scenarios
Extensive set of StarCraft II Maps used for testing and evaluating the machine learning agents
Results from the extensive testing on various StarCraft II scenarios
September 1 – 30
> Project research. Review of existing work.
> Phase One Deliverable - Detailed Project Plan and Project Web Page.
October 1 – 14
> Research on various classes of games and approaches to apply machine learning.
> Choose Game.
October 15 - 31
> Setup environment with chosen game.
November 1 – 30
> In-depth research on reinforcement learning and evolutionary approaches.