EVOLVING HUMAN-LIKE MICROMANAGEMENT IN STARCRAFT II

WITH NEURO-EVOLUTION AND REINFORCEMENT LEARNING

Project Overview

Artificial Intelligence

Video Games

Video games have always been a popular proving ground for artificial intelligence techniques. Traditional AI agents have relied on scripted, rule-based approaches that have several flaws such as the inability to handle massive state spaces and the specificity of logic to a game that can be exploited. Recent breakthroughs in this domain have come through the application of novel approaches in machine learning such as reinforcement learning and neuro-evolution. After preliminary testing in the Arcade and Real-Time Strategy genre, we chose to further explore the applications of machine learning in StarCraft II which is claimed to be the next “grand challenge” for AI research. In our project, we implement neuro-evolution using NEAT and reinforcement learning using Sarsa(λ) on micromanagement scenarios in StarCraft II involving the small-scale precise control of combat units. Using our developed training framework for applying NEAT to StarCraft II, we evolved neuroevolutionary agents that learned to demonstrate precise hit-and-run strategies to beat the in-game AI in ranged vs melee matchups. Our reinforcement learning agents using Sarsa(λ) learned to be successful in more complex micromanagement scenarios involving enemy engagement selection and timing. Our results serve as a proof-of-concept of the benefits and potential of the applications of these techniques in video games and represent meaningful contributions to the wider video gaming and artificial intelligence communities.

Project Objective

• Explore various classes of games and gaming environments. Given the diversity in video games and the time constraint, we decided to choose StarCraft II as our chosen game environment for further study.
• Develop a framework for interaction between our machine learning agents and the StarCraft II game environment.
• Experiment with various traditional forms of machine learning approaches to identify the most promising approaches
• Focus on the most promising approach and fine-tune it to increase its performance to a reasonable level aiming to develop agents that can outperform conventional scripted AI and possibly beat humans without any prior knowledge of the rules of the game.
• Report on the degree of success of our various approaches after extensive testing and experimentation

Project Deliverables


Machine Learning Implementation

Applied Neuroevolution and Reinforcement Learning to StarCraft II micromanagement

Frameworks

Frameworks built for machine learning agents to interact with SC2

Machine Learning Agents

Generic training agents with helper functions to speed up development

Trained Agents

Trained agents that successfully learnt strategies for micro-scenarios

SC2 Maps

Extensive set of StarCraft II Maps used for testing and evaluating the machine learning agents

Results

Results from the extensive testing on various StarCraft II scenarios

Project Schedule

2018
September, 2018

September 1 – 30

> Project research. Review of existing work.

> Phase One Deliverable - Detailed Project Plan and Project Web Page.
October, 2018

October 1 – 14

> Research on various classes of games and approaches to apply machine learning.

> Choose Game.

October 15 - 31

> Setup environment with chosen game.
November, 2018

November 1 – 30

> In-depth research on reinforcement learning and evolutionary approaches.
December, 2018

December 1 – 31

> Implement researched approaches onto chosen game.
2019
January, 2019

January

| Evaluation of implementation.

| Iterate and improve on results.

January 7 - 11

> First Presentation.

January 20

> Phase Two Deliverable - Preliminary implementation & Detailed interim report
February, 2019

February 1 – 28

> Apply neuro-evolution to adversarial scenarios.

> Testing and experimentation.
March, 2019

March 1 – 31

> Apply reinforcement learning to adversarial scenarios.

> Testing and experimentation.
April, 2019

April

| Final Cleanup.

| Report Writing.

April 14, 2019

> Phase Three Deliverable - Finalized Tested Implementation, Development Report & Testing and Evaluation report.

April 15 - 19, 2019

> Final Presentation.

April 29, 2019

> Project Exhibition.

Our Team

Dr. Dirk Schnieders

Zain Ul Abidin

Fawad Masood Desmukh

Results

Before Training - Banshee & Marine V Corruptor & Ultralisk

After Training - Banshee & Marine V Corruptor & Ultralisk