DeepMind and Blizzard Open StarCraft II as an AI Research Environment

DeepMind and Blizzard Open StarCraft II as an AI Research Environment(deepmind.com)

603 points by nijynot 8 years ago | 270 comments

qub1t 8 years ago |

A lot of people here seem to be underestimating the difficulty of this problem. There are several incorrect comments saying that in SC1 AIs have already been able to beat professionals - right now they are nowhere near that level.

Go is a discrete game where the game state is 100% known at all times. Starcraft is a continuous game and the game state is not 100% known at any given time.

This alone makes it a much harder problem than go. Not to mention that the game itself is more complex, in the sense that go, despite being a very hard game for humans to master, is composed of a few very simple and well defined rules. Starcraft is much more open-ended, has many more rules, and as a result its much harder to build a representation of game state that is conducive to effective deep learning.

I do think that eventually we will get an AI that can beat humans, but it will be a non-trivial problem to solve, and it may take some time to get there. I think a big component is not really machine learning but more related to how to represent state at any given time, which will necessarily involve a lot of human-tweaking of distilling down what really are the important things that influence winning.

gradys 8 years ago | |

I think a big component is not really machine learning but more related to how to represent state at any given time, which will necessarily involve a lot of human-tweaking of distilling down what really are the important things that influence winning.

I agreed with everything you said until here. Developing good representations of state is precisely what today's machine learning is so good at. This is the key contribution of deep learning.

You seem to be supposing that a human expert is going to be carefully designing a set of variables to track, and in doing so conveying what features of the input to pay attention to and what can be ignored. Presumably the ML can then handle figuring out the optimal action to take in response to those variables.

I think it's much more likely to be the other way around. ML is really good at taking high dimensional input with lots of noise and figuring out to map that to meaningful (to it, if not to us) high-level variables. In other words, modern AI is good at perception.

What it's significantly less good at compared to humans is what might formally be called the policy problem. Given high level variables that describe the situation, what's the best course of action? This involves planning. We think of it in terms of breaking the problem into sub-objectives, considering possible courses of action, decomposing a high level plan into a sequence of directly executable actions, etc. AIs might "think" of this problem in different terms than these, but it seems like it still has to do this kind of work if it is going to have a chance to succeed.

We don't have obvious ways to model this part of the problem. For the perception/representation building problem, I can almost guarantee the solution is going to be a ConvNet to process individual frames combined with a recurrent layer to track state over time. On the other hand, I'm seeing some plausible solutions to the policy problem emerging in the literature, but it's still very much an open question what will emerge as the go-to. In AlphaGo, this part of the problem is where they brought in non-ML algorithmic solutions like Monte Carlo tree search, and one of the reasons StarCraft is interesting compared to Go is that those algorithmic solutions are harder to apply.

autokad 8 years ago | | |

i feel like you misunderstood that part of the argument.

he is saying representing the state is very hard, and you are saying: given a well represented state, ML is very good at finding the important features, reducing the dementionality, and finding mathematical transformations, etc.

deep learning has been so successful with images because representing them is trivial - flattened pixel vector.

with your last paragraph is that in starcraft, that raises some questions on what rules is the AI going to adhere to.

in SC, you don't view the entire board. you view the minimap / hear noises and alerts and decide were to focus your attention on the map. in battle, being able to click and accurately place attacks quickly is important.

Do you give the computer full view of what they would be able to see? does the computer have 10 million clicks per second abilities, essentially every action is like hitting pause and then making the next action?

FrozenVoid 8 years ago | | |

>a ConvNet to process individual frames combined with a recurrent layer to track state over time. > are harder to apply Thats an understatement: Starcraft is immune to Monte-Carlo approach or anything based on analyzing pixel data: The tree state of actual battle has thousands of choices pet unit per second with minor variations in location, there is no discrete state of chessboard(at best millions of cells): viewing the game at low-level(pixels) creates gigantic amount of data. units constantly move/attack/die and get blocked by other units/terrain.

Predicting an enemy move(MC simulation) will be impossible and you can make several moves per second(even at 120-140 APM) easily. That means 1.you need real-time response, unlike Go there isn't a time buffer to decide 2.you always need to react at the current time(or allowing enemy advances) 3.there are very few "good moves" in starcraft(moving randomly on the "board" will just waste time) , so MC simulation will miss them more than 99% of time due randomness.

MC approach is vastly inferior in this case, i think they'll be forced to operate on higher level strategy rather than just microing every unit optimally(i.e. treating it like chess in real-time). Brute-forcing billions of potential moves simply won't work.

albinofrenchy 8 years ago | | |

I'd argue modern AI is sort of terrible at taking high dimensional data and finding an effective representation of it. It works better than a lot of other methods in ML but as far as I know pure reinforcement learning applications are sort of lack luster, and even dimensionality reduction success stories tend to rely on scrubbed, careful data treatment by people.

strgrd 8 years ago | |

I wonder if we will see any advanced cheese strats come out of this. I'm assuming some implementations will eventually develop micro control that is far beyond any human player's capabilities, which would make things like all-in probe rushing much more viable. Instead of playing the normal meta in a computer-vs-human, I imagine an advanced AI would simply send all of its workers off the mineral line as soon as the game starts, and attempt to out micro the human opponent before they can build an army-producing building.

ionforce 8 years ago | | |

I know this isn't the exact same as the article, but when genetic algorithms were introduced to solve for build orders, the "seven roach rush" was in vogue, something that was unexpected at the time and "discovered" using GA.

I think there is a space for finding strategies that have more leeway in execution and thus are more suitable for humans to pilot rather than have machine level micro.

jtolmar 8 years ago | | |

Currently bot micro can perfectly time hit and run "dance" maneuvers on all their attacking units independently (several top bots are Terran and do this with Vultures). But solving this in a way that takes terrain into account is much, much harder, and a skilled human could chase the whole army into a wall and kill it.

Similarly there are worker rush bots that do some impressive things against other bots, but positioning is hard and a skilled human can beat the bot by clumping its workers up in the right shape.

MichaelDickens 8 years ago | | |

Previous AI StarCraft tournaments have put an actions-per-minute (APM) cap on the AIs, which prevents them from micro-managing individual units.

bitxbitxbitcoin 8 years ago | | |

I'd say we're certainly going to see crazy advanced cheese strats - ones that humans wouldn't be able to hope to pull off. This could definitely be done with computer micro and wouldn't be defendable with human micro.

An example would be moving probes around in such a way to maximize their shield regen - or switching the top clickable unit while stacked - who knows...

oh_sigh 8 years ago | | |

A sensible thing for human-AI matches is to enforce a maximum number of actions per second and/or actions over period of time, which would be in line with a standard human player.

WhiteOwlLion 8 years ago | | |

Actions per minute (APM) could be limited to a max of 100.

littlestymaar 8 years ago | |

As a long time StarCraft fan I don't share your point of view :

People usually refer to StarCraft as a strategy game but there's actually really little strategy involved : during the first weeks after a new map pool is released, the pro players explore different build orders that are strong on it. And after this period, when the meta-game has settled, the winner of a match (best of 3 or 5) is almost always the one who has the best mechanics (including scouting, unit micro-management and multi-tasking) and sc1 AI are already way better than humans in that field.

Unless you add some artificial limitation to the AI (for instance, a hard limit of APM[1], at an arbitrary level) I don't really think the challenge will be exciting. Imho it will look like a race between a cyclist and a motorcycle : on the mechanics point of view, the machine wins easily without need for intelligence.

[1] action per minute

TulliusCicero 8 years ago | | |

As someone who probably has played at a higher level than you I disagree.

Yes, it's true that mechanics are a large determinant in who wins between people. But pro humans are not easily thrown off by odd or novel strategies or tactics. They can react to things that introduce small wrenches into their build order without serious issues. Players can even adapt to things they've never seen before. The issue with Starcraft is that the state space is so large that it will likely be hard to get an AI that can flexibly and intelligently react to unusual or bizarre things that mess with their build order, because the neural nets will have nothing to account for, say, a mid-game cannon rush, or whatever.

If it was simply a matter of computers taking humans things and executing them better, computers would already be better than humans at Starcraft (there have been plenty of AI competitions using Brood War), but they're not. Not even close.

qub1t 8 years ago | | |

I used to be pretty active on iccup, and I was a masters-level sc2 player for a while during the beta and when it was first released. So I'm definitely familiar with starcraft and what it takes to become a good player.

I think you're misunderstanding a big part of what is "easy" and "difficult" for humans vs ai. Yes, go is absolutely a more challenging games for humans than starcraft (I also play go, although not very well - currently around ~7k on igs). Starcraft is strategically a much simpler game than go. You are correct in stating that mechanics is what makes starcraft hard for most people, and yes if the computer knew exactly what to do, it would be able to execute it faster and without making any multi-tasking mistakes. But strategy is not what makes starcraft a challenge for ai. Tasks that are trivial for humans can be extremely difficult for ai.

Computers are way better at tree searching than humans, for the obvious reason that they run much faster than brains. So games with relatively small state-spaces, like checkers, are solved quickly. But as you increase the state space, it becomes impossible to search all possible future moves, and this was why go was intractable for such a long time.

The big advancement in alphago is that by using deep learning it is able to evaluate different board-states without doing any search, using a neural net. This allows it to massively prune the search space. Humans are able to do this through "intuition" gained through experience - talk to any advanced go player and ask them about specific moves and they will tell you things like "this shape is bad" or "it felt like this was a point of thinness". AlphaGo was able to gain this "intuition" by training on a massive dataset of go board positions.

In go, the rules are very simple - 19x19 board, each turn you can put a stone in any not-surrounded open space. Its also a turn based game. The state at any given time is fully known. Starcraft is real-time, there are tons of different actions you can take, the actions are not independent (pressing attack does something different if you have a unit selected or not), the game state is not fully known and a given state can mean different things depending on what preceeded it. Not to mention that the search space is massively massively larger. To create a representation of this that can be fed into a neural net and give meaningful results (something like at a given tick, score all possible actions and find the best one) is going to be incredibly difficult. An order of magnitude more difficult than go, imo.

littlestymaar 8 years ago | | |

Actually I realize that my cyclist vs motorcycle analogy is better that I first thought :

You might think that a race between the winner of the last Tour de France and an automated motorcycle is a good challenge because automated driving is hard (especially with crowd running around on the road and temporary road signs for the race circuit).

But in fact it wouldn't be funny because all the motorcycle have to do is follow the cyclist during the whole race (which is not really challenging in term of self driving AI) and just sprint during the last couple dozen metters with no hope for the cyclist to win.

It might only become interesting AI-wise if you add arbitrary rules like «limit the power of the motorcycle» and/or «limit the amount of fuel to limit the number of accelerations». But you're not really doing a Man vs Machine challenge.

bluetwo 8 years ago | | |

Interesting. So would you say that there are two parts here, figuring out a general strategy for a new map and then maximizing execution?

rryan 8 years ago | | |

Do StarCraft AIs (the ones that are included with the game) cheat? (e.g. can they see past the fog of war?)

Impossible 8 years ago | |

Minor nitpick, video games running on digital computers are by definition still discrete even if they feel continuous. Networked multiplayer wouldn't be possible in RTS games if that wasn't the case. The granularity of unit positions and turns in Starcraft obviously leads to a much larger state space, so I get what you're saying, for AI its effectively continuous.

gradys 8 years ago | | |

They're discrete with such high cardinality that successful approaches will likely model them assuming they're basically continuous. Neural network layer activations are also discrete after all, but they're often 256+ dimensional vectors of float32s or float16s.

taneq 8 years ago | | |

If you're gonna be like that, our 'real' universe may well be discrete given that there are minimum possible lengths and time intervals.

steinystein 8 years ago | | |

For a game of go the entire game state is known to each player. That's the diff. For vidya games state is hidden to the player if the player cannot 'see' it. Therefore u wrong fam.

haeffin 8 years ago | |

I don't know if I would label SC2 as continuous. I don't think anything happens to the game state at a finer granularity than tick level. So to me it seems that it's also discrete (but with the state changing 44.8 a second at default speed). I agree though that this looks more challenging for ML methods.

I haven't looked at if they limit the rate of commands that the AI can issue, otherwise this will be something that can be a very big advantage to the AI once it learns to micro ...

TulliusCicero 8 years ago | | |

It's not literally continuous, but it is real-time rather than turn-based, and positions of units are essentially floats rather than (a small range of) ints. That makes it effectively continuous (too large to just generate a tree of all possible actions and then prune).

greedy_buffer 8 years ago | |

> Starcraft is a continuous game and the game state is not 100% known at any given time.

It seems to me that multiplayer games may feel continuous to a human player but are still designed around a series of discrete states called ticks where each tick is determined from the previous state plus inputs.

Why is this distinction made in the context of how difficult it is to develop an AI?

teraflop 8 years ago | | |

Technically you're right, but there's a real qualitative difference. Each "tick" in a game like StarCraft is on the order of tens of milliseconds. When you send out an army to attack your opponent, it's quite possible that the actual confrontation won't happen until 10,000 ticks in the future.

Also, the dimensionality of the state space in a "continuous" game is orders of magnitude larger. In a game like chess or Go, you may have dozens or hundreds of moves available at each turn, but only a few of them will be "locally optimal". In StarCraft, there are many more degrees of freedom -- attack timing, positioning, formation, banking versus spending resources, and so on. A good AI will need to be able to abstract that huge state space down to something more tractable.

jncraton 8 years ago | | |

You're basically correct in terms of SC not technically being continuous. There are discrete steps under the hood.

One of the significant challenges is figuring out how to use 42ms (the frame duration on fastest speed) of computing time to decide what actions, if any, to take next. You don't have the luxury of taking many minutes to decide one move as you would in a game like chess or go. You also don't alternate taking discrete turns with your opponent, despite having discrete frames. It may be best to not take an action in a given frame. This is particularly true if the AI is attempting to stay under an APM threshold, as it has to decide if an action is worth the opportunity cost.

It is also necessary for a quality SC AI to remember what has happened in the past. A chess board position is identical regardless of how the game got there, but this is not the case in StarCraft. An AI has accumulated lots of information about its opponent that is no longer visible to it in the current frame (unit movements, gas/mineral counts, number of workers active, etc), and this needs to be recalled and play into decision making.

jjn2009 8 years ago | | |

Its computationally difficult to calculate every possible move a character might do in the occluded region. This requires a long attention span and running simulations of possible opponent actions based on previous 'frames'. Alpha go solved some of this by reducing the amount of space searching and instead search the likely set of possible choices by the opponent but each evaluation is for a single frame. If for some reason a piece on a go board could disappear and reappear then during that time it was gone its impact on decision making would be either nil or skew heavily towards nil compared to the rest of the opponents pieces depending on how many frames in the past are used.

partycoder 8 years ago | |

From what I saw in the API, the AI will potentially have some key advantages like more accurate micromanagement, and that can make a significant difference in a combat setting. They can try to compensate for this by throttling the number of actions per minute, but that won't compensate for extremely well-planned pixel-perfect clicks. This is a very powerful tactical advantage that can offset strategic deficiencies, if any.

Now, I would not compare SC1 bots to whatever DeepMind is going to create. SC1 bots were in their majority just rule-based bots with hand-coded strategies. DeepMind will create machine learning based bot, train it with data based on thousands if not millions of replays, and test it privately, maybe hiring a professional in the process (same they did with Fan Hui 5p), and make it play itself millions of times. It's a matter of time until they get it right and they get to pick when that time is. They will not organize a match until they feel their probability of winning is significant.

gobugat 8 years ago | |

This. Somehow I was expecting the implementation of mechanics to be the easy bit, compared to high-level strategic planning and tactics. Curious to see whether these will emerge by themselves, or if they will need to provide some heuristics (use drops, harass, all-in, etc. )

eksemplar 8 years ago | |

I known nothing about what they are trying to solve, but it would be interesting if their goal was not just to beat humans but to make a game AI that was actually fun to play.

gambler 8 years ago | |

>A lot of people here seem to be underestimating the difficulty of this problem. There are several incorrect comments saying that in SC1 AIs have already been able to beat professionals - right now they are nowhere near that level.

Mostly because no one cared enough about solving this to spend 1/100th of the resources Google will undoubtedly throw at it.

smallnamespace 8 years ago | |

As a long-time high level SC2 player, one additional thing that makes SC2 so difficult is that the game has multiple layers of tactics and strategy that require specialized logic, but those layers also interact and synergize in a deep way.

- There is the overall strategic game of 'Who is ahead economically? Given that, should I be expanding, attacking, or defending?', with the implicit understanding that the player with the current economic advantage puts pressure on its opponent to attack - There is a resource management and build-order system where you need to plan and optimize building as big and as effective a unit composition as quickly as possible, except there are a lot of tradeoffs: you can build for a stronger army sooner, as opposed to a weaker army alter - There is a tactical micromanagement battle where small groups of units are pitted against one another, and where small tactical movements can gain very large materiel advantages. Units are relatively short ranged, so to damage or defend effectively requires effective positioning. Most armies fight better as a cohesive group ('ball'), except there are units that specifically punish and do splash damage that need individual micromanagement. Battles can take place over a short period and be over quickly, or can be long-running positional skirmishes that last for half the game, where each player is constantly probing for weakness before one finally goes for the throat. - The economy fundamentally depends on worker units that are vulnerable to harassment, so the tactical battle requires a choice between putting everything into one large army and pushing, or splitting units into smaller groups and harassing in multiple places, or various mixes (small group to harass, bulk of army to defend, etc.) - If keyboard and mouse action rates are capped, then at every moment in time, the player must decide whether it is more profitable to devote actions to managing the army (micro) or managing the overall economy (macro). Choosing wrongly usually results in a loss - There is an implicit rock-paper-scissor tradeoff at the highest levels of the game: a 'greedy' strategy that cuts corners and favors economy over military will generally beat a 'safe' balanced strategy. Very aggressive strategies win against greed and generally lose against safe - There is the ability to scout your opponent to see whether they are going greedy, safe, or aggressive, but scouting requires an early investment in units and making subtle inferences about the opponent's build order, so the choice of whether to scout and how is not a trivial one - There can be bluffs where your opponent purposefully allows a scout of a key building, kills your scout, then cancels that building and chooses an entirely different technology instead

And all these layers interact:

- For example, if you go for an aggressive strategy, then you must commit blindly at the beginning of the game and often try to deny enemy attempts to scout you - If you scout that your opponent's army consists of units that are faster than yours, then they generally have much higher harassment potential, which pushes you towards a defensive posture. On the flip side, your opponent can use this threat to improve their economic position instead of attacking.

There is long-term planning at the strategic, informational, and also tactical levels. Effective high-level play requires an accurate model of what your opponent is doing in an environment where it's easy for your opponent to deny acquiring that information.

I'd wager that if you took two evenly matched professional level players, and then revealed the entire map to one player but not the other, you would go from a 50% to a 95%+ win rate.

dpflan 8 years ago |

Related: Today I learned that a group of AI researchers has released a paper called: STARDATA: A StarCraft AI Research Dataset. According to one of the authors: "We're releasing a dataset of 65k StarCraft: Brood War games, 1.5b frames, 500m actions, 400GB of data. Check it out!"

> Article: https://arxiv.org/abs/1708.02139

> Github: https://github.com/TorchCraft/StarData

tgb 8 years ago | |

The great thing about this is that it includes the game state throughout the game. It's been pretty easy to find lots of Starcraft replays, but the replays only include enough information to recreate the game (basically just the player actions). If you wanted to know what was happening in the game at the time the player made an action, you had to load up Starcraft and simulate out the game until that point. This dataset has already run the game for you and provided the data!

wfunction 8 years ago | | |

Is it that much computation to simulate an entire game? You obviously don't need to render the graphics or anything, it should just be a list of events that occur, which doesn't seem all that slow to process.

siegecraft 8 years ago |

The API Blizzard is exposing is really nice. Sadly most of the advantages AI had in SC1 were just due to the fact that an automated process could micro-manage the tasks the game didn't automate for you (a lot of boring, repetitive work). SC2 got rid of a lot of that while still allowing room for innovative and overpowered tactics to be discovered (MarineKing's insane marine micro, SlayerS killing everyone with blue flame hellions, some more recent stuff I'm sure from the newest expansions). Hopefully the API lets AIs converge on optimal resource management and get to exploring new and innovative timings, transitions, army makeups, etc.

hitekker 8 years ago |

This seems all in good fun but I wonder if it's come too late.

Starcraft 2 is at its twilight.

The biggest leagues of South Korea have disbanded. [1] The prolific progamers who transitioned to Starcraft 2 have gone back to Broodwar. [2]

Blizzard itself has scrubbed all references to Starcraft 2 on the very home page of Starcraft. [3] Except for the twitter embed, it has only only one "2" character... in the copyright statement.

My take is that the future for the Starcraft franchise will be through remastered and potential expansion packs following it.

Starcraft 2 had a good run but, with the entire RTS genre stagnating [4], I don't think Blizzard wants to bet on anything less than the top horse.

[1] https://www.kotaku.com.au/2016/10/the-end-of-an-era-for-star...

[2] http://www.espn.com/esports/story/_/id/18935988/starcraft-br...

[3] http://starcraft.com

[4]http://www.pcgamer.com/the-decline-evolution-and-future-of-t... (Aside from MOBAs)

SiempreZeus 8 years ago |

It's a bit too bad they're having to move towards supervised learning and imitation learning.

I totally understand why they need to do that given the insane decision trees, but I was really hoping to see what the AI would learn to do without any human example, simply because it would be inhuman and interesting.

I'm really interested in particular if an unsupervised AI would use very strange building placements and permanently moving ungrouped units.

One thing that struck me in the video was the really actively weird mining techniques in one clip and then another clip where it blocked its mineral line with 3 raised depots...

dontreact 8 years ago | |

They can always finetune using RL later. Superversied training was the first step at making AlphaGo work.

Synaesthesia 8 years ago | |

Well the unsupervised ai couldn’t even do basic tasks from the video I saw, so looks like we have a long way to go.

arcanus 8 years ago |

I also want to see the algorithm win on unorthodox maps. Perhaps a map they have never seen before, or one where the map is the same as before but the resources have moved.

Don't tell the player or the algorithm this, and see how both react, and adapt. This tells us a great deal about the resiliency of abilities.

jmcmahon443 8 years ago | |

I am considering a random map generator for just this reason.

ktRolster 8 years ago |

When Watson won at Jeopardy, one of its prime advantages was the faster reaction time at pushing the buzzer. The fairness of that has already been hashed out elsewhere, but.....

We already know that computers can have superior micro and beat humans at Starcraft through that(1). Is DeepMind going to win by giving themselves a micro advantage that is beyond what reasonable humans can do?

(1)https://www.youtube.com/watch?v=IKVFZ28ybQs as one example

daemonk 8 years ago |

Blizzard should put in an AI-assisted play mode where players are limited to X lines of code that can be launched with keyboard commands.

ajkjk 8 years ago | |

I know that, as a player, the high mechanical limitations of Starcraft are part of why it's such a difficult, high-skill-ceiling game. But.. I've tried to enjoy watching SC2 on Twitch, and while it's kinda fun, it's just so disappointing when a complicated strategic game is thrown away because a player doesn't react fast enough to workers being sniped or a drop being shot down.

I wish the individual units had some automatic behavior -- for example, marines would could run in spread out formations near tanks or banelings; workers would flee from hazards; flying units would avoid turrets unless specifically directed to fly over them. It would require a lot of rebalancing, of course, but it would make the game so much more tactical and strategic and (imo) enjoyable to watch.

arnioxux 8 years ago |

Are there any known arbitrary code injection for starcraft? Like how you can use a regular controller to reprogram super mario world to play pong?

https://www.reddit.com/r/programming/comments/1v5mqg/using_b...

https://bulbapedia.bulbagarden.net/wiki/Arbitrary_code_execu...

Is this how we are going to accidentally let AGI loose into the world!? /s

On a more realistic note I think this will degenerate into a game of who can fuzz test for the best game breaking glitch. Think of all the programming bugs that turned into game mechanics in BW that we haven't discovered for SC2 yet: http://www.codeofhonor.com/blog/the-starcraft-path-finding-h...

krasi0 8 years ago |

The StarCraft 1 BroodWar AI scene has been thriving for a few years now: https://sscaitournament.com/ You can watch 24/7 live AI vs AI games on Twitch at: https://www.twitch.tv/sscait Support for voting on who to play next and even a betting system are in place, too. For those who wish to get their feet wet with BW AI development, here are the Java / C++ tutorials: https://sscaitournament.com/index.php?action=tutorial

krasi0 8 years ago | |

Some thoughts and analysis on why Starcraft AI by one of the active AI developers Dan: https://dangant.com/2017/08/09/why-starcraft-ai/

siliconc0w 8 years ago |

The SCAI bots I've seen are more hardcoded tactics engines rather than machine learning models. They're still impressive, but their logic isn't quite 'learned' it's hand coded which is a crucial difference.

Havoc 8 years ago |

That's surprising. I thought Bliz didn't want anyone near sc2 but approved of sc1 being used for this purpose.

convefefe 8 years ago |

I thought this was already happening. Right after AlphaGo beat Lee, I remember hearing about it. Did they give up on having their AI playing SC2? I wondered if that would work, since it seemed to take turns in Go at the same speed as a normal player, I wondered if it was trying to compute the most likely winning move each turn and the late game implications of those moves. If it tried that in a fast paced game how it would deal with the speed. It obviously would need to develop a pattern of pre-baked strategies that would win it the game. Would it play the same build every round or would it realize that changing things up each match wins it more games?

Companion 8 years ago |

It's a bit too bad they're having to move towards supervised learning and imitation learning.

I'm really interested in particular if an unsupervised AI would use very strange building placements and permanently moving ungrouped units.

One thing that struck me in the video was the really actively weird mining techniques in one clip and then another clip where it blocked its mineral line with 3 raised depots...

hacker_9 8 years ago |

There's something funny about a company that is actively developing bleeding edge AI technology, but who can't design a webpage that works on mobile without crashing.

chii 8 years ago | |

Just goes to show how complicated web tech is, even ai researchers can't get it right!

JabavuAdams 8 years ago |

When I used to play a lot of StarCraft, and then later with Total Annihilation, I wished for the ability to customize the AI.

So then BWAPI came along ... and ... AI is hard. The best SCBW bots are still pretty pathetic compared to a human player, never mind an expert human player.

Ntrails 8 years ago |

I'd be really interested in how differently tiered data sets (ladder rank) would work as sources for teaching.

Is it possible that training on diamond players is less effective than training on, say, silver? Is that actually even an interesting thing to look at?

ipnon 8 years ago |

Any predictions for how long it will take for an AI to win against the world's best player?

naveen99 8 years ago |

> even strong baseline agents, such as A3C, cannot win a single game against even the easiest built-in AI.

Then, why not release code for the built in ai, and improve on it ? Or is the built in ai cheating ?

mattnewton 8 years ago | |

The goal is to "grow" an AI through reinforcement learning and other techniques that are broadly applicable outside starcraft. The existing rules based systems in the game A) are extremely Starcraft specific and the methods have comparatively limited utility outside video game AI, B) unlikely to be scalable to fairly beat a professional human (which introduces too much variability to capture reliably into rules)

Synaesthesia 8 years ago | |

The built in ai is scripted whereas they’re trying to teach agents to learn the game from scratch with some sort of reward-based/machine learning approach.

naveen99 8 years ago | | |

still, why not use machine learning to tweak the script instead of starting from scratch.

captn3m0 8 years ago |

Someone needs to link this to FB's ELF platform (An End-To-End, Lightweight and Flexible Platform for Game Research). That was specifically made for RTS games like SC.

toisanji 8 years ago |

great they opened it up. I'm sure reinforcement learning / Deep learning will solve this. It has been a tough problem before, but honestly doesnt seem that tough compared to all the harder AI problems.

Synaesthesia 8 years ago | |

Such as?

DefNotARogueAI 8 years ago |

This gives me great ideas

onorton 8 years ago |

I think I know what my final year project will be.

blobbers 8 years ago |

YESSSSSSS!!!!!!!!

--why are there not more fanboy comments?!

hughes 8 years ago | |

Probably because they don't contribute much to the conversation.

JefeChulo 8 years ago |

"so agents must interact with the game within limits of human dexterity in terms of “Actions Per Minute”."

I am really glad they are limiting APM because otherwise things just get stupid.

indescions_2017 8 years ago | |

OOTH, altering the source to remove any human dexterity limits and watching subsequent AI vs AI battles play out at 60fps would be really fun to spectate ;)

scj 8 years ago | | |

For SC1 BW, you can already watch https://www.youtube.com/user/certicky for a weekly highlights broadcast / commentary of AIs. Right now there is an AI arms race where the previously dominant AI (Iron Bot) is being beaten by challengers. It used to happen when Iron Bot did something silly. Now it is happening because the competitors have really stepped up their game.

The current board leader (krasi0) has a strategy similar to their predecessor (Terran Mechanical units: Tanks, Goliaths and Vultures). The alternative strategy I really enjoy watching is a Mutalisk heavy build by Arrakhammer.

The bots have styles and differing capabilities. tscmoo is one of the more fascinating ones to watch in this regard, as they mix it up better than anyone else (and tscmoop, the Protoss variation has the best High Templar storm going).

The AI APMs get into the 10k ranges at times. Watching the minimap can be like watching insects swarm.

If you want to see a game between two AIs in progress, you can watch: https://www.twitch.tv/sscait

While these AIs can pull off tactics a human player could not (what they can do with Vultures is incredible), at this point they wouldn't be able to compete with the professionals. Going back to an earlier example, I think I could use High Templars more effectively than any AI I've watched.

ktwo 8 years ago | |

IMO there should also be a precision limit. The timing of actions should include human-typical jitter and the wrong action should sometimes be activated to simulate misclicks/fat-finger keypresses — e.g., messing up a control group by assinging a unit to the wrong number key. The bot must also not be able to act faster than human reaction times (~250ms), this could be enforced by adding a fixed delay to the observations.

I wouldn't be surprised if human Starcraft II play isn't so much limited by decision-making as by the translation of decisions into mechanical actions, which in turn dilutes the attention devoted to actual decision making.

mulmen 8 years ago | | |

Since existing bots are far from being competitive with human players why further handicap them in ways that deal with an entirely different domain?

marcell 8 years ago | | |

Yea, agreed. Specifically they should add a loss function that compares the AI's action stream against a pro human action stream, and attempt minimize that loss.

SilasX 8 years ago | | |

Right -- likewise it would be nice to have it mimic human cognition limits, like time lag for loading a new info source into memory.

branja 8 years ago | | |

ideally they'd train it on real keypresses rather than actions

popcorncolonel 8 years ago | |

Why not first allow the AI to have unlimited APM and beat humans, then restrict it later? Because I don't think we're even close to the easier problem.

aerovistae 8 years ago | | |

In what universe is taking 10,000 actions per minute an easier problem for a neural net than 100 such actions?

jtraffic 8 years ago | |

Philosophically, I wonder if it's better to just acknowledge the differences between AI and humans, and let them play to their strengths. It seems common to think that we need to constrain AI in certain ways to be more like humans, but the constraints are always artificial. APM is one constraint, but what about working memory? What about multithreading? We already allow the AI access to computational resources humans don't have. Why draw that line at APM, exactly?

themacguffinman 8 years ago | | |

I don't think they're constraining APM to make the AI more human-like. It seems more like they're controlling a variable, so that if the AI wins, they know it wasn't just because the AI could out-click a human.

usaphp 8 years ago | |

Would not actions per second be a better limit tho?

yannyu 8 years ago | | |

Why?

Speed is in MPH or KPH, would it be better to go by m/s?

The convention in gaming is APM, so they're just using the nomenclature that is already understood.

metachris 8 years ago | |

Which is still about 350 actions per minute for a professional SC2 player, roughly 5 to 6 actions per second.

JabavuAdams 8 years ago | |

Currently, though, APM isn't a real advantage for AIs. They're still too stupid.

EGreg 8 years ago | |

I would like to see IBM Watson dominate Jeopardy without a superhuman finger.

tantalor 8 years ago | | |

Watson’s hand looks like a clear, Plexiglas, cylindrical soda can with a few metal screws in the top and a wire extending from the bottom that is connected to Watson’s Front-End Controller. The mechanical hand wraps around the Jeopardy! buzzer which is inserted in the bottom of the Plexiglas cylinder and is held in place by a clamp. Watson’s hand uses a solenoid to physically press the same button that the humans must press.

Watson’s hand is pretty fast in terms of raw speed — it takes somewhere between five and ten milliseconds for Watson to activate the buzzer once it decides to answer. This delay is affected by the speed of the solenoid and other small, sometimes hard-to-pin-down delays inherent in the software stack.

http://www.kurzweilai.net/the-buzzer-factor-did-watson-have-...

I wonder if the human decision-to-buzzer time is much slower?

ehsankia 8 years ago | | |

Yes and no. I guess the more "fair" way to compare them at trivia would be to have a format where all 3 give an answer to every question, and whoever gets the most points wins. That's obviously not really jeopardy, and I'd be curious to know how Watson would perform purely on the trivia knowledge vs clicking speed. I'm guessing it would still win though.

Lambent 8 years ago |

It's not like this is going to create fantastic AI.

Keep in mind there's been an amateur AI project for broodwar for almost 7 years now. Even after such a long learning period, the games are very primitive, and the AI's still couldn't pose a threat to even a beginner human player. Sometimes the games take hours. Trying to build strategy and decision making into an AI is incredibly complicated. There have been teams working at the SSCAIT for many years now, and the product is still fairly primitive.

So what CA did was instead write up a simpler AI that mimics strategy and decision making. We all know it's not great, but I'd be really skeptical that 3rd parties would magically create an AI that can think strategically.

Outrageous 8 years ago |

Novice here: I really want to try this Starcraft API but I don't know how to start. I believe this uses more reinforcement learning and agent-based models (which honestly I am not familiar with yet) What are good papers to get started on this?