Simplest general way is literally asking the model for the X and Y coordinates to click, given an input image. Tends to work with most multi-modal models, although some have better accuracy than others, obviously.

More involved integration, would be to expose reading and writing to the game state or character controls via tools that the LLM can use (either in your own harness however you want, or via MCP). This is basically what most "AI tooling" connected to editors do today, one way or another, with fallback to "click on this UI element after taking a screenshot" often, or "simulate click on this widget with #id".

So given a goal, the LLM would first call check_inventory(), then craft_building(:building), then walk_to_coords(x,y), then build_building(:building) and so on, which tends to indeed be a lot faster than doing it by screenshots and inferring coordinates.

Ultimately I think the harness and prompt matters more than the model here, but I guess what they're exploring, is the shortest possible prompt to have it do it correctly. Otherwise, setting your harness to "Continue until you've launched a rocket in the game" together with a bunch of added rules to the prompts as you discover where it gets stuck, is an easy way to probably get most LLMs to be able to do this correctly and relatively fast. Some of them requires more handholding than others, which I guess might be another thing they're testing for here, it isn't 100% clear what the intention/motivation nor exact environment was/is.

Edit: As fitting as it is, I was just trying out the Unreal Engine Editor MCP/tools for the first time with Codex, which ended up also being a good example for how a "real turn" would look like, with a LLM connected to some other editor: https://gist.github.com/embedding-shapes/b16c50ed591569d3c70...

Factorio has an extremely advanced modding API, so the codex harness has access to the game data via the client side modding LUA api to read things, it also uses the same api that is exposed as MCP tool to send Input Actions - input actions are internal Factorio representation of mouse and keyboard. It also does use screenshots sometimes to understand what's goiung on, so it's not just the modding api proxy.

Inputs wise it plays exactly as a human does, there is no difference.

There are people much smarter than me to answer this, but the basic idea are that models have a harness around them that include something called "tools." Each tool is basically a text description of its functionality. So for something like this, for example, the model might have a "move" tool. All of the tools are added to the model's context. Then the basic idea is that when the model detects it needs to "move west 5 tiles" it will spit out a bunch of JSON to "call" the tool, and then at that point a normal program picks up that tool call output and actually performs the action. So in other words, the LLM isn't directly manipulating the environment, it's spitting out JSON that tells the harness it is in what to do.

Since I posted, it got to Fulgora and Vulcanus, successfully colonized then and at the moment is preparing to tackle Gleba.

It's not even showing signs of stopping or getting stuck.

There's vod: https://m.twitch.tv/videos/2867906622

Sleep is a bottleneck

Factorio has an extremely broad LUA modding API, so a mod was made that allows it to get data about the game and send input actions to control the game. It also has ability to make screenshots and it does use it from time to time to inspect whats going on.

What does Ultra offer in this context? Better code? Or is the problem being solved extremely complicated that requires Ultra?