You could have complex realistic dialogue for innkeeper #28917 in the little hamlet of ass backwards, but:
One: do you really want that? There is such a thing as too much realism in a video game.
Two: do you really want all NPCs in all games to speak nigerian business english?
Reminder for 2: https://www.theguardian.com/technology/2024/apr/16/techscape...
Who wants to read AI-generated articles and respond to AI-generated comments? Probably not many. For same reason gamers mostly don't want to chat with AI-backed NPCs. More is only better when it has meaning.
Who would want to have a non-deterministic, unreliable, out-of-your-control companion in a single player game?
What kind of game designer would find it acceptable to have no control over large swathes of their own game? Who would like to play a game that no one really created?
Games are where art meets engineering to create entertainment and awe. LLMs don’t fit.
Well there are a lot of AAA gaming companies that should learn that. But instead they feel they have to ad MOAR CONTENT and then complain of balooning costs...
The moment where the dog is going on about "something foul in the air" as the player is attacked by a wolf ("F--- dude you could have warned me!") was great comedy.
I do think it is an interesting exploration but the jagged frontier makes it really challenging to know what will consistently work and what will not, to the point that I bet (maybe pessimistically) one will be gradually less daring with creative plan with their "companion" simply because they can't trust it.
The Uncanny Valley effect would be too big if a human companion randomly started talking to you while you were sneaking up on an enemy. But a dog? You'd be pissed off at it for a bit, and then forgive it because it's a dog.
And of course the opposite too, how much the dog trained me to speak to it a certain way to maximize outcome success.
But then I thought, when people play games they are not using highly sophisticated vocabulary and there is probably lots of repetition since they are always under some form of multi-tasking stress (playing and replying/speaking). So maybe... maybe, the system can adjust itself. Use a big LLM offline to say "user said X, we did Y - was that good?" - then retrain itself.
The decomposer is basically a bunch of old-school embeddings/classifiers stitched together, it can train super fast and doesn't need tons of data. Could the thing calibrate itself to the user? Does it even need to? (because as I said I 'm a datapoint of 1 and I am not ready for the potentially huge stream of bug reports when I ship (add some perfectionism to the mix and you get the idea)).
edit: typos
Also the author really gets the "tortitude" right.
Now AI does all the coding for me, I couldn’t have been more wrong. You will be wrong too.
I do wonder if this is an avenue for console gaming that might be practical in a few years; AI-centric hardware that might be too beefy or expensive for regular users, but can extend new or existing games. Kinda like the expansion paks of old.
unfortunate that the "ALE" design wasn't opensourced (couldn't find a link in their post) but I would be interested in learning more about the design, in particular what sort of data pipeline was necessary from skyrim to give this sort of action flexibility?
Ale is what makes this work locally, I felt a little conscious about it as I am not sure if it is a novel approach or somebody comes out and claims I rediscovered BERT or something (though ale runs at 1/10 the cost of BERT).
It either might be some neural engine like what Apple is doing and tensor units like what high end android phones already have or something dedicated.
Based on the current capacity issues around the globe, perhaps 1-5 years?
But we have for sure crossed a price point were you just might buy 10 ai credits and will be able to just play 100h without running ot of your ai credits. basic conversations etc. finetuned for a game, doesn't need a frontier model.
If the AI Bubble bursts a little and the inflated prices for tech dwindle down to normal maybe, otherwise, it will be too expensive.
Or some other advancement that challenges the giants financially but incentivizes companies to build for local AI usage.
If nothing else - this is how NPCs should work in games moving forward!
An LLM predicts the next token. If you're trying to predict the next token in a mathematics competition, or while playing a deep strategy game, being a much larger and more capable model helps enormously. To predict that next token correctly, the model effectively needs to model a bunch of possible future states - even if that is a second order (unintended) effect, it is what is seems to be happening.
This is basically the Ilya (and Dario) argument that prediction, understanding, and compression are the same thing (deep rabbit hole) from a few years ago.
----
In my opinion; this is a beautiful idea, but videogames do not need most of that. Videogames (and games in general) shine when character behavior is predictable, and when NPCs are a little dumb (just a little).
We already have very good small roleplaying models — Qwen 3.5 4B/9B/30B-A3B. Nowhere near frontier models at general reasoning. But they can act and write in a very engaging way. Good at roleplaying but very weak at reasoning. They just need a little nudge at reasoning...
And that's the key. The player has already expressed their intent: attack that guy, go look over there, cover me, find the key that shines and is golden, etc. A constrained world, with a constrained set of actions. Instead of asking the model to reason over an enormous space of possible futures, we're mostly asking it to map: player intent + current world state → a small sequence of plausible actions.
As for the "dump context to an LLM". It's basically. "You are roleplaying as X - you experienced Y - you like/dislike (dispositions) Z, you remember Alpha, your journal says Delta. Player orders you to do Gamma. - "What do you respond and do?"
It kind of works (as you can see in the videos I posted). I am not going against the grain, big models are better, but do we need those models for everything?
Say for instance when you ask the dog to do an action when you launch an arrow upwards how is the LLM continuously tracking the state of the game to be able to respond?
> ALE is designed to be largely invariant to phrasing. You can say pick up, you can say grab, fetch, go get the damn sword you fool - it doesn't matter, it will still understand you > It creates embeddings from the full text as well as its extracted structure
I get how you utilize embeddings, but most interesting part for me is how you decompose multiple commands? You decompose sentences before embedding?
What you'd want is maybe some kind of Live model with voice warping so it can be given different Skyrim themed 'Nordic' voices, and then custom tools to interact with the game engine.
The fun technical challenges (that can also act as any sort of weak moat) are being taken away one by one, on an almost weekly cadence now! :)
So even when it chokes or stumbles on a command, the kind of frustration the user expresses when correcting it feels natural and part of the game even.
I'm surprised how good the talk is (i.e. the voice recognition). My disabled brother is struggling with good native polish voice to text btw. If you people can recommend anything good or local
Runs local and has several models that should support Polish, though I can not personally validate this.
The only thing it has trouble with are brand/product/tooling/person names. You can add custom vocabulary, but then it tends to over-correct and insert those custom words when I never said them. However this isn't unique to local STT, frontier cloud models also struggle with this.
Does this mean the prototypes and classifications need to follow what you can actually do in the game? Were they all hand-coded or generated somehow?
> "Maybe it's because Varkos is a dog, and who doesn't like dogs"
Me
Talking to it alleviates pressure so that mechanism doesn't fire much, but in the void-mode video on the website, at the very end, it chooses to chase down an elk on its own much to my surprise.
After play testing it I had to tone it down quite a bit as it caused continuity issues...
In one scenario the dog was hungry so he went and ate something that was owned (counts as a crime attributed to the player) so we ended up in jail. Being transported to jail fired a new location event and a switch from "exterior->interior" which the dog responded to by having the llm author something along the lines of "It's nice to be finally indoors, it was getting cold outside". Which made for a very funny moment but a frustrating gaming experience.
tl;dr: let's you talk to all NPCs via LLM. Last time I tried it, latency was too much for it to be enjoyable, your approach seems to be much faster.
How is the personality evolution saved? Plain text?
Would be better if he just barked
Ive seen something like this in sci fi films.
When watching the vids, I remembered talking (mostly text chat) in MMORPG's 20 years ago. Even when we started using Skype, etc., chatting was frequently short, slang, and quick info. Like, "pulling", "OTM", "BRB", "OOM". Or even just "let's go", and so on.
But with some practice sessions, I suspect this system would pretty quickly gravitate to using any players slang speech. Am I wrong?
Some grumpy old school hunters (in less civilized areas) would literally shoot such a dog though for barking in the wrong moment.
People often love to fus ro dah Skyrim companions off of cliffs because they can be incompetent annoying little dumb things that won’t shut up.
The character taking on canine form won’t suddenly make gamers more likely to put up with it messing about their gameplay. Thats just not how things work.
source?
For the arrow example: wait here -> bind: arrow event -> pick_up $target -> goto player -> drop $target.
A plan can bind to multiple steps and actions. eg you can tell the dog, "when I say apple you say banana" this creates a short lived "player has said %apple%" -> say "banana" rule. Hide and seek does "say: count to 10" (this then gets fed to llm that converts it to "one, two, three"). So, the plan decomposition is fuzzy, but it gets decomposed to concrete steps, these steps might invoke the LLM back if needed so, but mostly for color, not logic.
Where the LLM is used afterwards, is for evaluating the result. The dog might ask for a treat if it got right or get frustrated is it misses a beat, etc (depending on emotional state, hunger levels, etc).
Basically trying to use the best of each system (fuzzy/vague/emotional for LLM, "hard" game actions for determinism). You can think of Ale's output as a small flowchart.
I am using a pretty dumb model. It's great for speaking, terrible for thinking, so there's a need for creativity
How often are you running the world JSON through the planner? When do you give up on fuzzy matching if an action doesn't fit?
Do you use any vision tools? such as the plugin from https://github.com/MinLL/SkyrimNet-GamePlugin
Super cool project btw, thanks for sharing!
Or maybe it won’t play the game for me but will do whatever it wants, mess up about my gameplay because it inferred it should do something I did not order it to do.
Either way, awesome! Lovely gaming experience. I remember being a kid and thinking “wouldn’t it be so much more fun it if I didn’t have to actually play the game?”.
"Twitch"
It will construct the game around you as you play. It will customize the game according to you and the situation dynamically.
The first step is AI npcs and AI generated quest lines.
The next step is the entire world, the entire story, the entire game will be dynamically constructed as soon as you start it up.