game-brains

On-device models for games: a generic SDK, and a factory for game-specific models.

Ideas deck, 2026-11-20. From the Console Wars / Data Wars voice-RTS work.

The problem: voice in games today goes to the cloud

Data Wars (a voice-commanded RTS) on 17 consoles, through a cloud voice agent. Measured:

  • ~0.6 s to hear the end of speech
  • ~0.7 s more to a function call
  • + the network, + the round trip back to the game

It works, it's impressive, and it's clunky and slow for a game.

What a game actually needs

  • Tens to hundreds of milliseconds
  • Works offline, private, no per-request cost
  • Understands this game: its units, places, verbs, state
  • Never breaks the rules: the game validates everything

The bet: every platform will have an accelerator for this

When the hardware lands, studios will want ready-made, game-tuned models and a way to ship them. Get there first.

The principle: generic where it scales, specific where it wins

generic built once, every gamespecific made per game
The runtime: load a model, run it on the best device (CPU / GPU / NPU), tensors in, tensors outThe model weights: fine-tunes, adapters, heads
Thin engine plugins (Godot, Unity, Unreal): the game pipes in audio, state, eventsThe game's schema: actions, entities, state, events
The schema formatThe game's plan vocabulary: its verbs, conditions, resources
A deterministic plan executorTraining data from the game's own simulator
The factory: data generation, synthetic speech, training, evaluationWhat "correct" and "good" mean in this game
Base models: speech encoder, small LM, TTSVoices, persona, strategy knowledge

The game always validates: models propose, the game's own rules accept or reject.

Beyond "hey Pikachu": reference resolution

"Send the 3 data drones collecting at the data centers in A1, the 1 drone that isn't collecting, and the 2 in production to the enemy data center at D2."
Audiofrom the game
+
Game stateevery unit: type, team, cell, activity; every building: production
→
Modeltuned to Data Wars
→
Calls with IDsset_target(u17, dc31) ×4
rally(sw5, 2, dc31)

Beyond commands: standing orders and strategy

"Have my villagers prioritize berries, then sheep, and once we have enough food build this unit or that upgrade, whichever is more strategic given my overall plan."
villagers: priority [berries > sheep]
when food >= 500: choose(knights, blacksmith_upgrade) by objective "pressure early"
rally: next 2 drones from skunk_works#5 -> data_center#31

The layers

1 Understandaudio + state → intent, refs resolved
on-device, ms
→
2 Planintent → standing orders in the plan language
small LM, only when the player speaks
→
3 Executethe game runs the plan every tick
deterministic: netcode, replays
→
4 Decidedecision points → one option
rare; later a tiny learned scorer

The model catalog (1/2)

A. Understanding the player

  • Speech → intent (closed set)
  • Speech + state → resolved function calls
  • Text + state → resolved function calls
  • Voice UI / menu navigation
  • Turn-taking: talking to the game or to friends?
  • Wake words

B. Holding intent over time

  • Standing-orders compiler (→ plan language)
  • Strategic decision points
  • Companion / teammate agents
  • Opponents and adaptive difficulty (self-play)

C. Knowing the game

  • Game Q&A over state, quests, lore
  • Coach / hints at the right moment
  • "Previously on..." summaries

The model catalog (2/2)

D. Speaking

  • Game voices (TTS)
  • NPC dialogue locked to lore
  • Lip-sync from audio
  • Live translation

E. Players

  • Voice-chat transcription + moderation
  • Scene description (accessibility)
  • Player modelling
  • Behavioural anti-cheat

F. Content & dev

  • Procedural content the rules accept
  • Playtest bots by instruction
  • Telemetry summaries

G. Rendering / sim

  • Upscaling, neural textures, physics, animation (vendor space)

The factory: games have simulators

Sample statesrun the game: self-play, scripts
→
Generate commandsgrammars with exact labels from the state
→
Paraphrasea teacher LM; labels kept
→
Synthesize speechmany voices, noise, game audio
Trainshared bases + per-game heads / adapters
→
Plansteacher plans, played out in the sim; keep winners; RL later
→
Evaluatereal recordings, exactness, win rate, latency per device
→
Ship + retrainversioned with the build; per patch in CI

Unlimited labelled data for free, because the game can compute the right answer for any state.

Who already does what

WhoWhatGap for us
Unreal NNEOne API over runtimes: ONNX Runtime (CPU, DirectML), OpenVINO incl. NPU, QualcommPlumbing only; consoles unclear
Unity Inference EngineONNX on all Unity platforms; DirectML on XboxPlumbing only
NVIDIA In-Game Inferencing + ACEOne API across TensorRT / ONNX / llama.cpp / DirectML; on-device ASR, SLM, TTS; in shipped gamesNVIDIA-first, PC-only, characters not game-tuned brains
Windows MLOS-level ONNX Runtime, picks NPU / GPU / CPUWindows only
Godot addonsCommunity ONNX wrappers, a Vulkan inference addonNo supported SDK
Inworld, ConvaiCharacter middleware, mostly cloudNot on-device, not game logic

Nobody sells "a model tuned to your game, built from your simulator, retrained every patch, running on every device."

The SDK pieces

Runtime (C core)

  • Model package: weights + schema version + device hints + budgets
  • Backends: ONNX Runtime + its providers, a Vulkan / compute path, ggml for LMs
  • Async, frame-safe; per-platform memory / latency / power budgets

Game-facing

  • Plugins for Godot, Unity, Unreal: the game pipes in audio, state, events
  • Schema from engine reflection (one annotation per action)
  • Plan executor + orders panel helpers
  • Editor playground: speak → see the parsed calls → see the game accept or reject

First testbed: Data Wars in Godot

PhaseWhat
0 PlumbingGodot model runner (start from an ONNX Runtime extension), CPU first, then GPU and NPU
1 CommandsSchema + reference resolution; baselines: STT + small LM, STT + pointer model, then end-to-end speech + state
2 Standing ordersThe plan language, its executor, the compiler, an orders panel
3 StrategyDecision points, self-play distillation, win rate as the judge
4 BeyondUnity / Unreal plugins, a second game, console programs (ID@Xbox first)

Test device: a Ryzen AI 9 HX 370 laptop (Radeon 890M GPU, XDNA 2 NPU, visible on Linux; NPU tooling on Linux is young).

Open questions

Next

  1. Pick the Data Wars schema and the plan language's first verbs.
  2. Build the factory's first cut from dwcore and a Godot model runner.
  3. Beat the cloud path on latency, then on accuracy for complex commands.
← → / space / click · F fullscreen