On-device models for games: a generic SDK, and a factory for game-specific models.
Ideas deck, 2026-11-20. From the Console Wars / Data Wars voice-RTS work.
Data Wars (a voice-commanded RTS) on 17 consoles, through a cloud voice agent. Measured:
It works, it's impressive, and it's clunky and slow for a game.
| generic built once, every game | specific made per game |
|---|---|
| The runtime: load a model, run it on the best device (CPU / GPU / NPU), tensors in, tensors out | The model weights: fine-tunes, adapters, heads |
| Thin engine plugins (Godot, Unity, Unreal): the game pipes in audio, state, events | The game's schema: actions, entities, state, events |
| The schema format | The game's plan vocabulary: its verbs, conditions, resources |
| A deterministic plan executor | Training data from the game's own simulator |
| The factory: data generation, synthetic speech, training, evaluation | What "correct" and "good" mean in this game |
| Base models: speech encoder, small LM, TTS | Voices, persona, strategy knowledge |
The game always validates: models propose, the game's own rules accept or reject.
villagers: priority [berries > sheep] when food >= 500: choose(knights, blacksmith_upgrade) by objective "pressure early" rally: next 2 drones from skunk_works#5 -> data_center#31
Unlimited labelled data for free, because the game can compute the right answer for any state.
| Who | What | Gap for us |
|---|---|---|
| Unreal NNE | One API over runtimes: ONNX Runtime (CPU, DirectML), OpenVINO incl. NPU, Qualcomm | Plumbing only; consoles unclear |
| Unity Inference Engine | ONNX on all Unity platforms; DirectML on Xbox | Plumbing only |
| NVIDIA In-Game Inferencing + ACE | One API across TensorRT / ONNX / llama.cpp / DirectML; on-device ASR, SLM, TTS; in shipped games | NVIDIA-first, PC-only, characters not game-tuned brains |
| Windows ML | OS-level ONNX Runtime, picks NPU / GPU / CPU | Windows only |
| Godot addons | Community ONNX wrappers, a Vulkan inference addon | No supported SDK |
| Inworld, Convai | Character middleware, mostly cloud | Not on-device, not game logic |
Nobody sells "a model tuned to your game, built from your simulator, retrained every patch, running on every device."
| Phase | What |
|---|---|
| 0 Plumbing | Godot model runner (start from an ONNX Runtime extension), CPU first, then GPU and NPU |
| 1 Commands | Schema + reference resolution; baselines: STT + small LM, STT + pointer model, then end-to-end speech + state |
| 2 Standing orders | The plan language, its executor, the compiler, an orders panel |
| 3 Strategy | Decision points, self-play distillation, win rate as the judge |
| 4 Beyond | Unity / Unreal plugins, a second game, console programs (ID@Xbox first) |
Test device: a Ryzen AI 9 HX 370 laptop (Radeon 890M GPU, XDNA 2 NPU, visible on Linux; NPU tooling on Linux is young).