S3LFSP4RK mesh ulam lic AGPL --:--:-- UTC
// field note

We Built an AI Dungeon Master That Lives in Our House

2026·07·13#local-ai#embodiment#gamedev13 min read

How one gaming PC became K0R3: a cyberpunk game master with a face, a voice, a cast of characters, and a strict rule that she is never, ever allowed to touch the dice.

K0R3, holographic host of the table, keyed into a rain-soaked Night City street K0R3 — the host. A hologram with an opinion, projected into every scene she narrates.


The pitch, in one breath

Friends on the couch. Phones in hand. A TV that performs. You hold your phone like a walkie-talkie, say "I kick the door," and a synthetic game master called K0R3 — running on the PC in the corner of the room, not in some datacenter — narrates what happens next in her own voice, with her own face on screen, while a very polite 1980s robot named Retrobo spins a wheel to decide your fate.

That's the project. This post is the full story of how it got built: the architecture, the cast, the disasters, and the one design rule that makes the whole thing work.


The rule that makes it work: the AI never touches the dice

Here's the thing nobody tells you about using an AI as a game master: it cheats.

Not maliciously. Enthusiastically. Early playtests with our local model were a comedy of corruption — a player would fail a roll and the AI would just… roll again until they passed. Failure never cost anything. Ask it for your character's stats and it would invent them on the spot, confidently, differently each time.

An AI game master that fudges dice isn't a game master. It's an improv partner wearing a referee's shirt. Games need stakes, and stakes need math that nobody — human or machine — can sweet-talk.

So the whole system is built on a three-layer doctrine:

Humans LEAD. Python GROUNDS. AI DRIVES.

  • Humans lead — players make every meaningful choice, through their phones.
  • Python grounds — boring, deterministic code owns the dice, hit points, clocks, and consequences. When you roll an 8, an 8 is what happened. There is a ledger. There are no re-rolls. We call this "the dice are LAW."
  • AI drives — the language model gets exactly one job: performance. It narrates the result the engine already decided, in character, with menace or mercy as the scene demands.

The AI is the actor, not the accountant. The moment we split those jobs, the game stopped being a chatbot and started being a table.


The brain lives in the house

K0R3's mind is a 14-billion-parameter open-weights model (Qwen3-14B) running on a single consumer graphics card — an RTX 4080 Super, the kind of GPU people buy to play games, now employed to run one. Speech recognition (Whisper), voice synthesis, face animation — nearly the whole stack runs on that same box.

Why local? Three reasons:

  1. Privacy. A game night is people being silly and vulnerable in their living room. That audio doesn't need to visit anyone's cloud.
  2. Cost. A four-hour session generates thousands of AI calls. Locally, the marginal cost is electricity.
  3. Ownership. Nobody can deprecate our game master. She's ours.

(One honest asterisk: two hero voices in the cast use a commercial cloud voice service, because after a lot of A/B testing those two specific performances were worth it. Everything else — brain, ears, most voices, faces — is local. More on that below.)


Casting call: eight characters, no actors

A campaign needs a cast. Ours is a cyberpunk tragedy called Sandevistan Burnout — 37 scenes about a courier whose military-grade implant is slowly eating them alive — and it needed faces: a back-alley surgeon, an information broker, a terrified street medic, a corporate netrunner, a grieving lover, a loan shark, and the cold chrome mask of the implant itself.

The cast of Sandevistan Burnout — eight AI-generated faces The cast. Every face is generated — short video loops of each character breathing and blinking, waiting for their line.

The pipeline for each character:

  1. Generate an "idle loop" — a few seconds of video of the character just existing: blinking, breathing, holding still. We learned to prompt for perfectly symmetric motion, because…
  2. Boomerang it — the clip plays forward, then in reverse, seamlessly, forever. A character can now sit on screen indefinitely without a visible loop point. (This is also why the motion must be symmetric — a head-turn played backwards looks possessed.)
  3. Lip-sync on demand — when a character speaks, a lip-sync model (Wav2Lip) grafts the actual mouth movements for that exact sentence onto the moving face. Not canned mouth-flapping — the mouth you see is shaped by the audio you hear.
  4. Cut them into the scene — an AI matting model (Robust Video Matting) lifts the character cleanly off their generated background so they can be composited into whatever location the scene demands, like a weatherman in front of the map.

Marshell, the loan shark, matted into a neon-rain street Marshell — "I'm a businessman. You pay me, we stay friends, capisce?" — cut out of his original footage and placed into the scene.

The partner, composited into a crowded club The grieving partner, standing in a club full of people who don't exist either.

The robot that broke the matting model

My favorite failure of the entire project: the matting model that cuts characters out of their backgrounds is trained on humans. When we fed it the Implant — the fractured chrome death-mask you can see in the cast grid — it looked at our beautiful robot face, decided "not a person," and erased the character entirely, delivering a lovely clean shot of an empty street.

The machine built to find people in video refused to acknowledge our machine. The fix was to route non-human faces through an older color-keying technique instead. But there's a whole essay hiding in that bug.

The Implant's chrome mask over a time-dilated street The Implant. Too inhuman for the human-detector — which, honestly, is on-brand.


Voices: describe them into existence

The voices were the biggest surprise of the whole build.

Most text-to-speech makes you pick from a menu of stock voices. The local model we use (Qwen3-TTS "VoiceDesign") works differently: you write a paragraph describing a voice, and it becomes that voice. Our character sheets already contained descriptions like:

"Veteran back-alley ripperdoc, female voice in her 40s, dry cynical smoker's rasp, short clinical sentences, slows and sharpens each word when she's lying…"

Feed that description in, and that comes out. The casting document literally is the voice. When we A/B tested it against the commercial cloud service for our side characters, the local describe-a-voice won every single matchup — because the cloud service was ignoring our descriptions and doing its best with a fixed stock voice, while the local model was performing the character sheet.

Two exceptions survived, on merit: K0R3 herself (a hand-picked, heavily tuned commercial voice that has become her identity) and Retrobo (whose flat synthetic courtesy was too perfect to give up — so we pre-rendered his entire repertoire, 35 lines, into an instant-fire soundboard).

The bug where half the cast changed gender

One gotcha worth sharing for anyone building with describe-a-voice models: gender cues buried in the middle of a description get ignored. "A smooth male voice in his 40s…" reliably produced a woman. Three of our male characters came out female before we caught it.

The fix is almost comically blunt: front-load it, repeat it, hammer it. "A deep, gruff, unmistakably MALE voice — a man…" Works every time. These models weight the start of the description the way people weight first impressions.


Retrobo, the dice roller who dreams of ballet

The dice needed a face too, and giving them one accidentally produced the soul of the whole project.

R3TR0B0 — Retrobo — is a courteous 1980s service robot stranded in 2077. His understanding of technology froze in 1989 (a cyberdeck is "a Commodore 64 you wear as a hat"). He never swears. He announces every dice result with polite, devastating melancholy. And he secretly dreams of dancing ballet for the Bolshoi.

Retrobo presents the Wheel of Chance, landed on 8 The Wheel of Chance. The number it lands on is the actual number the Python engine rolled — the animation and the math can never disagree, because the math comes first.

When a roll matters, Retrobo doesn't just announce it — he presents a spinning two-tone wheel on screen that decelerates and lands on the exact number the engine already rolled. The spectacle is generated from the truth, never instead of it. Every layer of this project repeats that pattern: the show is downstream of the math.

And tonally, he's load-bearing. Sandevistan Burnout is a heavy campaign — loss-of-self horror, the slow arithmetic of a body being spent. Retrobo is the pressure valve. The design doctrine we landed on: the darker the campaign, the harder the comedy has to work — a laugh right before the gut-punch makes the gut-punch land. A one-joke robot turned out to be a precision instrument.


The ghost in the machine, literally

One more character deserves her close-up. dead_input is the player character's dead partner, surfacing inside the implant as a recorded personality — warm, intimate, and increasingly wrong as the campaign progresses.

dead_input — pale, desaturated, glitching The engram. Pale wash, drained color, and a periodic digital glitch — because she's not a ghost, she's a corrupted file that loves you.

Her look is built in the compositing layer: a cold desaturated wash over her face, plus a chromatic glitch that tears through the image every couple of seconds. Her voice arc is written into the campaign itself — she opens as comfort and curdles, one notch per chapter, into something hungry. She's the thesis of the campaign in a single character: the things that love you, running on hardware you can't trust.


Game night: phones, TV, and a nagging hologram

So what does it actually look like when people play?

  • Your phone is your controller. Scan in, and you get a hold-to-talk button plus your live character sheet — hit points, gear, the works. Say your action out loud; Whisper transcribes it; the engine and K0R3 take it from there.
  • The TV is the stage. Animated scene backdrops, the speaking character's living face, a persistent HUD: an in-world clock and weather, a GPS-style minimap of Night City, the crew's health bars, and a scrolling transcript of everything said and done.
  • Everything routes by name. Talk to someone — "Hey Retrobo," "Needles, what'll it cost?" — and that character answers, in their face and voice. Declare an action and K0R3 adjudicates it through the engine.
  • The engine has teeth. Fail a roll and a clock ticks somewhere. Fumble and you bleed. There's a mini-game layer too — the campaign's climax resolves on a single honest spin of Retrobo's wheel, because by then the story has earned it.

THE SHAKEDOWN — a full scene with HUD, minimap, crew health, and scrollback A full table beat: K0R3 narrating, the HUD live (clock and acid-rain forecast top right, district map top center, crew health top left), and the scrollback logging every line in the lower left.

Even joining the game is a bit. During onboarding, K0R3 walks each phone through character setup — and if you take too long, she starts heckling you from the TV, escalating from patient to theatrical despair ("I am a hyper-advanced intelligence and I'm stuck waiting on a MEAT KEYBOARD"). There's a skip button for veterans. She blesses your skip with grudging respect.

Title card: THE SHAKEDOWN Scene title cards slam in with an RGB-split glitch — the table is a show, and the show has production values.


What we learned (the transferable stuff)

If you're building anything in this space, the lessons that cost us the most:

  1. Separate truth from performance. The single most important architecture decision. Deterministic code owns state and consequences; the model owns delivery. Every failure mode we hit — cheating dice, hallucinated stats, drifting rules — traced back to letting the model near the truth layer.
  2. Local models are further along than you think. A 14B model on a consumer GPU runs a coherent, branching, 37-scene campaign — if you don't ask it to also be the referee. Describe-a-voice TTS beat a paid cloud service in our blind tests, on our own hardware.
  3. Specialist models have blind spots shaped exactly like their training data. The human-matting model erased our robot. The voice model ignores mid-sentence gender. You don't fix these with cleverness; you learn each tool's shape and route around it.
  4. Latency is a theater problem, not just an engineering problem. Local models pause to think. Pre-rendered filler — Retrobo's dice quips firing instantly while K0R3 composes her real response — makes the wait invisible. The audience forgives a beat of silence if someone on stage is talking.
  5. Protect the work like it's already precious. Halfway through, we realized weeks of the build existed as a single uncommitted copy on one drive. Version control plus off-machine backups became a ritual after that. Learn from our near-miss, not your own.

What's next

The first full couch playtest is imminent — real players, real phones, the whole campaign end-to-end. After that: recorded sessions on YouTube, more campaigns (the engine is campaign-agnostic; the cyberpunk noir just came first), and a second table's worth of cast.

The long-term shape is a local, private, embodied game master for any table — a machine that performs, grounded by math it cannot cheat, owned by the people whose living room it lives in.

If that's a thing you want to build, argue with, or play — reach out. The table has room.


— S3LFSP4RK

The stack, for the technically curious

LayerWhat we use
Game master brainQwen3-14B (local, via ollama)
Rules, dice, HP, clocksDeterministic Python engine — "the dice are LAW"
Speech-to-textWhisper (local)
Character voicesQwen3-TTS VoiceDesign (local, describe-a-voice) + 2 tuned cloud hero voices
FacesGenerated idle video loops → boomerang → Wav2Lip lip-sync (local)
CompositingRobust Video Matting (humans) / luma key (non-humans) + ffmpeg
Scene artLocal image gen (Flux via ComfyUI) + generated animated loops
The tablePhones (web portal, hold-to-talk) · TV (browser "web-face" with HUD) · one RTX 4080 Super

← all field notes