Case Study

How We Turned an AI Video Into a Playable Game Level

From 62 seconds of generated gameplay to a level you can play in a browser: how we read the video, what each playtest changed, and why the record of every step mattered as much as the game.

10 min read
A generated game reference sheet: the boy adventurer Kai, six numbered gameplay beats across floating islands, and a cast of creatures
Where it started: one generated reference sheet for the hero, the beats of the level and its creatures. View it in the collection

We had 62 seconds of AI-generated gameplay footage for a game that did not exist. A boy in a blue scarf ran across floating islands toward a citadel at dusk, and every frame looked like a game. None of it was one. This is how we turned that video into a level you can play in your browser, and why the record of every step mattered as much as the game.

About the media in this post. The images and videos come from two public collections, the Skyground Citadel walk-through and the earlier extension collection. They are shared copies with the generation metadata stripped. The game's own models and textures are a separate public collection under CC BY 4.0.

What Did We Start With?

The video began as an experiment in extending AI video. A generated reference sheet (one image laying out a 30 second gameplay flow, with a hero, six numbered beats, and a cast of creatures) became the first frame of a 30 second clip. We then extended the clip to 62 seconds with LTX-2.5 in ComfyUI. That process has its own write-up: How to Extend an AI Video in ComfyUI.

The starting point: 62 seconds of a game that did not exist yet. View it in the collection

So we had two inputs: the reference sheet and the footage. The question was whether they could become a design document for a real level.

How Do You Read a Video as a Design Document?

Our first two design drafts got it wrong, and how they went wrong taught us the method. They were written from the text prompts used to make the clip, not from what the sheet and the footage showed. They invented a level layout and a single enemy. The sheet itself shows six enemies and a six-beat flow.

The fix was to tag every line of the design document by where it came from:

  • Sheet: printed on the reference sheet.
  • Clip: seen in frames of the footage.
  • Prompt: only in the text prompts, so unverified.
  • Proposed: invented so the build has something to be checked against.

That one habit changed the conversation, because disagreements became visible. The sheet promised a spring pad and moving platforms in beat two; frame by frame, the footage showed a plateau, a boardwalk with a flying creature, and a grapple swing over a river. The sheet promised a grind rail and a key in beat four; the footage showed a rope bridge and a chain of islands. Both differences were recorded as deviations and kept in view, not quietly smoothed over.

Why Build the Hero First?

The audience for an AI-built game has seen what current image-to-3D tools can do, so "it runs" is not the bar. We set a rule: the hero passes a human review before any level is built.

Kai went from the sheet to 3D in three steps. An image model produced clean front and back views from the sheet. An image-to-3D model built a mesh from them. A rigging step gave him a 41-joint skeleton and preset animation clips: idle, run, jump, fall, hurt, and later a flip for the double jump. The scenery (islands, castle, fences, coins) came through the same route, under one hard rule: reject and regenerate weak assets rather than patch them.

Front view of the hero Kai in an A-pose: spiky brown hair, blue neck scarf, goggles and a leather harnessBack view of the hero Kai, generated from the front view
Hero first: the front and back views the 3D model was built from. View in the collection

By our estimate, the whole generated asset set for Level 1 cost about $6 of cloud generation.

What Did the Playtests Change?

Each human playtest was short and specific, and each one reshaped the game.

The first playtest found three things. The camera, lighting, and animation worked. A cape-like cloth, a side effect of one of our own regenerations, rode through Kai's legs. And a young first-time player kept missing jumps, because the single jump demanded a run-up right to the edge. The cape went. A double jump arrived, with a short grace period after leaving a ledge and a buffer for early presses. The verdict on the next build was "it plays great now," along with one firm decision: keep the chance of falling. A seven metre gap that needs the double jump stayed in.

Frame rate was measured, not assumed: 59.9 frames per second median on a desktop with integrated graphics only, and no frame over 100 milliseconds. That build still had placeholder scenery, so it set a baseline rather than a sign-off.

Then six short sections became one journey: about 420 metres across six acts, from a start meadow to the citadel gate, with music and sound effects synthesised in the browser and a sky that drifts from blue day to pink dusk, as the video's last seconds do.

The next playtest judged the game against the video's feel, not against the sheet. The notes read like a list of what the footage had and the game lacked: Kai's shadow sitting right under him, grass that looks real, air that feels filled in, mists, water that flows into waterfalls he can splash through, and music that "can go a lot further." Every item was built: contact shadows, wind-blown grass, drifting motes, 92 mist banks, five streams with waterfalls, and a score rewritten as one tune in eight arrangements.

Before: the hero on a lawn with only a faint, small shadow and sparse grassAfter: denser, lighter grass that sways around the hero, a contact shadow under him, and a blossom tree
Before and after one playtest: the shadow and the grass. View in the collection
A stream curls over the edge of a floating island and falls into the mist below
Water that flows somewhere: one of five streams that end in a waterfall. View in the collection

How Did a Platformer Become a Game About Stewardship?

The third playtest asked for something bigger. The creatures should live: find food, play, make friends, raise young. Falling should cost a climb, not a restart. Grappling should feel unique ("not Spider-Man"). And coins and gems needed a purpose.

The answer was to make Kai the steward of the citadel grounds:

  • Three species live in three habitats (lawns, blossom trees, running streams). Eight start out of place, and Kai can carry them home.
  • The islands' vines catch a falling Kai, and he climbs back up. Vine blooms over the wide gaps take his hand. He never shoots a line; the grounds look after their steward.
  • Dry springs wake when Kai brings a sky gem, and a running stream is the only home for one of the species.
  • Coins pay for travel between waypoints Kai has already reached, which gives him a reason to go back.

Can an AI Player Tell You If a Level Works?

Before showing the game, we ran a check whose pass criteria were written down and frozen before any run. Five independent AI players, each seeing only text descriptions of what was in sight, played the level from the title screen.

Nothing softlocked. The enemy gauntlet was where they slipped most; the vines caught 20 slips, with only one return to a checkpoint. Their notes also found a real rule gap, which we fixed the same day.

What this did not show matters just as much. The level was easy for these players: none needed more than two attempts at anything. Their times are not human times. And no AI score says whether a game is fun. The human playtests carry that.

What Did Numonic Keep?

This is the part that turns a side project into a demonstration. Every pass was filed into Numonic as it happened: the sheet and the source footage, the hero's views and model, the generated scenery with the workflow that made each piece, screenshots of every build, and the design notes behind each pass. Each item went in as a Draft, tagged and grouped by pass, so the history reads in order.

When the game went public, that record made the release a publishing step:

  • The game's code moved to a public repository under the MIT licence.
  • Its 27 models, animations, and textures became a public Numonic collection under CC BY 4.0, so anyone can run the game from source.
  • The design history stayed private, but complete.

Building in public is easier when nothing has to be reconstructed afterward.

How Did We Get Ready for Level 2?

Level 1 ends with a steward's report and one line: "The citadel doors open. Next: inside the citadel." Before launch the game was renamed Skyground Citadel, a nod to Kai as groundskeeper, and the public repository became its source of truth, so outside contributions can be merged.

The hero on a fenced lawn before a tall spired citadel on a floating rock, under a pastel dusk sky
The end of Level 1, and the door to Level 2. View in the collection

Level 2 started the way Level 1 should have: with a design written before any build, pass criteria set in advance (stricter this time, because the AI players found Level 1 easy), and a spend cap for new models. It also started with a "dream": a generated reference sheet and a video of the level, which went through three rounds of review before it was approved. That is the next post: How We Reviewed an AI Video to Approval in Three Rounds.

Key Takeaways

  • Tag your sources. When a design comes from generated media, mark what the image shows, what the footage shows, what only the prompt says, and what you invented.
  • Build the hero first, and review it. A character that fails review will drag down every level built around it.
  • Judge against the feel, not the spec. The most useful playtest notes compared the game with the video.
  • Decide in advance what "works" means. AI playtesters can show a level is finishable; they cannot show it is fun.
  • Keep the record as you go. Filing every pass made the public release a publishing step, not an archaeology project.

Frequently Asked Questions

Turning an AI video into a game.