Luan-vP/fire-sprites

Fire performance video to sprites

You are guiding a performer from a raw video to sprite sheets they can play in a game. They know their performance; you run the tools. Keep them involved at the two points where their eye matters (which moves, and where each one starts and ends) and do everything else yourself.

The tools live in the tools/ folder of this kit (two levels up from this file). Call it $KIT. Every tool runs inside a project directory that you create for this performer; source videos are referenced, never copied.

Read references/how-it-works.md once before starting. It explains the three ideas the whole pipeline rests on: separate body and fire layers, exposure handling, and feet-anchored normalisation.

0. Set up (once per machine)

cd $KIT && python3 -m venv .venv && .venv/bin/pip install -r requirements.txt

Use a native Python build for the machine (on Apple Silicon an x86 Python silently installs an old, CPU-only PyTorch). ffmpeg and ffprobe must be on PATH. A GPU (Apple MPS or CUDA) is strongly recommended: rendering costs roughly 3 s per output frame on an M2 Max. Model weights download on first use.

Below, py means $KIT/.venv/bin/python and commands run from the project directory.

1. Ask for the right footage

Ask the performer for the original camera files, not an edited or social-media export. Look in the folder they point you at: a 4K, 10-bit, 50 fps original is often sitting next to a 720p final cut, and everything downstream is better from it. Also ask whether there is footage of them walking or standing before the fire was lit: fire routines rarely contain a clean walk cycle, and games need one.

mkdir -p ~/sprite-projects/NAME && cd ~/sprite-projects/NAME
py $KIT/tools/ingest.py main "/path/to/original.mov"      # repeat with other names for other clips

Read what ingest prints. It measures the real black level (file range tags are unreliable) and warns about 8-bit sources, low resolution and variable frame rate. Variable frame rate must be fixed first (re-encode to constant frame rate), or frame numbers drift between proxy and source.

2. Find usable footage

py $KIT/tools/pose_pass.py main                             # slow: about 15 proxy frames per second
py $KIT/tools/find_segments.py main --strips                # add --performer skin if a clothed crowd is in shot
py $KIT/tools/find_bursts.py main                           # fire breaths, found by brightness

• find_segments keeps runs where the whole body is in frame and upright. Expect a fraction of the footage to qualify: handheld cameras cut feet off constantly. That is normal; tell the performer the number.
• Look at a couple of strips/*.jpg yourself. If they track the wrong person, switch --performer mode.
• Look at bursts/main.jpg. Keep only real fire breaths, then py $KIT/tools/catalog.py add-bursts main (and delete any wrong entries from catalog/selected.json). Fire breaths are found this way because the pose-based search rejects blown-out frames.

3. Catalogue the moves (use subagents)

py $KIT/tools/catalog.py batches 5

Give each batch to a subagent using the prompt in references/catalog-prompt.md (fill in the batch number and describe the performer in one line so reviewers can reject the wrong person). They read the frame-numbered filmstrips and write catalog/outK.json. This is the step that benefits most from parallel agents: it is pure looking, and each batch is independent. Then:

py $KIT/tools/catalog.py select            # shortlist: quality >= 4, plus walks/turns/spins/strikes at 3
py $KIT/tools/catalog.py list

Tell the performer plainly what was found and what was not (for example: "13 idles, 8 spins, but only one real walk, from the clip before the fire was lit").

4. Performer trims and chooses

py $KIT/tools/make_trim.py                 # a context clip per move + trim.html
open trim.html

The reviewers' ranges are deliberately tight; real moves want to breathe into what comes before and after. In trim.html the performer drags the start and end of each move over a few seconds of surrounding footage, marks Keep or Drop, and copies the list from the box at the bottom. Ask them to paste it to you. The page stores its state in their browser, so you cannot read it yourself. Save the pasted text to a file and:

py $KIT/tools/catalog.py apply-trim kept.txt

5. Render, pack, review

py $KIT/tools/render_move.py --kept        # long-running: run it in the background
py $KIT/tools/pack.py --kept
open review.html

Before reporting, look at several moves/NAME/preview.jpg files yourself. Things to check and say out loud:

• another person merged into the cutout (set "gate": true on that move in catalog/selected.json if the move has no prop, then re-render with --force);
• fire-lit wall showing up in the fire layer;
• cropped feet or head: render_move prints the share of each range with the whole body in shot. Extended trims often run into footage where the performer leaves the frame.

review.html plays every move with background and layer switches, including "body + additive fire glow", which is how a game should draw it. atlas.json has the frame grid, fps, feet anchor and source range per move.

6. Optional: 8-bit look

If they want a retro version, use the fire-sprites-retro skill.

Honesty rules for this work

• Report counts and limits, not just successes: how many seconds were usable, which moves have cropped limbs, which fire layers include wall.
• The footage sets hard limits. A fire cloud that leaves the camera frame has a straight edge; a blown-out flame has no internal texture to recover. Say so instead of promising a fix.
• Never put the performer's footage, name, or rendered sprites anywhere public. Outputs stay in the project directory unless they ask otherwise.

See references/troubleshooting.md for problems met in practice and what fixed them.