chengyansen-ai/h3-multimodal-lora-training

H3 Multimodal LoRA Training

Turn a training idea and local multimodal dataset into a reproducible H3 LoRA package through exactly ten gated steps. Treat the workflow as a state machine: finish and verify the current layer before entering the next one. Never call a LoRA “Ref2VA-trained” merely because it can be loaded by a Ref2VA inference workflow.

Start here

1. Read references/task-routing.md and classify the request before proposing parameters.
2. Initialize a project with scripts/h3_project.py init when the user wants files created.
3. Record progress with scripts/h3_project.py status and advance; do not report a later step as complete while an earlier gate is open.
4. Read only the reference associated with the active step:
   • dataset design or audit: references/dataset-contract.md
   • trainer choice or configuration: references/backend-recipes.md
   • loss, timestep, shift, AdaLN, or distillation: references/h3-math-and-guards.md
   • LoRA variants, loss weighting, Turbo, or new algorithm proposals: references/algorithm-options.md
   • checkpoint comparison or delivery: references/evaluation-and-export.md
   • provenance or disputed claims: references/source-ledger.md

The initializer copies ready-to-adapt examples from assets/ into the project’s examples/ directory. Adapt the matching example into the required path; do not edit the example in place and mistake it for a completed gate.

Initialize:

python scripts/h3_project.py init --root <project-directory> --name <experiment-name> --route <route> --backend <backend> --model-mode <fl2va-or-ref2va>

If python is unavailable, locate an existing Python runtime; do not install software without authorization. All bundled scripts use the Python standard library; the selected trainer supplies its own dependencies. ffprobe is optional for early preparation but required for final media evidence.

Executable backend priority:

1. diffsynth: production Ref2VA adapter for image, video, audio, and video+audio references;
2. ai-toolkit: low-memory local adapter for image/video references and embedded target/reference-video sound, with explicit rejections for unsupported audio sidecars;
3. simpletuner: FL2VA, joint AV, or audio-only adapter; the reviewed path does not claim Ref2VA training.

Authority and safety boundary

• Inspection, planning, hashing, local project scaffolding, and configuration generation are reversible preparation.
• Downloading large checkpoints, installing dependencies, launching/resuming training, using paid compute, uploading data, or publishing weights requires the user’s explicit request or approval at that point.
• Never upload private faces, voices, credentials, or unreleased media as part of diagnosis.
• Do not train without a recorded rights basis for every sample. Escalate ambiguous likeness, voice, copyright, minor-safety, or geographic/license questions; do not convert uncertainty into a permissive assumption.
• Preserve existing project files. The project initializer refuses non-empty targets; generated artifacts must not overwrite a run silently.
• Pin the base-model revision and trainer commit. Community defaults are hypotheses, not official H3 requirements.

Routes

Use one primary route and optional secondary evaluation goals:

Route	Learns	Minimum useful data	Required evaluation
identity	person, character, object, clothing	varied images plus motion clips when possible	identity, prompt adherence, motion preservation
style	visual rendering or cinematography	diverse subjects/scenes; images and clips	style strength, content leakage, motion preservation
motion	action, physics, camera movement	temporal clips with structured captions	held-out subjects, trajectory, timing, camera compliance
joint_av	video and target audio together	real synchronized stereo audio/video clips	video, audio, lip/event sync, silence behavior
true_ref2va	use of reference image/video/audio conditions	targets explicitly separated from references	zero/one/multi-reference and order-swap tests
turbo	few-step DMD/Turbo acceleration	teacher/student-specific data	speed-quality frontier and teacher retention

turbo is a separate objective. Never reuse its rank, shifts, or loss recipe as a default identity/style recipe.

The ten nested steps

1. Freeze the outcome

Capture the subject, route candidates, target inference workflow, quality priority, trigger, intended use, budget, deadline, and whether the user wants preparation only or an actual run. Write records/01-intake.json.

Gate: one measurable primary outcome, one inference target, and an explicit execution boundary exist. If the request is vague, propose a conservative default and label it as a proposal.

2. Prove rights and feasibility

Record dataset rights/consent in records/02-rights.json. Inspect GPU model/count/VRAM, system RAM, disk, OS, Python, CUDA, BF16, ffprobe, base checkpoint availability, and inference host in records/02-environment.json.

Collect machine evidence with:

python scripts/probe_environment.py --root <project> --backend-repo <pinned-checkout>

Gate: authorized=true, every sample can map to a rights record, the H3 license is acceptable for the intended territory/use, and the chosen hardware route is plausible. Do not use inference-memory figures as training estimates.

3. Select the learning route

Apply references/task-routing.md. Separate these claims:

• load_compatible: a LoRA file can be attached to that inference graph;
• condition_trained: reference media actually entered the training conditioning sequence;
• audio_trained: target audio contributed a non-masked loss.

Write records/03-route-decision.json with accepted route, rejected alternatives, evidence, primary backend, fallback backend, and evaluation obligations.

Gate: data semantics and backend capabilities match the claim. A T2V LoRA loadable in Ref2VA remains condition_trained=false.

4. Build the dataset contract

Create data/manifest.jsonl using references/dataset-contract.md. Keep target video/audio separate from reference image/video/audio. Split by source or subject, never by adjacent frames from the same source. Captions describe visible/audible variation and temporal structure, not just quality slogans.

Gate: unique IDs, existing local paths, non-empty captions, explicit split, rights IDs, no target/reference self-copy, and all route-required modalities are present.

5. Audit and preprocess without leakage

Run:

python scripts/lint_captions.py --manifest <project>/data/manifest.jsonl --route <route> --trigger <trigger> --output <project>/reports/04-caption-lint.json
python scripts/audit_dataset.py --manifest <project>/data/manifest.jsonl --rights-ledger <project>/records/02-rights.json --project-root <project> --route <route> --trigger <trigger> --output <project>/reports/05-data-audit.json

Use --hash-media for exact-file duplicate checks and --strict when warnings must fail CI. Review contact sheets and representative clips manually; automated probes cannot prove identity, anatomy, motion, sync, or aesthetics.

Gate: hard_failures is empty; every rights_id maps to training_allowed=true; audit hashes match the current manifest and rights ledger; caption warnings are reviewed; duplicate/leakage findings are resolved; 24 fps, dimensions, duration, frames, 32 kHz stereo, A/V offset, clipping, and silence warnings are either corrected or recorded as backend-handled exceptions.

6. Lock one falsifiable experiment

Choose resolution/frame buckets, rank/alpha, target modules, learning rate, optimizer, precision/quantization, checkpoint cadence, validation seeds, shift recipe, target sign, modality loss weights, and distillation guard. Do not present guessed values as user-approved facts.

Create records/06-experiment-lock.json:

python scripts/h3_project.py lock --root <project> --dataset-manifest <project>/data/manifest.jsonl --profile <project>/configs/training-profile.json --model-revision <sha-or-tag> --trainer-revision <commit>

Gate: dataset/profile hashes, base revision, trainer commit, route, conditioning claims, math convention, validation prompts, and one-change-only hypothesis are immutable for this run. Step 7 translates this neutral profile into a pinned backend config without changing its semantics.

7. Resolve the pinned backend configuration

Read the selected backend’s local README/config examples at the pinned commit and then references/backend-recipes.md. Generate configs/backend-config.* plus records/07-launch-plan.json containing the exact command, environment, estimated resources, output paths, resume behavior, and stop conditions.

For the three executable adapters run:

python scripts/prepare_backend.py --root <project> --backend-repo <pinned-checkout>

Add --materialize hardlink (or symlink/copy) for AI Toolkit and SimpleTuner because those trainers consume folder datasets. DiffSynth consumes the generated configs/diffsynth-metadata.json directly. The command rejects unknown profile fields, writes requested/resolved/defaulted/rejected semantics to configs/resolved-profile.json, and verifies the local Git revision and required source symbols before writing the launch plan.

For AI Toolkit and SimpleTuner, add a source-verified backend_options.inference_validation contract based on assets/configs/inference-validation.example.json. Do not invent a host CLI. DiffSynth gets a built-in reference-ablation validator. If DiffSynth uses a DeCFG preset adapter, set distillation_guard=diffsynth_decfg_adapter, provide preset_lora_path, and keep preset_lora_model=dit.

Gate: dry-run/config parser succeeds; model variant, dataset schema, LoRA target modules, audio mode, reference mode, shifts, adapter/CFG/drift guard, preview settings, and output format are explicit. Never invent a CLI flag from memory.

8. Run a bounded smoke test

Only after execution is authorized, cache a small representative subset and run 50–300 steps. Save frequent checkpoints and fixed-seed base/LoRA-off/LoRA-on samples. Record separate video loss and audio loss when audio is in scope, GPU/RAM/disk peaks, throughput, NaN/OOM events, and sample paths in reports/08-smoke-test.json.

Execute the exact locked argv without a shell:

python scripts/run_backend.py --root <project> --authorization-token I_AUTHORIZE_H3_GPU_TRAINING

Immediately before execution the runner revalidates the launch-plan hash, resolved profile, backend Git SHA, and source-file hashes. It writes pass only after a zero exit code, finite observed loss, no OOM/NaN marker, and at least one structurally valid SafeTensors checkpoint. DiffSynth currently exposes a combined H3 video+audio loss; record that observability limit and require held-out audio evaluation before final audio_trained claims.

Gate: decision=pass; losses are finite; every claimed modality contributes; samples show directional learning without major base drift; resume works; memory headroom is acceptable. Fix the cause and repeat with a new experiment ID rather than silently changing the lock.

9. Train, monitor, and select evidence-first

Run the approved schedule with high-frequency checkpoints. Evaluate multiple checkpoints on fixed held-out prompts and seeds. Fill reports/checkpoint-metrics.csv, then run:

python scripts/score_checkpoints.py --input <project>/reports/checkpoint-metrics.csv --route <route> --output <project>/reports/09-checkpoint-selection.json

Write reports/09-training-summary.json with actual settings, interruptions, drift, and the chosen—not necessarily final—checkpoint.

Gate: the selected checkpoint wins on the route scorecard and passes base-preservation/artifact floors. Stop early when late checkpoints degrade; more steps are not inherently better.

10. Prove inference compatibility and package

Test the chosen LoRA in its declared H3 host with matching base variant, rank/alpha semantics, target modules, LoRA strength range, shifts, scheduler, and any special node/pass. For Ref2VA, test 0/1/multiple references and order changes. For AV, test speech, non-speech, silence, and event sync.

For DiffSynth the backend preparer generates configs/validate-diffsynth.py and a fixed-seed 0/1/all/reversed-reference matrix. AI Toolkit and SimpleTuner use the locked inference argv template. After choosing the checkpoint run:

python scripts/validate_lora.py --weights <checkpoint.safetensors> --project-root <project> --authorization-token I_AUTHORIZE_H3_GPU_INFERENCE

Without --project-root, validate_lora.py performs structural SafeTensors and LoRA A/B-pair validation only; that is not inference proof. With a project root, each generated sample must also pass ffprobe stream, duration, fps, frame-grid, dimensions, sample-rate, and channel checks declared by the launch plan.

Create exports/delivery-manifest.json listing weights, hashes, trigger, recommended strength, base/trainer revisions, conditions actually trained, limitations, sample matrix, license/NOTICE, dataset provenance summary, inference workflow, and reproduction command.

Gate: reports/10-lora-validation.json proves structural validity and successful inference samples; a clean environment can load the package; claims do not exceed tests; no dataset media, secrets, optimizer state, or private metadata leaks into the export.

Non-negotiable H3 invariants

• H3’s public training convention uses x_sigma=(1-sigma)*x0+sigma*noise, model timestep t=1-sigma, and data-ward target x0-noise. Verify backend sign rather than assuming a generic diffusion target.
• Map one base noise level to modality-specific schedules; do not sample unrelated video/audio timesteps for a synchronized target.
• Missing target audio must be masked or rejected, never silently trained as long silence. Audio-only rows must mask video loss.
• Record video and audio losses separately when the backend exposes them. If it exposes only a combined AV loss, record that observability limit and require held-out audio/sync inference before claiming audio learning; a healthy total loss alone is insufficient.
• Public inference commonly uses video shift 12 and audio shift 3, while some training recipes use other empirical shifts. Save the exact training and inference values; do not silently normalize them.
• H3 checkpoints are CFG-distilled. Use and record a supported preservation strategy such as training adapter plus contrastive/CFG-augmented loss, frozen-base drift regularization, or a tested backend-specific equivalent. Always keep early checkpoints.
• Default to excluding AdaLN from ordinary LoRA when cross-compatibility matters. Train or approximate AdaLN only for an explicit, validated backend/model objective.
• H3 video is fixed at 24 fps in the official pipeline. Treat frame-grid and dimension mismatches as explicit preprocessing decisions, not invisible coercions.

Read references/h3-math-and-guards.md before changing any of these invariants.

Do not add rsLoRA, LoRA+, DoRA, PiSSA, LoftQ, Min-SNR, sparse attention, or DMD merely because it is newer. Read references/algorithm-options.md; treat each as a one-change-only experiment and require backend implementation plus inference round-trip evidence.

Completion response

Report the current step and gate status, not merely “training completed.” On final delivery include:

• route and exact capability claims;
• dataset counts by modality and split;
• base/trainer revisions and experiment-lock hash;
• selected checkpoint and why it beat the final checkpoint if applicable;
• evaluated inference modes, recommended strength, and known failures;
• local paths to the project, report, configuration, weights, and delivery manifest;
• any action still requiring user authorization.

The skill is complete only after Step 10 passes or the user explicitly requested an earlier stopping point.