lingyu9495-source/lowvram-ai-video-comfyui

Local AI Video Generation (ComfyUI + Wan / LTX)

Baseline machine: RTX 4060 Laptop 8GB VRAM / 32GB RAM / Win11. Positioning: Local output of low-resolution, a few seconds of anime short videos (sufficient for self-media). Don't treat 8GB as 16GB.

When to Use

When the user says "run a video model / text-to-video / image-to-video", or when you need to hand over the local video production environment to an executor (team member, etc.) for further experimentation.

Build Understanding First (Avoid Wrong Choices)

• Text-to-Video (T2V) and Image-to-Video (I2V) are two separate models; they cannot replace each other:
  • Wan2.1-1.3B only supports T2V; Wan's I2V is only available as 14B (8GB can't run it).
  • I2V that 8GB can run = LTX-2B fp8 (requires t5xxl encoder additionally).
• Long video / character consistency: Pure T2V cannot achieve "continuous segments with the same protagonist"; the correct approach is I2V using the same character image as the first frame, generating segments one by one and then concatenating them. Hard concatenating N segments of T2V will inevitably cause character drift.
• Ollama and ComfyUI models are not interchangeable: Ollama is a language model (for writing prompts / scenarios), ComfyUI is a diffusion model (for generating videos). They work cooperatively, not as replacements.

Deployment Order (Do in sequence; skipping will cause issues)

1) Install ComfyUI Portable Version (place on D drive, don't occupy C drive)

• Get ComfyUI_windows_portable_nvidia.7z from Comfy-Org/ComfyUI GitHub releases (includes Python + torch + CUDA).
• ⚠️ Under bash, 7z x -o"/d/..." will corrupt MSYS paths, and the output won't land in the target directory (extraction reports success but the directory is empty). → Use Python subprocess with native Windows paths to call 7z.

2) Upgrade ComfyUI to the latest version (older versions lack Wan/LTX nodes)

• Direct GitHub connection often fails with Connection was reset → First configure the mirror: git config --global url."https://ghfast.top/https://github.com/".insteadOf "https://github.com/", then git pull --ff-only origin master.

3) Upgrade Python dependencies: python_embeded\python.exe -s -m pip install -r ComfyUI\requirements.txt (new version will add comfy-kitchen etc.).

4) torch must match comfy-kitchen (troubleshoot step by step based on symptoms):

Symptom	Root Cause	Action
module 'torch.library' has no attribute 'custom_op'	torch < 2.4	Upgrade torch
infer_schema ... unsupported type list[int]	torch and comfy-kitchen mismatch	Upgrade to cu130 torch (2.14+)
cudaErrorNotSupported / likely using older driver	Driver CUDA version insufficient	Upgrade NVIDIA driver

• First check driver support: nvidia-smi to see CUDA Version; if CUDA 13 is needed, upgrade the driver.
• Get the latest driver: NVIDIA driver query API (AjaxDriverService, for RTX 40 series laptop psid=138/pfid=1005) to obtain DownloadURL; download requires browser UA + Referer, otherwise 403. Silent install with -s -noreboot; after installation, confirm with nvidia-smi, usually no reboot needed for torch to recognize the GPU.

5) Models must be in ComfyUI standard format (most common pitfall)

• From Comfy-Org/Wan_2.1_ComfyUI_repackaged's split_files/ safetensors:
  • diffusion_models/wan2.1_t2v_1.3B_fp16.safetensors (~2.8GB)
  • text_encoders/umt5_xxl_fp8_e4m3fn_scaled.safetensors (~6.7GB)
  • vae/wan_2.1_vae.safetensors (~0.25GB)
• ⚠️ Using Wan's official raw .pth files (e.g., models_t5_umt5-xxl-enc-bf16.pth) → errors with Cannot copy out of meta tensor; no data! (during CLIPTextEncode). This is a format/name mismatch, not a VRAM issue—don't adjust VRAM parameters.
• Domestic download: HuggingFace direct connection may fail → use hf-mirror.com; curl -C - --retry 999 --retry-all-errors for resumable download.
• Destination: ComfyUI\models\{diffusion_models,text_encoders,vae}\.

6) Install ffmpeg (required for video encoding): python_embeded\python.exe -s -m pip install imageio-ffmpeg imageio (includes static ffmpeg). Without it, SaveVideo step will report [Errno 22].

7) Start the service

python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --disable-async-offload > log.txt 2>&1

• --disable-async-offload: For 8GB, prevents text encoder weights from staying on the meta device.
• ⚠️⚠️ Never use | head / | tee piping to start such long-running services: tqdm progress bar writing to a closed pipe → [Errno 22] Invalid argument directly interrupts KSampler, looking like a model error but actually a startup method problem. Use > log 2>&1 for redirection. This pitfall applies to any long-running process/gateway—a closed pipe can kill the process.
• Check readiness: netstat -ano | grep :8188 shows LISTENING.

Run Text-to-Video (T2V)

• Node chain: follow templates/wan21_t2v_workflow.json exactly: CLIPLoader(type=wan) + VAELoader + UNETLoader + EmptyHunyuanLatentVideo + CLIPTextEncode(positive/negative) + KSampler + VAEDecode + CreateVideo + SaveVideo
  • Use EmptyHunyuanLatentVideo for video latent; output must go through CreateVideo → SaveVideo.
  • ⚠️ VAEDecode output is IMAGE; connecting directly to SaveVideo will report received_type(IMAGE) mismatch input_type(VIDEO).
• Submit: POST http://127.0.0.1:8188/prompt, body {"prompt": {node_id: {class_type, inputs}}}; returns node_errors:{} on success.
• Check progress: GET /history/<prompt_id> to see status.status_str (running/success/error); on error, read messages for execution_error (contains node_id/node_type/exception_message).
• Tested and usable parameters for 8GB: 832×480, length 33 (~2 seconds), steps 15-20, cfg 5.5-6.0, fps 16. Exceeding this will offload to memory, making it extremely slow.
• Outputs in ComfyUI\output\*.mp4; use imageio_ffmpeg.get_ffmpeg_exe() to call ffmpeg to verify duration/resolution.

Run Image-to-Video (I2V)

• Use LTX-2B: ltxv-2b-0.9.8-distilled-fp8.safetensors (~4.5GB) → models\diffusion_models\; text encoder uses t5xxl: t5xxl_fp8_e4m3fn.safetensors (~4.9GB, from comfyanonymous/flux_text_encoders) → models\text_encoders\.
• ⚠️ LTX uses t5xxl, Wan uses umt5; they are not interchangeable; both need to be downloaded.
• Consistency strategy: Use a fixed character image as the first frame for each segment, or use IPAdapter/InstantID to inject the character.

Preparation Before Running (8GB VRAM)

• Close VRAM-consuming software like Chrome (system/desktop typically use ~1.2GB, Chrome can use ~1.9GB).
• Run tasks sequentially; don't run multiple generations concurrently.

Deliver to Executors

Handover document structure (follow this): Environment path → Startup command (with parameters) → Model list (filename + directory + size) → Reusable workflow JSON → 8GB parameter tips → Troubleshooting quick reference → Next steps.

References

• templates/wan21_t2v_workflow.json — Tested and working Wan2.1-1.3B text-to-video workflow (ComfyUI API format, can be POSTed directly)

---

About the Author

Nine-rank Jinli e ｜ Encapsulating the pitfalls I've stepped into into a "ready-to-run" skill, not a textbook.

This skill is the version I use myself; every pitfall mentioned here is one I actually encountered. If you get stuck during use, want advanced techniques, or have other scenarios you'd like me to package into a skill, just add me on WeChat directly, mention "SkillHub" for priority approval:

Scan the QR code to add the author on WeChat · Mention "SkillHub" for priority approval