Ai Video Producer

AI Video Producer

Create professional AI-generated videos through a structured, iterative workflow using local models.

Prerequisites & Auto-Setup

Requires: WAN 2.1 + Qwen Image Edit via ComfyUI (GGUF quantization, ~40GB models)

Setup Commands

python {baseDir}/scripts/setup_comfyui.py         # Full setup (first time)
python {baseDir}/scripts/setup_comfyui.py --check # Verify setup
python {baseDir}/scripts/setup_comfyui.py --start # Start server before generating

System Requirements

| Component | Minimum | |-----------|---------| | GPU VRAM | 10GB | | RAM | 16GB | | Storage | 40GB free |

See README.md for detailed setup instructions.

MANDATORY WORKFLOW REQUIREMENTS

YOU MUST FOLLOW THESE RULES:

1. ALWAYS use TodoWrite at the start to create a task list for the entire workflow
2. ALWAYS ask about Approval Mode at the very start (see below)
3. NEVER skip phases - complete each phase in order before proceeding
4. ALWAYS create required files - philosophy.md, style.json, scene-breakdown.md, and pipeline.json are REQUIRED
5. ALWAYS break videos into multiple scenes - minimum 2 scenes for any video over 5 seconds
6. NEVER generate without a complete pipeline.json - plan ALL prompts first, execute second
7. ALWAYS use execute_pipeline.py for generation - deterministic execution, no ad-hoc commands
8. ALWAYS review generated outputs using VLM - view images after each stage, assess quality

Approval Mode Selection (FIRST STEP)

At the very beginning of the workflow, ask the user to choose an approval mode:

Use AskUserQuestion with these options:

• "Manual approval" - User approves each phase before proceeding (philosophy, scenes, pipeline, assets, keyframes, videos)
• "Automatic approval" - LLM proceeds automatically, user only reviews final output

| Mode | User Interaction | Best For | |------|------------------|----------| | Manual | Checkpoint at each phase | First-time projects, precise control, learning the workflow | | Automatic | Only final review | Trusted workflow, quick generation, batch production |

Store the selected mode and apply it to all checkpoints throughout the workflow.

Pipeline-Based Architecture

This skill uses a two-phase approach:

Phase A: Planning (LLM-Driven)

• LLM creates philosophy.md, style.json, scene-breakdown.md
• LLM generates ALL prompts and stores in pipeline.json
• User reviews and approves the complete plan before any generation

Phase B: Execution (Programmatic)

• execute_pipeline.py reads pipeline.json and executes deterministically
• LLM reviews outputs using VLM capability after each stage
• User approves or requests regeneration

Benefits:

• All prompts visible before ANY generation starts
• Deterministic execution - no LLM deviation during generation
• Reproducible - same pipeline.json = same commands executed
• Traceable - status tracking in pipeline.json

Pipeline Mode: Scene/Segment v3.0

Proper hierarchical structure distinguishing Scenes (narrative/cinematographic units) from Segments (5-second technical chunks).

Key Concepts:

• Scene: A continuous shot from a single camera perspective (e.g., "woman at cafe table", "phone screen close-up")
• Segment: A 5-second video chunk within a scene (due to model limitations)
• Transition: How scenes connect ("cut", "continuous", "fade", "dissolve")

Scene 1 (cut)           Scene 2 (continuous)       Scene 3 (fade)
┌─────────────────┐     ┌─────────────────┐       ┌─────────────────┐
│ KF (generated)  │     │ KF (extracted)  │       │ KF (generated)  │
│ ┌─────────────┐ │     │ ┌─────────────┐ │       │ ┌─────────────┐ │
│ │ Segment A   │ │     │ │ Segment A   │ │       │ │ Segment A   │ │
│ └─────────────┘ │     │ └─────────────┘ │       │ └─────────────┘ │
│ ┌─────────────┐ │     └─────────────────┘       └─────────────────┘
│ │ Segment B   │ │             │                         │
│ └─────────────┘ │             │ continuous              │ fade
└─────────────────┘             ▼                         ▼
        │ cut          ─────────────────          ─────────────────
        ▼              Final merged video with transitions

Pros:

• Semantic clarity (scenes = narrative units, segments = technical units)
• Scene-level keyframes (generated for cuts, extracted for continuous)
• Automatic video merging with transitions (cut, fade, dissolve)
• Hierarchical status tracking

Cons: More complex schema

When to use each transition: | Transition | Use When | Keyframe | |------------|----------|----------| | cut | Camera angle/location changes | Generated (new) | | continuous | Same shot continues (landscape only) | Extracted (from previous scene) | | fade | Time skip, dramatic moment | Generated (new) | | dissolve | Smooth transition between related scenes | Generated (new) |

CRITICAL: Character Scenes and Continuous Transitions

When a scene contains a character (even partially visible, like hands or clothing), do NOT use "extracted" keyframes. Extracted keyframes lose character identity anchoring and cause visual drift (e.g., clothing color changes, style inconsistency).

| Scene Type | Transition | Keyframe Type | Why | |------------|------------|---------------|-----| | Landscape (no characters) | continuous | extracted | OK - no character identity to preserve | | Character visible | continuous | generated | REQUIRED - re-anchor character identity | | Character visible | cut | generated | Standard - new camera angle |

Rule: If ANY part of a character is visible (hands, clothing, body), use "type": "generated" with character references.

Standard Checkpoint Format (ALL PHASES)

Checkpoint behavior depends on the selected Approval Mode:

Manual Approval Mode (default)

1. Show the output to user (file path or display content)
2. Ask for approval using AskUserQuestion:

• "Approve" - Proceed to next step
• User can select "Other" to specify what needs to be changed

1. If user does not approve:

• User specifies what to change
• Make the requested adjustments
• Show updated result → Ask again → Repeat until approved

1. Do NOT proceed to next phase until approved

Automatic Approval Mode

1. Show the output to user (file path or display content)
2. LLM reviews the output using VLM capability (for images/videos)
3. If LLM assessment is positive → Proceed automatically
4. If LLM detects issues → Fix and regenerate before proceeding
5. User only reviews final output at the end

Workflow Phases (MUST COMPLETE IN ORDER)

| Phase | LLM Actions | Required Outputs | Manual Mode | Auto Mode | |-------|-------------|------------------|-------------|-----------| | 1. Production Philosophy | Create visual identity & style | philosophy.md, style.json | User approval | LLM proceeds | | 2. Scene Breakdown | Plan scenes with segments and transitions | scene-breakdown.md | User approval | LLM proceeds | | 3. Pipeline Generation | Generate prompts with v3.0 schema | pipeline.json | User approval | LLM proceeds | | 4. Asset Execution | Run execute_pipeline.py --stage assets | assets/ folder | VLM review + user approval | VLM review + proceed | | 5. Scene Keyframes | Run execute_pipeline.py --stage scene_keyframes | keyframes/scene-*.png | VLM review + user approval | VLM review + proceed | | 6. Scene Execution | Run execute_pipeline.py --stage scenes | scene-*/merged.mp4 + final/video.mp4 | User approval | LLM proceeds | | 7. Review & Iterate | Handle regeneration requests | Refinements | User signs off | User reviews final |

---

Phase 1: Production Philosophy (REQUIRED)

DO NOT PROCEED TO PHASE 2 UNTIL BOTH FILES EXIST:

• {output_dir}/philosophy.md
• {output_dir}/style.json

Step 1.1: Create philosophy.md

Create this file with ALL sections filled in:

# Production Philosophy: [Project Name]

## Visual Identity
- **Art Style**: [e.g., cinematic realistic, stylized animation, painterly]
- **Color Palette**: [primary colors, mood, temperature]
- **Lighting**: [natural, dramatic, soft, high-contrast]
- **Composition**: [rule of thirds, centered, dynamic angles]

## Motion Language
- **Movement Quality**: [smooth/fluid, dynamic/energetic, subtle/minimal]
- **Pacing**: [fast cuts, slow contemplative, rhythmic]
- **Camera Style**: [static, tracking, handheld, cinematic sweeps]

## Subject Consistency
- **Characters/Products**: [detailed descriptions for consistency]
- **Environment**: [setting details that persist across scenes]
- **Props/Elements**: [recurring visual elements]

## Constraints
- **Avoid**: [unwanted elements, styles, or actions]
- **Maintain**: [elements that must stay consistent]

Step 1.2: Create style.json

Create this file for programmatic use with generation scripts:

{
  "project_name": "Project Name Here",
  "visual_style": {
    "art_style": "description",
    "color_palette": "description",
    "lighting": "description",
    "composition": "description"
  },
  "motion_language": {
    "movement_quality": "description",
    "pacing": "description",
    "camera_style": "description"
  },
  "subject_consistency": {
    "main_subject": "detailed description",
    "environment": "detailed description"
  },
  "constraints": {
    "avoid": ["list", "of", "things"],
    "maintain": ["list", "of", "things"]
  }
}

Step 1.3: CHECKPOINT - Get User Approval

1. Inform user that philosophy.md and style.json have been created
2. Use AskUserQuestion:

• "Approve" - Proceed to scene breakdown
• User selects "Other" to specify changes

If user requests changes → make adjustments → ask again → repeat until approved

---

Phase 2: Scene Breakdown (REQUIRED)

DO NOT PROCEED TO PHASE 3 UNTIL scene-breakdown.md EXISTS AND USER APPROVES

Step 2.1: Analyze Video Requirements

Before creating scenes, determine:

• Total video duration needed
• Number of scenes required (minimum 2 for videos > 5 seconds)
• Key story beats or content moments
• Transitions between scenes

Step 2.2: Create scene-breakdown.md

MANDATORY FORMAT - v3.0 Scene/Segment structure:

# Scene Breakdown: [Project Name]

## Overview
- **Total Duration**: [X seconds]
- **Number of Scenes**: [N]
- **Video Type**: [promotional/narrative/educational/etc.]
- **Pipeline Mode**: Scene/Segment v3.0

---

## Scene Overview

| Scene | Description | Camera | Transition | Duration |
|-------|-------------|--------|------------|----------|
| 1 | [Brief description] | [Camera type] | - | 5s |
| 2 | [Brief description] | [Camera type] | cut/continuous | 5s |

---

## Scenes

### Scene N: [Title]

**Type**: character | landscape
**Duration**: 5 seconds
**Camera**: [static/tracking/pan/zoom]
**Purpose**: [What this scene communicates]

**First Keyframe**: Generated (character scenes) OR Extracted (continuous landscape only)
- Characters: [list IDs - REQUIRED if visible]
- Background: [background ID]

**Segments**:
| ID | Motion | Duration |
|----|--------|----------|
| seg-Na | [Motion description] | 5s |

**Transition to Next**: cut | continuous | fade

[Repeat for all scenes]

Scene Type Explanation:

• character: Scenes with characters - MUST include character references in keyframe
• landscape: Scenes without characters - can use extracted keyframes for continuous transitions

Scene Count Guidelines

WAN generates 5-second clips (81 frames at 16fps)

| Total Video Length | Minimum Scenes | Recommended Scenes | |--------------------|----------------|-------------------| | 1-5 seconds | 1 | 1 | | 6-10 seconds | 2 | 2 | | 11-15 seconds | 3 | 3 | | 16-20 seconds | 4 | 4 | | 20+ seconds | 5+ | Break into 5s beats |

Step 2.3: CHECKPOINT - Get User Approval

1. Inform user that scene-breakdown.md has been created with [N] scenes
2. Use AskUserQuestion:

• "Approve" - Proceed to asset generation
• User selects "Other" to specify changes

If user requests changes → make adjustments → ask again → repeat until approved

---

Phase 2.5: Asset Generation (REQUIRED)

DO NOT PROCEED TO PHASE 3 UNTIL assets.json EXISTS AND USER APPROVES

This phase creates reusable assets that maintain consistency across all scenes.

Step 2.5.1: Analyze Required Assets

Review scene-breakdown.md and identify all unique:

• Characters (each character that appears in scenes)
• Backgrounds (each unique location/environment)
• Styles (the visual style to apply consistently)
• Objects (any recurring props or items)

Step 2.5.2: Create assets.json

MANDATORY FORMAT:

{
  "characters": {
    "samurai": {
      "description": "Feudal Japanese warrior, red armor, stern expression, dark hair",
      "identity_ref": "assets/characters/samurai.png"
    }
  },
  "backgrounds": {
    "temple_courtyard": {
      "description": "Ancient temple with cherry blossoms, stone paths, morning light",
      "ref_image": "assets/backgrounds/temple_courtyard.png"
    }
  },
  "styles": {
    "ghibli": {
      "description": "Studio Ghibli anime aesthetic, soft colors, painterly",
      "ref_image": "assets/styles/ghibli.png"
    }
  },
  "objects": {
    "katana": {
      "description": "Traditional Japanese sword with black sheath",
      "ref_image": "assets/objects/katana.png"
    }
  }
}

Step 2.5.3: Asset Generation

Assets are generated via execute_pipeline.py --stage assets. This phase defines assets in assets.json for later execution.

Step 2.5.4: CHECKPOINT - Get User Approval

1. Inform user that assets.json and asset images have been created
2. Show the generated assets to user
3. Use AskUserQuestion:

• "Approve" - Proceed to keyframe generation
• User selects "Other" to specify which assets need adjustment

If user requests changes → regenerate specific assets → ask again → repeat until approved

---

Phase 3: Pipeline Generation (REQUIRED)

DO NOT PROCEED TO EXECUTION UNTIL pipeline.json EXISTS AND USER APPROVES

This phase consolidates all prompts into a single structured file that will be executed deterministically.

Step 3.1: Create pipeline.json

Based on philosophy.md, style.json, scene-breakdown.md, and assets.json, create a complete pipeline.json.

Pipeline Schema v3.0

{
  "version": "3.0",
  "project_name": "project-name",
  "metadata": {
    "created_at": "ISO timestamp",
    "philosophy_file": "philosophy.md",
    "style_file": "style.json",
    "scene_breakdown_file": "scene-breakdown.md"
  },
  "assets": {
    "characters": {
      "protagonist": {
        "prompt": "Character sheet description...",
        "output": "assets/characters/protagonist.png",
        "status": "pending"
      }
    },
    "backgrounds": {
      "location": {
        "prompt": "Background description...",
        "output": "assets/backgrounds/location.png",
        "status": "pending"
      }
    }
  },
  "scenes": [
    {
      "id": "scene-01",
      "description": "Scene description",
      "camera": "medium shot",
      "transition_from_previous": null,
      "first_keyframe": {
        "type": "generated",
        "prompt": "Keyframe description...",
        "background": "location",
        "characters": ["protagonist"],
        "output": "keyframes/scene-01-start.png",
        "status": "pending"
      },
      "segments": [
        {
          "id": "seg-01-a",
          "motion_prompt": "Motion description...",
          "output_video": "scene-01/seg-a.mp4",
          "output_keyframe": "keyframes/scene-01-seg-a-end.png",
          "status": "pending"
        }
      ],
      "output_video": "scene-01/merged.mp4",
      "status": "pending"
    }
  ],
  "final_video": {
    "output": "final/video.mp4",
    "status": "pending"
  }
}

Additional scene examples:

• Cut transition: `"transitionfromprevio

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: zysilm
• Source: zysilm/ai-video-producer-skill
• License: MIT

Install and usage instructions live in the source repository linked above.