Comfyui

ComfyUI

Generate images, video, audio, and 3D content through ComfyUI using the official comfy-cli for setup/lifecycle and direct REST/WebSocket API for workflow execution.

What's in this skill

Reference docs (references/):

• official-cli.md — every comfy ... command, with flags
• rest-api.md — REST + WebSocket endpoints (local + cloud), payload schemas
• workflow-format.md — API-format JSON, common node types, param mapping

Scripts (scripts/):

| Script | Purpose | |--------|---------| | _common.py | Shared HTTP, cloud routing, node catalogs (don't run directly) | | hardware_check.py | Probe GPU/VRAM/disk → recommend local vs Comfy Cloud | | comfyui_setup.sh | Hardware check + comfy-cli + ComfyUI install + launch + verify | | extract_schema.py | Read a workflow → list controllable params + model deps | | check_deps.py | Check workflow against running server → list missing nodes/models | | auto_fix_deps.py | Run checkdeps then comfy node install / comfy model download | | run_workflow.py | Inject params, submit, monitor, download outputs (HTTP or WS) | | run_batch.py | Submit a workflow N times with sweeps, parallel up to your tier | | ws_monitor.py | Real-time WebSocket viewer for executing jobs (live progress) | | health_check.py | Verification checklist runner — comfy-cli + server + models + smoke test | | fetch_logs.py | Pull traceback / status messages for a given promptid |

Example workflows (workflows/): SD 1.5, SDXL, Flux Dev, SDXL img2img, SDXL inpaint, ESRGAN upscale, AnimateDiff video, Wan T2V. See workflows/README.md.

When to Use

• User asks to generate images with Stable Diffusion, SDXL, Flux, SD3, etc.
• User wants to run a specific ComfyUI workflow file
• User wants to chain generative steps (txt2img → upscale → face restore)
• User needs ControlNet, inpainting, img2img, or other advanced pipelines
• User asks to manage ComfyUI queue, check models, or install custom nodes
• User wants video/audio/3D generation via AnimateDiff, Hunyuan, Wan, AudioCraft, etc.

Architecture: Two Layers

┌─────────────────────────────────────────────────────┐
│ Layer 1: comfy-cli (official lifecycle tool)        │
│   Setup, server lifecycle, custom nodes, models     │
│   → comfy install / launch / stop / node / model    │
└─────────────────────────┬───────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────┐
│ Layer 2: REST/WebSocket API + skill scripts         │
│   Workflow execution, param injection, monitoring   │
│   POST /api/prompt, GET /api/view, WS /ws           │
│   → run_workflow.py, run_batch.py, ws_monitor.py    │
└─────────────────────────────────────────────────────┘

Why two layers? The official CLI is excellent for installation and server management but has minimal workflow execution support. The REST/WS API fills that gap — the scripts handle param injection, execution monitoring, and output download that the CLI doesn't do.

Quick Start

Detect environment

# What's available?
command -v comfy >/dev/null 2>&1 && echo "comfy-cli: installed"
curl -s http://127.0.0.1:8188/system_stats 2>/dev/null && echo "server: running"

# Can this machine run ComfyUI locally? (GPU/VRAM/disk check)
python3 scripts/hardware_check.py

If nothing is installed, see Setup & Onboarding below — but always run the hardware check first.

One-line health check

python3 scripts/health_check.py
# → JSON: comfy_cli on PATH? server reachable? at least one checkpoint? smoke-test passes?

Core Workflow

Step 1: Get a workflow JSON in API format

Workflows must be in API format (each node has class_type). They come from:

• ComfyUI web UI → Workflow → Export (API) (newer UI) or

the legacy "Save (API Format)" button (older UI)

• This skill's workflows/ directory (ready-to-run examples)
• Community downloads (civitai, Reddit, Discord) — usually editor format,

must be loaded into ComfyUI then re-exported

Editor format (top-level nodes and links arrays) is not directly executable. The scripts detect this and tell you to re-export.

Step 2: See what's controllable

python3 scripts/extract_schema.py workflow_api.json --summary-only
# → {"parameter_count": 12, "has_negative_prompt": true, "has_seed": true, ...}

python3 scripts/extract_schema.py workflow_api.json
# → full schema with parameters, model deps, embedding refs

Step 3: Run with parameters

# Local (defaults to http://127.0.0.1:8188)
python3 scripts/run_workflow.py \
  --workflow workflow_api.json \
  --args '{"prompt": "a beautiful sunset over mountains", "seed": -1, "steps": 30}' \
  --output-dir ./outputs

# Cloud (export API key once; uses correct /api routing automatically)
export COMFY_CLOUD_API_KEY="comfyui-..."
python3 scripts/run_workflow.py \
  --workflow workflow_api.json \
  --args '{"prompt": "..."}' \
  --host https://cloud.comfy.org \
  --output-dir ./outputs

# Real-time progress via WebSocket (requires `pip install websocket-client`)
python3 scripts/run_workflow.py \
  --workflow flux_dev.json \
  --args '{"prompt": "..."}' \
  --ws

# img2img / inpaint: pass --input-image to upload + reference automatically
python3 scripts/run_workflow.py \
  --workflow sdxl_img2img.json \
  --input-image image=./photo.png \
  --args '{"prompt": "make it watercolor", "denoise": 0.6}'

# Batch / sweep: 8 random seeds, parallel up to cloud tier limit
python3 scripts/run_batch.py \
  --workflow sdxl.json \
  --args '{"prompt": "abstract"}' \
  --count 8 --randomize-seed --parallel 3 \
  --output-dir ./outputs/batch

-1 for seed (or omitting it with --randomize-seed) generates a fresh random seed per run.

Step 4: Present results

The scripts emit JSON to stdout describing every output file:

{
  "status": "success",
  "prompt_id": "abc-123",
  "outputs": [
    {"file": "./outputs/sdxl_00001_.png", "node_id": "9",
     "type": "image", "filename": "sdxl_00001_.png"}
  ]
}

Decision Tree

| User says | Tool | Command | |-----------|------|---------| | Lifecycle (use comfy-cli) | | | | "install ComfyUI" | comfy-cli | bash scripts/comfyui_setup.sh | | "start ComfyUI" | comfy-cli | comfy launch --background | | "stop ComfyUI" | comfy-cli | comfy stop | | "install X node" | comfy-cli | comfy node install | | "download X model" | comfy-cli | comfy model download --url --relative-path models/checkpoints | | "list installed models" | comfy-cli | comfy model list | | "list installed nodes" | comfy-cli | comfy node show installed | | Execution (use scripts) | | | | "is everything ready?" | script | health_check.py (optionally with --workflow X --smoke-test) | | "what can I change in this workflow?" | script | extract_schema.py W.json | | "check if W's deps are met" | script | check_deps.py W.json | | "fix missing deps" | script | auto_fix_deps.py W.json | | "generate an image" | script | run_workflow.py --workflow W --args '{...}' | | "use this image" (img2img) | script | run_workflow.py --input-image image=./x.png ... | | "8 variations with random seeds" | script | run_batch.py --count 8 --randomize-seed ... | | "show me live progress" | script | ws_monitor.py --prompt-id | | "fetch the error from job X" | script | fetch_logs.py | | Direct REST | | | | "what's in the queue?" | REST | curl http://HOST:8188/queue (local) or --host https://cloud.comfy.org | | "cancel that" | REST | curl -X POST http://HOST:8188/interrupt | | "free GPU memory" | REST | curl -X POST http://HOST:8188/free |

Setup & Onboarding

When a user asks to set up ComfyUI, the FIRST thing to do is ask whether they want Comfy Cloud (hosted, zero install, API key) or Local (install ComfyUI on their machine). Don't start running install commands or hardware checks until they've answered.

Official docs: https://docs.comfy.org/installation CLI docs: https://docs.comfy.org/comfy-cli/getting-started Cloud docs: https://docs.comfy.org/get_started/cloud Cloud API: https://docs.comfy.org/development/cloud/overview

Step 0: Ask Local vs Cloud (ALWAYS FIRST)

Suggested script:

> "Do you want to run ComfyUI locally on your machine, or use Comfy Cloud? > > - Comfy Cloud — hosted on RTX 6000 Pro GPUs, all common models pre-installed, > zero setup. Requires an API key (paid subscription required to actually run > workflows; free tier is read-only). Best if you don't have a capable GPU. > - Local — free, but your machine MUST meet the hardware requirements: > - NVIDIA GPU with ≥6 GB VRAM (≥8 GB for SDXL, ≥12 GB for Flux/video), OR > - AMD GPU with ROCm support (Linux), OR > - Apple Silicon Mac (M1+) with ≥16 GB unified memory (≥32 GB recommended). > - Intel Macs and machines with no GPU will NOT work — use Cloud instead. > > Which would you like?"

Routing:

• Cloud → skip to Path A.
• Local → run hardware check first, then pick a path from Paths B–E based on the verdict.
• Unsure → run the hardware check and let the verdict decide.

Step 1: Verify Hardware (ONLY if user chose local)

python3 scripts/hardware_check.py --json
# Optional: also probe `torch` for actual CUDA/MPS:
python3 scripts/hardware_check.py --json --check-pytorch

| Verdict | Meaning | Action | |------------|---------------------------------------------------------------|--------| | ok | ≥8 GB VRAM (discrete) OR ≥32 GB unified (Apple Silicon) | Local install — use comfy_cli_flag from report | | marginal | SD1.5 works; SDXL tight; Flux/video unlikely | Local OK for light workflows, else Path A (Cloud) | | cloud | No usable GPU, is renamed to /experiment/models/` on cloud (the scripts route automatically).

• clientId in WebSocket is currently ignored — all connections for a

user receive the same broadcast. Filter by prompt_id client-side.

• subfolder is accepted on uploads but ignored — cloud has a flat namespace.
• Concurrent jobs: Free/Standard: 1, Creator: 3, Pro: 5. Extras queue

automatically. Use run_batch.py --parallel N to saturate your tier.

Queue & System Management

# Local
curl -s http://127.0.0.1:8188/queue | python3 -m json.tool
curl -X POST http://127.0.0.1:8188/queue -d '{"clear": true}'    # cancel pending
curl -X POST http://127.0.0.1:8188/interrupt                      # cancel running
curl -X POST http://127.0.0.1:8188/free \
  -H "Content-Type: application/json" \
  -d '{"unload_models": true, "free_memory": true}'

# Cloud — same paths under /api/, plus:
python3 scripts/fetch_logs.py --tail-queue --host https://cloud.comfy.org

Pitfalls

1. API format required — every script and the /api/prompt endpoint expect

API-format workflow JSON. The scripts detect editor format (top-level nodes and links arrays) and tell you to re-export via "Workflow → Export (API)" (newer UI) or "Save (API Format)" (older UI).

1. Server must be running — all execution requires a live server.

comfy launch --background starts one. Verify with curl http://127.0.0.1:8188/system_stats.

1. Model names are exact — case-sensitive, includes file extension.

check_deps.py does fuzzy matching (with/without extension and folder prefix), but the workflow itself must use the canonical name. Use comfy model list to discover what's installed.

1. Missing custom nodes — "class_type not found" means a required node

isn't installed. check_deps.py reports which package to install; auto_fix_deps.py runs the install for you.

1. Working directory — comfy-cli auto-detects the ComfyUI workspace.

If commands fail with "no workspace found", use comfy --workspace /path/to/ComfyUI or comfy set-default /path/to/ComfyUI.

1. Cloud free-tier API limits — /api/prompt, /api/view, /api/upload/*,

/api/object_info all return 403 on free accounts. health_check.py and check_deps.py handle this gracefully and surface a clear message.

1. Timeout for video/audio workflows — auto-detected when an output node

is VHS_VideoCombine, SaveVideo, etc.; the default jumps from 300 s to 900 s. Override explicitly with --timeout 1800.

1. Path traversal in output filenames — server-supplied filenames are

passed through safe_path_join to refuse anything escaping --output-dir. Keep this protection on — workflows with custom save nodes can produce arbitrary paths.

1. Workflow JSON is arbitrary code — custom nodes run Python, so

submitting an unknown workflow has the same trust profile as eval. Inspect workflows from untrusted sources before running.

1. Auto-randomized seed — pass seed: -1 in --args (or use

--randomize-seed and omit the seed) to get a fresh seed per run. The actual seed is logged to stderr.

1. tracking prompt — first run of comfy may prompt for analytics.

Use comfy --skip-prompt tracking disable to skip non-interactively. comfyui_setup.sh does this for you.

Verification Checklist

Use python3 scripts/health_check.py to run the whole list at once. Manual:

• [ ] hardware_check.py verdict is ok OR the user explicitly chose Comfy Cloud
• [ ] comfy --version works (or uvx --from comfy-cli comfy --help)
• [ ] curl http://HOST:PORT/system_stats returns JSON
• [ ] comfy model list shows at least one checkpoint (local) OR

/api/experiment/models/checkpoints returns models (cloud)

• [ ] Workflow JSON is in API format
• [ ] check_deps.py reports is_ready: true (or only node_check_skipped

on cloud free tier)

• [ ] Test run with a small workflow completes; outputs land in --output-dir

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: kevinnft
• Source: kevinnft/ai-agent-skills
• License: MIT
• Homepage: https://kevinnft.github.io/ai-agent-skills/

Install and usage instructions live in the source repository linked above.