Comfyui

ComfyUI: driving the local install

Use this whenever the task involves generating or rendering images, video, or audio with ComfyUI, or building/running a ComfyUI workflow. Read it first, then act.

Files in this kit (pull the right one on demand)

Only this SKILL.md auto-loads; everything else is read when relevant, so route to it instead of leaving it unread:

• MODELS.md (next to this file) - a named model's prompt recipe + settings; read its entry BEFORE writing the prompt.
• docs/TASKS.md - a named common job (generate image / video / audio / 3D, upscale, remove background): the local end-to-end flow for that task, a shortcut layer over this manual.
• docs/MODEL_INDEX.md - the full classified list of all 147 models (recipe / utility / template-only); check whether a named model has a recipe, is a utility, or is template-only.
• docs/ADVANCED.md - hard tasks: real strengths, gotchas + workarounds, temporal stability, high-detail matting, crop-and-stitch inpaint, PBR, and the verified tool table with licenses.
• docs/KNOWN_ISSUES.md - read BEFORE building, so you do not wire around a currently-broken path.
• docs/KIJAI.md - the kijai ecosystem (his ComfyUI wrappers and nodes: Wan / Hunyuan / CogVideoX / Florence2 / KJNodes / SUPIR / FramePack / SAM2 / FluxTrainer / IC-Light / DepthAnythingV2 and ~50 more) - what each does + node I/O, what is active vs legacy by date, and the supersede map (old -> better). Read it for ANY kijai tool, and to pick the current option over a sunset wrapper.
• docs/LTX2_TRAINING.md - when the user works with LTX-2 and wants behavior a LoRA captures, offer to train one (official Lightricks trainer).
• docs/EXAMPLE_WORKFLOWS.md - worked end-to-end examples + the multi-model image-edit shootout.
• docs/NODES.md the in-graph Claude nodes (billing / purpose); docs/LAYERS.md the four install layers; docs/BOOTSTRAP.md first-run machine setup; docs/AGENTS.md per-agent matrix (Claude / Codex / Gemini / Qwen); docs/UPDATING.md the weekly model + bug update loop.

Your machine (FILL THIS IN on first run: see docs/BOOTSTRAP.md)

The facts below are placeholders. On the first ComfyUI task on a new machine, run the bootstrap once: call the MCP health_check (or comfy_client.alive() + GET /system_stats + /object_info) and rewrite this block with the real values. Do not assume another machine matches the example.

• ComfyUI: , core path `, API at **http://127.0.0.1:8188`**

(alive when the server/app is running). Check: GET /system_stats -> 200.

• GPUs: x (cuda:0, cuda:1, ...). VRAM per card ``.
• Models installed (query live, do not hardcode): GET /object_info/UNETLoader,

/object_info/CheckpointLoaderSimple, /object_info/CLIPLoader, /object_info/VAELoader.

• Shared models dir / extramodelpaths: ``.
• GUI workflows folder (the bridge, see below): /user/default/workflows/.
• Launch command (to auto-start the server headlessly when :8188 is down): ``.

> Example from the kit author's machine (yours WILL differ): ComfyUI Desktop, core E:\ComfyUI\ComfyUI\ComfyUI, > 2x RTX 3090 (24GB each), models z_image_turbo_bf16, ideogram4_fp8_scaled, VAE ae/flux2-vae, > text encoders qwen3vl_8b_fp8_scaled/qwen_3_4b. Treat as illustration only.

The four layers (what this kit installs)

1. Knowledge + client: this SKILL.md and comfy_client.py (stdlib, no deps).
2. MCP driver: comfyui-mcp (artokun, MIT): ~90 structured tools so Claude operates ComfyUI directly

(generate, build/edit/validate graphs, model download, queue, VRAM, diagnostics, restart). Prefer its tools over hand-POSTing /prompt when present.

1. In-graph Claude nodes: Claude as a step INSIDE a workflow (prompt enrichment, vision QA on the output).
2. Node-building skills: comfyui-node-* (V3 API) for when we write or modify a custom node.

docs/LAYERS.md explains each; install.ps1 / install.sh wires them up.

The client (no extra deps, stdlib only)

comfy_client.py lives next to this SKILL.md. Import and use:

import sys; sys.path.insert(0, r"")
import comfy_client as c
c.alive()                                  # True if the API answers (override host with COMFY_HOST env)
c.run("path/to/workflow_api.json",
      overrides={"6.text": "a cinematic dragon, dark studio light", "3.seed": 12345},
      outdir=r"...\assets")                # queues, waits, downloads -> returns saved file paths

API surface: alive(), run(workflow_path, overrides, outdir, timeout), and the pieces queue(workflow), wait(prompt_id), download_outputs(rec, outdir), apply_overrides(wf, overrides). Override keys are "." (node ids and input names come straight from the workflow JSON). The MCP driver (Layer 2) does the same and more; use it when available, fall back to this client otherwise.

Workflow JSON (API format)

ComfyUI runs the "API format" graph: a dict { "": { "class_type": "...", "inputs": {...} }, ... }. To get one: in ComfyUI enable Settings -> Enable Dev mode Options, build the graph, then Save (API Format). Official starting graphs: Workflow -> Templates browser (per model). Save those as API Format, then parameterize.

To parameterize a graph, read it and find:

• the positive prompt: a CLIPTextEncode node, override .text.
• the seed: a KSampler / sampler node, override .seed (use a varied seed per call; do not hardcode).
• dimensions: an EmptyLatentImage / EmptySD3LatentImage node, override .width / .height.
• steps/cfg/sampler/scheduler: on the sampler node. Keep the template's values unless asked, they are model-tuned.

Template library (the SOURCE OF TRUTH, mix and match)

The official Comfy-Org workflow templates are the source of truth for how to do any task in ComfyUI. The kit clones them (sparse) to a local folder and builds a compact lookup index. Default location set by the installer; record it in the machine block above. Master index: templates/_quick_index.json (name -> title, category, models, tags, mediaType, vram, description), regenerate with tools/gen_quick_index.py. Update: git pull in the clone, then rerun the generator.

Flow: read _quick_index.json, find the template whose name/models/tags match the request, read THAT one templates/.json, parameterize it. New templates use SUBGRAPHS: the real pipeline is inside definitions.subgraphs[0], exposed params (text, width, height, seed, steps, model names) are in subgraphs[0].inputs, traced to inner nodes via the outer node's properties.proxyWidgets. Mix and match the blueprints/ (reusable subgraph bricks: text_to_image_z_image_turbo, image_to_video_ltx_2_3, image_upscale_z_image_turbo, remove_background_birefnet, ...).

widgets_values are ORDER-based, no field names: KSampler = [seed, controlaftergenerate, steps, cfg, sampler, scheduler, denoise]; EmptySD3LatentImage = [width, height, batch]. Model filenames must match installed files exactly. Validate node types/inputs against /object_info/ before writing a graph.

Compose a NEW workflow from pieces (assemble + wire it correctly)

When no single template fits, BUILD one by chaining pieces. The skill is for assembling, not only running.

1. Decompose the task into stages, one brick per stage, e.g. text-to-image -> upscale -> image-to-video -> add audio. Pick a template or a blueprints/ subgraph for each stage (match via _quick_index.json / blueprint names), and read each one to see its real input and output nodes.

2. Know how nodes connect (the key mechanic).

• API format: every input is EITHER a literal value OR a reference to another node's output, written as a

2-item list ["", ]. To run stage B after stage A, set B's input to ["", ], where ` is the index of A's matching output. Example: feed a decode's IMAGE into an upscaler -> "image": ["8", 0]` (node 8, output 0).

• GUI format: connections live in the top-level links array; each link is

[link_id, src_node, src_slot, dst_node, dst_slot, type], and each node's inputs[].link / outputs[].links carry those link ids. Write THIS to show the graph in the canvas (the bridge); write the API form to run.

3. Match types, or convert. Every output and input has a TYPE: IMAGE, LATENT, MODEL, CLIP, VAE, CONDITIONING, AUDIO, MASK, CONTROL_NET, ... You may ONLY connect matching types. Read each node's input + output types from /object_info/ (input.required / output / output_name). If a seam's types differ, insert a converter: VAEEncode (IMAGE -> LATENT), VAEDecode (LATENT -> IMAGE), CLIPTextEncode (text -> CONDITIONING), ImageScale / an upscaler for size. Never wire an IMAGE into a LATENT input.

Common node I/O (memorize these; for anything else read /object_info/). A node's WIDGETS are values you set; its INPUT SLOTS must receive the matching TYPE from another node's OUTPUT. You cannot feed text into a LoRA input, or a MODEL into a text box.

• CheckpointLoaderSimple -> out: MODEL, CLIP, VAE. (Flux/newer split loaders: UNETLoader -> MODEL ; DualCLIPLoader/CLIPLoader -> CLIP ; VAELoader -> VAE.)
• LoraLoader: in MODEL + CLIP (+ name/strength widgets) -> out MODEL, CLIP. A LoRA is applied ONTO the MODEL+CLIP stream, never wired as text.
• CLIPTextEncode: in CLIP + text widget -> out CONDITIONING. Your prompt becomes CONDITIONING here; downstream nodes want CONDITIONING, not raw text.
• EmptyLatentImage / EmptySD3LatentImage: widgets only -> out LATENT.
• KSampler / KSamplerAdvanced: in MODEL + positive CONDITIONING + negative CONDITIONING + LATENT (+ seed/steps/cfg/sampler/scheduler/denoise widgets) -> out LATENT.
• VAEDecode: in LATENT + VAE -> out IMAGE. VAEEncode: in IMAGE + VAE -> out LATENT.
• ControlNetLoader -> CONTROLNET ; ControlNetApplyAdvanced: in CONDITIONING + CONTROLNET + IMAGE -> out CONDITIONING.
• LoadImage -> IMAGE, MASK ; SaveImage / PreviewImage: in IMAGE.

Basic txt2img stream: loader -> (LoraLoader) -> CLIPTextEncode x2 (pos/neg) -> KSampler (+ EmptyLatentImage) -> VAEDecode -> SaveImage. Before building, also check KNOWN_ISSUES.md (next to this file or docs/KNOWN_ISSUES.md) and ADVANCED.md for current bugs and workarounds, so you do not wire around a known-broken path.

4. Merge graphs cleanly. Splicing two templates: renumber one graph's node ids so they do not collide; SHARE the loaders (one CheckpointLoader / UNETLoader / VAELoader / CLIPLoader feeding both stages, do not duplicate the same model); then wire the seam (stage A's final output -> stage B's first input). Keep each model's own VAE / encoder with it (a Wan VAE is not an SDXL VAE; LTX bundles its VAE in the checkpoint).

5. Validate before running. Check: every class_type exists in /object_info; every input is a literal or a [node, slot] ref to an existing node; every seam's types match; model filenames exist locally; and the graph has at least one INPUT node carrying the user's intent AND at least one OUTPUT/save node wired to the final tensor (SaveImage / SaveAudio / SaveVideo / VHS_VideoCombine, or a PreviewImage). API / partner nodes (Kling, Nano Banana, Veo, Gemini, ...) often emit a tensor but include NO save node by default - add and wire one, or the job runs "successfully" and produces nothing retrievable, wasting the compute. Then run SMALL / low-res FIRST to confirm the wiring, before the full render. Emit both formats: GUI to show in the canvas, API to run. When unsure of a node's exact inputs/outputs, query /object_info/ rather than guessing.

Shared workflows + model shootout (pick the best model for a look)

Beyond the named template library, ComfyHub hosts thousands of community-shared workflows at comfy.org/workflows/. Any ComfyHub share downloads as plain JSON from a predictable URL: https://comfy.org/workflows/download/.json. So you can grab any shared workflow on demand, then read or run it. Helper: python shared/tools/fetch_workflow.py (stdlib). The ` is the id in the share URL. Note: cloud.comfy.org/?share=` links are Comfy Cloud only and are NOT downloadable this way (open in Comfy Cloud and export from the canvas).

Model shootout (which model is best for THIS prompt): the template library already ships a comparison grid, templates-all_in_one-image_edit_models ("1 input and multiple editing model comparison"): it fans one input image through 7 image-edit models at once (Flux.2 Dev/Klein, GPT-Image-1.5, Grok, Nano Banana Pro, Qwen-Image-Edit, Seedream) and saves each output side by side, so you pick the best look before committing. For video, the community "Adjustment Frame" share (hash 7dca0438edf4) compares video backends (Grok/Kling/Veo/Seedance/Wan2.2/LTX-2). Run small / low-res first, compare, then scale up the winner. This pairs with the per-model recipes below and the hardware-aware fit check.

Real production graphs to study: Comfy-Org/creative-campus (github.com/Comfy-Org/creative-campus) collects the actual workflows from Comfy Education Initiative case studies, real graphs from award-winning artists (e.g. Xindi Zhang's Song of Drifters, a Student Academy Award film: SD1.5 style transfer with IP-Adapter + ControlNet, plus a 3D + AI morphing graph). Open and study them for production technique. Link-and-study only (no license file; shared with the artists' permission), so reference it, do not bundle the JSONs.

Staying current (new models and workflows)

ComfyUI ships new models constantly, and they land in the template library first. To see what is new: git pull the templates clone and regenerate the quick index (gen_quick_index.py), then DIFF the model list (names not seen before = new models / new templates). Also read the announcements RSS at https://blog.comfy.org/feed. The kit ships shared/tools/check_updates.py, which does all of this in one command (pull + diff + RSS). When a genuinely new generative model appears without a recipe, research its OFFICIAL prompting (maker docs / model card / docs.comfy.org) and add it to MODELS.md in the same format; a new utility/upscaler goes to the Enhancement section. Do NOT scrape LinkedIn (auth-gated, anti-scraping, ToS); the blog RSS and the templates repo carry the same news, machine-readable. Full loop: the kit's docs/UPDATING.md.

Per-model prompting (the mega-brain): READ before prompting a named model

Every generative model has its own dialect. SDXL wants comma tags, FLUX wants natural-language sentences, video models want camera + motion direction, audio models want genre/tempo/instruments, and negative-prompt support varies (FLUX and many turbo models ignore or break on negatives). The kit ships a per-model prompting reference, MODELS.md (next to this file), distilled from OFFICIAL sources: each maker's docs / model cards, docs.comfy.org, and the anthropic-claude node's per-model templates.

Auto-pull rule: when a specific model is named in the request, the workflow, or the chosen template, READ that model's entry in MODELS.md BEFORE writing the prompt, and follow its prompt structure, its negative-prompt rule, and its settings. Never carry one model's style to another.

MODELS.md covers (image) FLUX.1/.2 + Kontext, Z-Image-Turbo, Qwen-Image/Edit, SDXL, SD1.5, SD3.5, HiDream, Ideogram, Nano Banana Pro/2, Seedream 4.x/5 Lite, Recraft, GPT-Image, Grok, Reve, Kandinsky, BRIA, OmniGen, Chroma, Krea, ERNIE-Image; (image edit) FLUX Kontext, Qwen-Image-Edit, FireRed, LongCat, ChronoEdit; (video) Wan 2.1-2.7, LTX-2.3 / 2 Pro, Hunyuan Video, SVD, Kling, Veo, Sora, Seedance, Luma, Runway, MiniMax, PixVerse, Vidu, Pika, HappyHorse, HuMo, SCAIL-2; (audio) Stable Audio, ACE-Step, ElevenLabs, ChatterBox, Sonilo; (3D) Hunyuan3D, Tripo, Rodin, Meshy; (newer/niche) Capybara, Bernini-R, Anima, NewBie, PixelDiT, Ovis-Im

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: SlavaSexton
• Source: SlavaSexton/ComfyUI-Agent-Kit
• License: MIT

Install and usage instructions live in the source repository linked above.