Video Use Pro

video-use-pro

Principle

1. LLM reasons from raw transcript + on-demand visuals. The only derived artifact that earns its keep is a packed phrase-level transcript (takes_packed.md). Everything else — filler tagging, retake detection, shot classification, emphasis scoring — you derive at decision time.
2. Audio is primary, visuals follow. Cut candidates come from speech boundaries and silence gaps. Drill into visuals only at decision points.
3. Ask → confirm → execute → iterate → persist. Never touch the cut until the user has confirmed the strategy in plain English.
4. Generalize. Do not assume what kind of video this is. Look at the material, ask the user, then edit.
5. Artistic freedom is the default. Every specific value, preset, font, color, duration, pitch structure, and technique in this document is a worked example from one proven video — not a mandate. Read them to understand what's possible and why each worked. Then make your own taste calls based on what the material actually is and what the user actually wants. The only things you MUST do are in the Hard Rules section below. Everything else is yours.
6. Invent freely. If the material calls for a technique not described here — split-screen, picture-in-picture, lower-third identity cards, reaction cuts, speed ramps, freeze frames, crossfades, match cuts, L-cuts, J-cuts, speed ramps over breath, whatever — build it. The helpers are ffmpeg and PIL. They can do anything the format supports. Do not wait for permission.
7. Verify your own output before showing it to the user. If you wouldn't ship it, don't present it.

Hard Rules (production correctness — non-negotiable)

These are the things where deviation produces silent failures or broken output. They are not taste, they are correctness. Memorize them.

1. Subtitles are applied LAST in the filter chain, after every overlay. Otherwise overlays hide captions. Silent failure.
2. Per-segment extract → lossless -c copy concat, not single-pass filtergraph. Otherwise you double-encode every segment when overlays are added.
3. 30ms audio fades at every segment boundary (afade=t=in:st=0:d=0.03,afade=t=out:st={dur-0.03}:d=0.03). Otherwise audible pops at every cut.
4. Overlays use setpts=PTS-STARTPTS+T/TB to shift the overlay's frame 0 to its window start. Otherwise you see the middle of the animation during the overlay window.
5. Master SRT uses output-timeline offsets: output_time = word.start - segment_start + segment_offset. Otherwise captions misalign after segment concat.
6. Never cut inside a word. Snap every cut edge to a word boundary from the transcript.
7. Pad every cut edge. Working window: 30–200ms. Transcript timestamps drift 50–100ms — padding absorbs the drift. Tighter for fast-paced, looser for cinematic.
8. Word-level verbatim ASR only. Never SRT/phrase mode (loses sub-second gap data). Never normalized fillers (loses editorial signal).
9. Cache transcripts per source. Never re-transcribe unless the source file itself changed.
10. Parallel sub-agents for multiple animations. Never sequential. Spawn N at once via the Agent tool; total wall time ≈ slowest one.
11. Strategy confirmation before execution. Never touch the cut until the user has approved the plain-English plan.
12. All session outputs in /edit/. ` defaults to the skill's own projects//` directory (skill code dirs — helpers/, SKILL.md — stay untouched). Use absolute paths only when the user gives them.

Everything else in this document is a worked example. Deviate whenever the material calls for it.

Directory layout

The skill lives in video-use-pro/. User footage lives wherever they put it. **默认 ` 是技能目录下的项目文件夹** video-use-pro/projects//——除非用户显式给出其他路径。All session outputs go into /edit/`.

video-use-pro/
├── SKILL.md / helpers/ ...     ← skill code (untouched by projects)
└── projects/
    └── /                  ← default 
        ├── 
        └── edit/
            ├── project.md               ← memory; appended every session
            ├── takes_packed.md          ← phrase-level transcripts, the LLM's primary reading view
            ├── edl.json                 ← cut decisions
            ├── transcripts/.json  ← cached raw transcript JSON
            ├── animations/slot_/    ← per-animation source + render + reasoning
            ├── clips_graded/            ← per-segment extracts with grade + fades
            ├── master.srt               ← output-timeline subtitles
            ├── downloads/               ← yt-dlp outputs
            ├── verify/                  ← debug frames / timeline PNGs
            ├── preview.mp4
            └── final.mp4

Setup

First-time install lives in install.md (clone, deps, ffmpeg, skill registration, API key). Don't re-run it every session; on cold start just verify:

• 转写凭证：中文口播主用火山 ASR（volc_transcribe.py 从技能目录 .env 读 VOLCAPPID/VOLCACCESSTOKEN，默认 video-use-pro/.env，可用 --env 覆盖）。本地兜底为 whisper（transcribe.py，离线可用，中文精度弱于火山）。
• ffmpeg + ffprobe on PATH.
• Python deps installed (uv sync or pip install -e . inside the repo).
• Node.js + npm available if the session needs HyperFrames or Remotion slots. HyperFrames currently requires Node.js 22+.
• yt-dlp, HyperFrames, Remotion, Manim installed only on first use.
• First-use animation setup happens inside the slot directory, never at the video-use-pro repo root. HyperFrames can be invoked with npx --yes hyperframes ...; Remotion can be scaffolded with npx create-video@latest or installed as a project-local dependency before using its remotion render command.
• This skill vendors skills/manim-video/. Read its SKILL.md when building a Manim slot.

Helpers (helpers/transcribe.py, helpers/volc_transcribe.py, helpers/render.py, etc.) live alongside this SKILL.md. Resolve their paths relative to the directory containing this file — the skill lives at tools/video/video-use-pro/.

Helpers

• transcribe.py — local Whisper transcription (word-level). 默认 --model medium（中文口播建议 medium 及以上）; --gain for quiet sources. Cached. 无火山凭证时的兜底方案。
• volc_transcribe.py — 火山引擎(豆包) ASR transcription. Best for 中文口播 (talking-head) video: verbatim (preserves repetitions/卡壳 that Whisper merges) + word-level timestamps. --gain N for quiet sources, --seg N chunks long audio, --env for credentials (VOLCAPPID/VOLCACCESSTOKEN). Cached. See SKILL.md §中文口播.
• transcribe_batch.py — 4-worker parallel transcription. Use for multi-take.
• pack_transcripts.py --edit-dir — transcripts/*.json → takes_packed.md (phrase-level, break on silence ≥ 0.5s).
• timeline_view.py — filmstrip + waveform PNG. On-demand visual drill-down. Not a scan tool — use it at decision points, not constantly.
• render.py -o — per-segment extract → concat → overlays (PTS-shifted) → subtitles LAST. --preview for 720p fast. --build-subtitles to generate master.srt inline.
• grade.py -o — ffmpeg filter chain grade. Presets + --filter '' for custom.

For animations, create /animations/slot_/ with Bash and spawn a sub-agent via the Agent tool.

The process

1. Inventory. ffprobe every source. transcribe_batch.py on the directory. pack_transcripts.py to produce takes_packed.md. Sample one or two timeline_views for a visual first impression.
2. Pre-scan for problems. One pass over takes_packed.md to note verbal slips, obvious mis-speaks, or phrasings to avoid. Plain list, feed into the editor brief.
3. Converse. Describe what you see in plain English. Ask questions shaped by the material. Collect: content type, target length/aspect, aesthetic/brand direction, pacing feel, must-preserve moments, must-cut moments, animation and grade preferences, subtitle needs. Do not use a fixed checklist — the right questions are different every time.
4. Propose strategy. 4–8 sentences: shape, take choices, cut direction, animation plan, grade direction, subtitle style, length estimate. Wait for confirmation.
5. Execute. Produce edl.json via the editor sub-agent brief. Drill into timeline_view at ambiguous moments. Build animations in parallel sub-agents. Apply grade per-segment. Compose via render.py.
6. Preview. render.py --preview.
7. Self-eval (before showing the user). Run timeline_view on the rendered output (not the sources) at every cut boundary (±1.5s window). Check each image for:

• Visual discontinuity / flash / jump at the cut
• Waveform spike at the boundary (audio pop that slipped past the 30ms fade)
• Subtitle hidden behind an overlay (Rule 1 violation)
• Overlay misaligned or showing wrong frames (Rule 4 violation)

Also sample: first 2s, last 2s, and 2–3 mid-points — check grade consistency, subtitle readability, overall coherence. Run ffprobe on the output to verify duration matches the EDL expectation.

If anything fails: fix → re-render → re-eval. Cap at 3 self-eval passes — if issues remain after 3, flag them to the user rather than looping forever. Only present the preview once the self-eval passes.

1. Iterate + persist. Natural-language feedback, re-plan, re-render. Never re-transcribe. Final render on confirmation. Append to project.md.

Cut craft (techniques)

• Audio-first. Candidate cuts from word boundaries and silence gaps.
• Preserve peaks. Laughs, punchlines, emphasis beats. Extend past punchlines to include reactions — the laugh IS the beat.
• Speaker handoffs benefit from air between utterances. Common values: 400–600ms. Less for fast-paced, more for cinematic. Taste call.
• Audio events as signals. (laughs), (sighs), (applause) mark beats. Extend past them.
• Silence gaps are cut candidates. Silences ≥400ms are usually the cleanest. 150–400ms phrase boundaries are usable with a visual check. Whisper（口播定位必须用火山）**
• Whisper merges repetitions — "太阳一出来×2" becomes one "出来" (drags 3.1s), drops "而且啊/也是要先/所以在这个月". You literally cannot find what the transcript erased.
• 火山 engine (volc_transcribe.py) is verbatim + gives word-level timestamps (ms). On a 5-min test: 838 chars vs Whisper-medium 713 — every repetition Whisper missed was present verbatim.

三级删除流程（每级都语义判断，禁止机械字符匹配）：

1. 第一级 大段粗筛 — energy (silencedetect) finds long silences, repeated 导入句 ("接下来我们看…案例" ×N, keep 1), 语气词集中区.
2. 第二级 句子级（核心） — for each sentence, judge its 语义关系 to neighbors: 开头接不上前句结尾 → 卡壳引子; 整句语义与前/后句重复 → 重复句; 半截没说完 → 半截话; 自然承接 → 保留. 文字错字（甲木/假木）不影响语义判断.
3. 第三级 词语级 — word-level 卡壳/重复. Real speech does NOT repeat verbatim — it inserts words ("而且，不太喜欢"→"而且我(加'我')不太喜欢"), rephrases, or trails off. 机械匹配 (SequenceMatcher) only粗筛 high-similarity candidates; final call is semantic (structure corresponds, meaning repeats → 卡壳).

卡壳铁律：取后者。 真人说话习惯是卡壳后修正，修正版永远在后。删废稿、留修正完整句。Exceptions: 并列重复（同一句两遍）取最完整那遍; 语气词（嗯啊这个那个）直接删.

判断重复必须用音频能量分析（转写只做定位）： silencedetect (quiet sources need volume=19dB first) shows the 语音块分布 — two repetitions = two blocks; a 卡壳 = isolated block followed by silence then the complete sentence. Never decide from transcript text alone.

画面层检查：静音≠可删（画板写字=画面信息）。 For every 删除区, run fps=5 frame-diff (adjacent-frame MSE). 手写/渐进展示 (连续≥3次中等diff) = keep the visuals even if speech is 废话. 静止 (all video. Pick the best take of each beat and assemble them chronologically by beat, not by source clip order.

INPUTS:

• takes_packed.md (time-annotated phrase-level transcripts of all takes)
• Product/narrative context:
• Speaker(s):
• Expected structure:
• Verbal slips to avoid:
• Target runtime:

Common structural archetypes (pick, adapt, or invent):

• Tech launch / demo: HOOK → PROBLEM → SOLUTION → BENEFIT → EXAMPLE → CTA
• Tutorial: INTRO → SETUP → STEPS → GOTCHAS → RECAP
• Interview: (QUESTION → ANSWER → FOLLOWUP) repeat
• Travel / event: ARRIVAL → HIGHLIGHTS → QUIET MOMENTS → DEPARTURE
• Documentary: THESIS → EVIDENCE → COUNTERPOINT → CONCLUSION
• Music / performance: INTRO → VERSE → CHORUS → BRIDGE → OUTRO
• Or invent your own.

RULES:

• Start/end times must fall on word boundaries from the transcript.
• Pad cut boundaries (working window 30–200ms).
• Prefer silences ≥ 400ms as cut targets.
• Unavoidable slips are kept if no better take exists. Note them in "reason".
• If over budget, revise: drop a beat or trim tails. Report total and self-correct.

OUTPUT (JSON array, no prose): [{"source": "C0103", "start": 2.42, "end": 6.85, "beat": "HOOK", "quote": "...", "reason": "..."}, ...]

Return the final EDL and a one-line total runtime check.

## Color grade (when requested)

Your job is to **reason about the image**, not apply a preset. Look at a frame (via `timeline_view`), decide what's wrong, adjust one thing, look again.

Mental model is ASC CDL. Per channel: `out = (in * slope + offset) ** power`, then global saturation. `slope` → highlights, `offset` → shadows, `power` → midtones.

**Example filter chains** (`grade.py` has `--list-presets`; use them as starting points or mix your own):

- **`warm_cinematic`** — retro/technical, subtle teal/orange split, desaturated. Shipped in a real launch video. Safe for talking heads.
- **`neutral_punch`** — minimal corrective: contrast bump + gentle S-curve. No hue shifts.
- **`none`** — straight copy. Default when the user hasn't asked.

For anything else — portraiture, nature, product, music video, documentary — invent your own chain. `grade.py --filter ''` accepts any filter string.

Hard rules: apply **per-segment during extraction** (not post-concat, which re-encodes twice). Never go aggressive without testing skin tones.

## Subtitles (when requested)

Subtitles have three dimensions worth reasoning about: **chunking** (1/2/3/sentence per line), **case** (UPPER/Title/Natural), and **placement** (margin from bottom). The right combo depends on content.

**Worked styles** — pick, adapt, or invent:

**`bold-overlay`** — short-form tech launch, fast-paced social. 2-word chunks, UPPERCASE, break on punctuation, Helvetica 18 Bold, white-on-outline, `MarginV=35`. `render.py` ships with this as `SUB_FORCE_STYLE`.

FontName=Helvetica,FontSize=18,Bold=1, PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,BackColour=&H00000000, BorderStyle=1,Outline=2,Shadow=0, Alignment=2,MarginV=35

**`natural-sentence`** (if you invent this mode) — narrative, documentary, education. 4–7 word chunks, sentence case, break on natural pauses, `MarginV=60–80`, larger font for readability, slightly wider max-width. No shipped force_style — design one if you need it.

Invent a third style if neither fits. Hard rules: subtitles LAST (Rule 1), output-timeline offsets (Rule 5).

## Animations (when requested)

Animations match the content and the brand. **Get the palette, font, and visual language from the conversation** — never assume a default. If the user hasn't told you, propose a palette in the strategy phase and wait for confirmation before building anything.

**Tool options:**

Pick the engine per animation slot. Do not default to Remotion just because the animation is web-adjacent.

- **HyperFrames** — Browser-native HTML/CSS/GSAP video compositions: product UI motion, website-to-video or mockup-to-video captures, kinetic typography, landing-page/storyboard promos, data-driven UI states, transparent WebM overlays, and clips that need deterministic frame capture plus HyperFrames lint/validate/render checks. Best when the animation should be authored and verified like a web composition instead of a React component tree.
- **Remotion** — React/CSS compositions with component state, reusable React primitives, or an existing Remotion brand system. Best when the user specifically asks for React/Remotion or when React composition is the simpler authoring model.
- **Manim** — formal diagrams, state mac

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [huige-opc](https://github.com/huige-opc)
- **Source:** [huige-opc/video-use-pro](https://github.com/huige-opc/video-use-pro)
- **License:** MIT

Install and usage instructions live in the source repository linked above.