Viewing Videos

Viewing Videos

Overview

Claude can't watch video, but it reads images well. Turn the video into the fewest frames that answer the question with ffmpeg, triage them on a contact sheet, then Read the interesting ones at full resolution. Requires ffmpeg + ImageMagick montage (both via Homebrew).

Step 0: Probe first

ffprobe -v error -show_entries format=duration:stream=codec_type,width,height -of default=noprint_wrappers=1 video.mp4

Duration picks the strategy; resolution sets expectations — 1080p+ screen content reads fine, at 720p big KPI numbers are fine but dense tables/spreadsheets need the crop+upscale step below.

Pick a strategy

| Situation | Strategy | |---|---| | You know the moment (timestamped transcript, user says "at 12:34") | Targeted single frames | | Screen shares / slides / UI demos, unknown moments | Scene-change sweep (default) | | Continuous motion (camera footage, gameplay) — scenes never "change" | Interval sampling (see frame budget below) |

Targeted frame

-ss before -i = fast seek:

ffmpeg -ss 12:34 -i video.mp4 -frames:v 1 /tmp/frames/t12m34s.png

Scene-change sweep → timestamp names → contact sheet

rm -rf /tmp/frames && mkdir -p /tmp/frames
ffmpeg -hide_banner -i video.mp4 -vf "select='gt(scene,0.2)',showinfo" -fps_mode vfr /tmp/frames/frame_%03d.png 2> /tmp/frames/log.txt
cd /tmp/frames && grep -o 'pts_time:[0-9.]*' log.txt | cut -d: -f2 | awk '{n++; t=int($1); printf "mv frame_%03d.png t%02dm%02ds.png\n", n, t/60, t%60}' | sh
montage -label '%t' -font /System/Library/Fonts/Helvetica.ttc t*.png -tile 6x -geometry 320x180+4+4 -pointsize 18 contact_%d.png

Read contact_0.png to triage (timestamps are printed under each tile), then Read only the frames that matter at full resolution. Threshold 0.2 works for meetings: webcam segments cluster harmlessly, screen shares fire exactly on slide/page changes. Footage with constant motion → hundreds of frames; switch to interval sampling.

Frame budget

Aim for ≤ ~100 frames per sweep — enough to cover a long video, few enough to triage on 1–2 contact sheets. Tune to hit it, don't accept whatever falls out:

• Scene detection over-firing? Raise the threshold (gt(scene,0.3)–0.4).
• Interval sampling: derive the interval from duration instead of fixing it — interval = max(5, duration_seconds / 100), i.e. -vf fps=1/$INTERVAL. A 10-min clip samples every ~6s; a 3-hour recording every ~108s (not 1080 frames at a blind 1/10s).
• If the question lives in one segment, sweep coarse first, then densify only that range: ffmpeg -ss -to -i video.mp4 -vf fps=1/2 ....

Small text unreadable? Crop + upscale

Re-extract just the region at 2× rather than squinting at the full frame:

ffmpeg -ss 17:40 -i video.mp4 -frames:v 1 -vf "crop=900:500:80:120,scale=2*iw:-1" /tmp/frames/zoom_t17m40s.png

Get crop coords by reading the full frame first.

With a transcript

Timestamped transcript (.vtt) → go targeted: find where figures/screens are discussed, seek those times. Plain text transcript (no timestamps) → sweep, then correlate by content order. If the user has a choice, ask for the timestamped export.

Common mistakes

• Globs in compound commands: zsh aborts the whole && chain on a no-match glob (rm -f *.png → ffmpeg never runs). Use rm -rf dir && mkdir -p dir.
• montage without -font: fails with "unable to read font" on macOS. Always pass -font /System/Library/Fonts/Helvetica.ttc.
• Blind 1-frame-per-second sampling: 30 min → 1800 redundant frames. Scene detection or 1/10s intervals first; densify only where needed.
• Skipping the contact sheet: reading 60+ full frames one by one burns context. Triage on the sheet, read few.
• Trusting frame numbers instead of timestamps: rename to tMMmSSs.png immediately, or you can't correlate with the transcript or seek back for a better frame.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: shaharsha
• Source: shaharsha/claude-skills
• License: MIT

Install and usage instructions live in the source repository linked above.