Video Analyser

Video Analyser Skill

Analyse a video file to extract bugs, errors, UI state, and reproduction steps. The pipeline uses ffmpeg for frame extraction, optional tesseract OCR for text, and optional whisper for audio narration. The default path (8 keyframes at 768 px) is the Pareto-optimal setting for screen recordings: best quality-per-token on the legibility curve.

Prerequisites

Required tools

| Tool | Check | If missing | |---|---|---| | ffmpeg | which ffmpeg | Print: ffmpeg is required. Install with: brew install ffmpeg (macOS) or apt install ffmpeg (Linux). Then exit. | | ffprobe | which ffprobe | Print: ffprobe is required. It ships with ffmpeg — reinstall ffmpeg. Then exit. |

Optional tools (silent degradation)

| Tool | If present | If absent | |---|---|---| | tesseract | Enable OCR mode for text-heavy frames | Skip silently; use vision-only mode | | whisper | Enable audio transcription when user mentions narration or voiceover | Skip audio step silently |

Run tool detection before any other step.

which ffmpeg  || { echo "ffmpeg is required. Install with: brew install ffmpeg (macOS) or apt install ffmpeg (Linux)."; exit 1; }
which ffprobe || { echo "ffprobe is required. It ships with ffmpeg — reinstall ffmpeg."; exit 1; }
OCR_ENABLED=false; which tesseract >/dev/null 2>&1 && OCR_ENABLED=true
AUDIO_ENABLED=false

Minimum ffmpeg version

Require ffmpeg 4 or later. The select='eq(pict_type\,I)' filter and -vsync vfr were introduced in ffmpeg 4. Document this requirement in the bail message if detection fails.

Temp Directory

Declare the temp directory and cleanup trap at the very start of execution, before source resolution. This ensures downloaded files, extracted frames, and audio are always removed, even on error.

WORK_DIR=$(mktemp -d /tmp/video-analyser-XXXXXX)
trap 'rm -rf "$WORK_DIR"' EXIT

Use $WORK_DIR as the staging area for all intermediate files throughout the pipeline.

Step 1 — Source Resolution

Resolve the user's input to a local file path before running anything else. Walk this table top-to-bottom; the first matching rule wins.

| Input shape | Detection rule | Resolution steps | |---|---|---| | Linear ticket URL | Input matches linear\.app/.+/issue/ | Follow the [Linear resolution procedure](#linear-ticket-url) below. | | Local file path | Input starts with /, ./, ~/, or ~ | Follow the [local path procedure](#local-file-path) below. | | Direct video URL | Input matches ^https?:// and does not match linear\.app | Download: curl -fL -o "$WORK_DIR/input.mp4" "$INPUT". Use $WORK_DIR/input.mp4 as VIDEO_PATH. | | Bare filename | Input contains no / and no http | Prepend $PWD/; then follow the local path procedure. | | Unresolvable | None of the above match | Print: Cannot resolve input to a video file. Provide a Linear ticket URL, a local file path, or a direct URL to a video file. Then exit. |

Linear ticket URL

1. Check Linear MCP availability — at the instruction level, not via a shell command.

Inspect the tool list available in this session for a Linear MCP issue-read tool; the name varies by host (for example mcp__Linear__get_issue or mcp__claude_ai_Linear__get_issue). If no such tool is present, do not run any shell command to detect MCP — instead ask the user to paste the ticket content (or download the video attachment and re-invoke with a local file path), then continue with whichever they provide.

Do not attempt to scrape the Linear web UI as a fallback.

1. Extract the issue ID from the URL (e.g., XYZ-123 from https://linear.app/team/issue/XYZ-123).
2. Call Linear MCP get_issue with the issue ID.
3. Scan the description field and all comments for attachment URLs matching \.(mp4|mov|webm|avi) (case-insensitive).
4. If no video attachment is found, print: No video attachment found in Linear issue . Attach a .mp4, .mov, .webm, or .avi file to the issue and retry. Then exit.
5. Download the first matching attachment.

curl -fL -o "$WORK_DIR/input.mp4" "$ATTACHMENT_URL"
VIDEO_PATH="$WORK_DIR/input.mp4"

Local file path

1. Expand ~ to the home directory.

VIDEO_PATH="${INPUT/#\~/$HOME}"

1. Confirm the file exists.

test -f "$VIDEO_PATH" || { echo "File not found: $VIDEO_PATH"; exit 1; }

1. Check the file extension.

If the extension is not .mp4, .mov, .webm, or .avi, print a warning but continue.

case "${VIDEO_PATH##*.}" in
  mp4|mov|webm|avi) ;;
  *) echo "Warning: unrecognised extension '${VIDEO_PATH##*.}'. Continuing anyway." ;;
esac

Step 2 — Video Probe

Run ffprobe to extract metadata before any frame work.

PROBE=$(ffprobe -v quiet -print_format json -show_streams -show_format "$VIDEO_PATH")
DURATION_S=$(echo "$PROBE" | python3 -c "import sys,json; d=json.load(sys.stdin); print(float(d['format']['duration']))")
WIDTH=$(echo "$PROBE" | python3 -c "import sys,json; d=json.load(sys.stdin); v=[s for s in d['streams'] if s.get('codec_type')=='video'][0]; print(v['width'])")
HEIGHT=$(echo "$PROBE" | python3 -c "import sys,json; d=json.load(sys.stdin); v=[s for s in d['streams'] if s.get('codec_type')=='video'][0]; print(v['height'])")
HAS_AUDIO=$(echo "$PROBE" | python3 -c "import sys,json; d=json.load(sys.stdin); print('true' if any(s.get('codec_type')=='audio' for s in d['streams']) else 'false')")

Step 3 — Bail Gates

Check these conditions in order before proceeding. Stop immediately when a gate triggers.

| Condition | Action | |---|---| | DURATION_S > 600 | Print the trim command below, then exit. | | `DURATION_S 600 s | Any | — | Bail at Step 3 |

Cost warning for 24-frame opt-in. Before proceeding to N=24, print:

Note: 24 frames at 768 px ≈ 18,864 tokens ≈ $0.057 on Sonnet 4.6.
Proceed? (yes/no)

Wait for explicit confirmation. If the user does not confirm, fall back to N=16.

Step 5 — Extract Frames

Use the keyframe-first strategy. I-frames in screen recordings mark scene transitions (page loads, error dialogs, modal appearances) and carry the most diagnostic signal.

Step 5a — Keyframe extraction

ffmpeg -i "$VIDEO_PATH" \
  -vf "select='eq(pict_type\,I)',scale=768:-2" \
  -vsync vfr -q:v 2 \
  "$WORK_DIR/frame_%04d.jpg" -y 2>/dev/null
IFRAME_COUNT=$(ls "$WORK_DIR"/frame_*.jpg 2>/dev/null | wc -l)

If $VIDEO_PATH is a .webm file and IFRAME_COUNT is 0, the container may not expose keyframe metadata. Fall back immediately to uniform sampling (Step 5b) with IFRAME_COUNT=0.

Step 5b — Uniform fill (when keyframes are sparse)

| Condition | Action | |---|---| | IFRAME_COUNT >= N | Use the first N keyframes; skip Step 5b. | | `IFRAME_COUNT /dev/null

### Step 5c — Select final N frames

1. List all keyframes first (`frame_*.jpg`), sorted by name.
2. Append uniform frames (`uniform_*.jpg`), sorted by name.
3. Keep the first N entries.
4. Copy selected frames to `$WORK_DIR/selected_%04d.jpg`.

If fewer than N total frames were extracted, proceed with however many were extracted.
Do not bail on a low frame count.

### Scaling note

If `scale=768:-2` produces an error (odd-dimension video), use `scale=768:trunc(ow/a/2)*2` instead.

## Step 6 — OCR (conditional)

Run OCR when ALL of the following are true:
- `OCR_ENABLED=true` (tesseract is present).
- The user's goal is text or error extraction (default — skip only if user explicitly requests visual-only analysis).

For each selected frame:

```bash
for FRAME in "$WORK_DIR"/selected_*.jpg; do
  FRAME_ID=$(basename "$FRAME" .jpg)
  tesseract "$FRAME" "$WORK_DIR/ocr_${FRAME_ID}" -l eng 2>/dev/null
done

Store OCR output in $WORK_DIR/ocr_selected_*.txt. Pass OCR text to the analysis prompt as TEXT blocks, one per frame. OCR text is cheaper context than asking vision to re-read the same text.

Non-English UI text. The default language is -l eng. If the user specifies a language (e.g., "the UI is in German"), substitute -l (e.g., -l deu). Tesseract language codes follow ISO 639-3.

Step 7 — Audio Transcription (conditional)

Run audio transcription when ALL of the following are true:

• whisper is available on $PATH.
• HAS_AUDIO=true (the video has an audio stream).
• The user's message contains "narration", "voiceover", "audio", or "they said".

Extract audio and transcribe:

ffmpeg -i "$VIDEO_PATH" -vn -acodec pcm_s16le -ar 16000 "$WORK_DIR/audio.wav" -y 2>/dev/null
whisper "$WORK_DIR/audio.wav" --model small --output_format txt --output_dir "$WORK_DIR" 2>/dev/null

Append the transcript to the analysis prompt as TEXT.

Step 8 — Assemble Analysis Prompt

Use the cheapest delivery mechanism for the chosen frame count. For N ≤ 16, inline base64 is typically cheapest; for N > 16, prefer the Files API if the runtime supports it. The executor decides at runtime.

Construct the prompt as follows:

System:
  You are analysing a screen recording of a software application.
  Your goal:  (default: "identify bugs, errors, UI state issues, and reproduction steps").
  You will receive  frames extracted from the recording.
  Cross-reference OCR text blocks with what you see in each frame.
  Do not hallucinate text — if OCR and vision disagree, note both.

For each frame (in order):
  [image content block — JPEG]
  [if OCR enabled: OCR TEXT HERE]

[if audio transcription enabled:]
  TRANSCRIPT TEXT HERE

Request:
  Return findings in the structured format specified in Step 9.

Step 9 — Deliver Structured Output

Return findings in this exact schema. Do not omit sections; use "None detected" for sections with no findings.

## Video Analysis

### Recording summary
- Duration:  seconds
- Resolution: ×
- Frames analysed:  ()
- OCR: 
- Audio transcription: 

### Findings

#### Errors and exceptions

#### UI state at key moments

#### Reproduction steps inferred

#### Recommended next steps

Token Budget

The default setting (8 frames, 768 px) is the Pareto-optimal point on the quality-vs-token curve for screen recordings.

768 px is the curve knee for screen-recording legibility. UI text (error messages, stack traces, console output) is fully legible at 768 px. Going lower (512 px) risks misreading small-font terminal output. Going higher (1568 px) roughly doubles token cost with no meaningful legibility gain for screen content.

8 frames is the sweet spot for recordings under 3 min. I-frame sampling captures the scene transitions that carry the diagnostic signal. Beyond ~10 frames, incremental frames are typically near-duplicates that add tokens without adding context.

| Frames | Resolution | Tokens/frame | Total image tokens | Cost (Sonnet 4.6, $3/M) | Use when | |---|---|---|---|---|---| | 8 (default) | 768 px | ~786 | ~6,288 | ~$0.019 | Standard bug recording; Pareto-optimal default | | 16 (escalation) | 768 px | ~786 | ~12,576 | ~$0.038 | User says "fine-grained" or "detailed" | | 24 (long clip opt-in) | 768 px | ~786 | ~18,864 | ~$0.057 | Clip 181–600 s + user explicitly opts in after cost warning | | 8 | 1568 px (cap) | ~1,568 | ~12,544 | ~$0.038 | Only if user reports text still unreadable at 768 px |

For clips longer than 10 min (after the user trims), or for frame counts > 24, consider Gemini's native video API as an escalation path. Gemini charges per-second rather than per-frame and handles long clips natively.

Sampling Method Reference

| Condition | Method | |---|---| | I-frame count ≥ N | Keyframe extraction only | | I-frame count > 0 and < N | Hybrid: I-frames + uniform fill | | I-frame count = 0 (e.g., some .webm) | Uniform time-based sampling only |

Risks and Mitigations

| Risk | Likelihood | Mitigation | |---|---|---| | select='eq(pict_type\,I)' rejected on ffmpeg < 4 | Low | Document minimum version (ffmpeg 4+); bail with version check if needed. | | scale=768:-2 fails on odd-dimension video | Low | Substitute scale=768:trunc(ow/a/2)*2; documented in Step 5. | | Tesseract produces garbage on non-English UI text | Medium | Default is -l eng; user can override with a language code. | | Very short clips (< 1 s) yield 0 frames | Low | Bail gate in Step 3. | | Claude vision hallucinates text not in frames | Low | Cross-reference OCR output when available; note disagreements. | | Linear attachment URL requires authenticated download | Medium | Use the Linear MCP tool to obtain a pre-signed URL rather than a raw curl. |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mthines
• Source: mthines/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.