Video Toolkit

Video Toolkit

Comprehensive video processing skill combining editing capabilities with multi-modal analysis (visual frames + audio transcription).

When to Use This Skill

Activate this skill when:

• User shares a video file (.mov, .mp4, .avi, .mkv, .webm, etc.)
• User asks to analyze, summarize, or understand video content
• User requests video editing operations (clip, merge, split)
• User asks questions about what's in a video
• You encounter "cannot read binary files" errors when trying to read video files
• Guides agents to use proper video analysis workflow

What this means for users:

When you share a video file, Claude will automatically recognize it and offer to analyze it properly using the video-toolkit, rather than attempting to read the binary file directly.

Prerequisites

Required Dependencies:

1. FFmpeg - Video processing and frame extraction

• Installation handled by scripts/install_dependencies.sh
• Verify: ffmpeg -version

1. Python 3.8+ with virtual environment

• Installation handled by scripts/install_dependencies.sh

1. OpenAI Whisper - Speech transcription (local, no API key required)

• Installed via pip in .venv
• Model: base (good balance of speed/accuracy)
• Other models available: tiny.en, small, medium, large

1. Google Gemini API - Audio analysis and music detection

• Installed via pip in .venv
• Requires API key (see setup below)
• Model: gemini-2.5-flash

1. Shazam API - Music identification

• Installed via pip in .venv (shazamio)
• No API key required (uses public endpoint)

Setup:

Step 1: Install Dependencies

Run the installation script on first use:

bash ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/install_dependencies.sh

This creates a Python virtual environment at ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/ and installs the required packages (ffmpeg-python, openai-whisper, google-genai, shazamio).

Step 2: Configure API Keys

Set up Gemini API key for audio analysis:

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/setup_api_keys.py gemini YOUR_API_KEY

Get your Gemini API key from: https://aistudio.google.com/app/apikey

Optional: If you want to use RapidAPI's Shazam endpoint instead of the public one:

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/setup_api_keys.py shazam YOUR_RAPIDAPI_KEY

Note: Shazam/music identification works without an API key using shazamio's public endpoint.

How to Use This Skill

IMPORTANT: Check API Key Setup First

Before analyzing videos with audio, verify that the Gemini API key is configured:

• Config file location: /Users/emdash/Dev/claude-code-plugins/emdashcodes/.video-toolkit-config.json
• If the file doesn't exist or doesn't contain gemini.apiKey, run the setup first (see Prerequisites above)
• The scripts will error with clear setup instructions if the API key is missing

Multi-Modal Video Analysis Workflow

When analyzing a video file, follow this comprehensive workflow:

1. Frame Extraction

Extract visual frames using either interval mode or scene detection mode:

Interval Mode (time-based):

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_frames.py   

• interval_seconds: Time between frames (e.g., 2 for every 2 seconds)
• output_dir: Temporary directory for frames
• Best for: Consistent sampling, slideshows, time-based analysis

Scene Detection Mode (change-based):

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_frames.py  --scene-detect  [threshold]

• threshold: Sensitivity (0.0-1.0, optional, default: 0.02)
• Automatically detects visual changes
• Best for: Screen recordings, presentations, variable content

Choosing the Right Mode:

| Video Type | Recommended Mode | Threshold/Interval | Recommended Value | |------------|------------------|-------------------|-------------------| | Screen recordings | Interval | 2-3 seconds | Use 2-3s | | Presentations/Slides | Scene detection | 0.05 - 0.10 | Use 0.05 | | Movies/Hard cuts | Scene detection | 0.20 - 0.30 | Use 0.20 | | Surveillance/Static | Interval | 5-10 seconds | Use 5s | | Interviews/Dialogue | Interval | 3-5 seconds | Use 3s | | Action videos | Interval | 1-2 seconds | Use 1s |

Agent Decision Making:

• For UI/screen recordings: Use interval mode with 2-3s (captures static periods reliably)
• For presentations/slides: Use scene detection with 0.05 (good for slide transitions)
• For movies with hard cuts: Use scene detection with 0.20
• For talking heads/static: Use interval mode with 3-5s
• For action/movement: Use interval mode with 1-2s
• DEFAULT: Interval mode with 2s (safest for most videos)

IMPORTANT: Evaluate and Re-run if Needed

• After extraction, check the frame count - does it seem reasonable for the video duration?
• If you get too few frames (e.g., 100 frames), the threshold may be TOO LOW
• Think critically about the results, check some frames, and RE-RUN with different settings if necessary
• Don't proceed with analysis if frame extraction clearly failed or produced poor results

Scene Detection Limitation: Scene detection only captures frames when visual content changes significantly. If the video has long static periods (e.g., same screen for 30+ seconds), scene detection will miss that content entirely. For videos with static content, USE INTERVAL MODE instead with --mode interval --interval 2 or --interval 3.

2. Audio Extraction

Extract audio track using scripts/extract_audio.py:

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_audio.py  

• Converts to WAV format (Whisper-compatible)
• Preserves original audio quality
• Returns: Path to extracted WAV file

3. Audio Analysis (Sequential Workflow)

The audio analysis follows a sequential workflow to maximize accuracy and efficiency:

3a. Speech Transcription (Whisper - Local)

Transcribe speech using Whisper (runs locally, no API key required):

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/transcribe_audio.py  [model_name]

• model_name: Whisper model (default: base)
• tiny.en: Fastest, English only
• base: Good balance (recommended)
• small: Better accuracy
• medium: High accuracy
• large: Best accuracy, slowest
• Returns: Timestamped transcript with speaker diarization info
• Purpose: Extract speech/dialogue from the audio

3b. Audio Understanding (Gemini Audio API)

Analyze audio comprehensively using Gemini Audio API:

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/analyze_audio_gemini.py  [output_markdown]

• Analyzes full audio content (not just speech)
• Detects music with precise timestamps (MM:SS to MM:SS format)
• Identifies non-speech sounds (applause, ambient noise, sound effects)
• Translates foreign languages: If Whisper detects non-English speech, Gemini provides both original and English translation
• Returns:
• Markdown analysis saved to [output_markdown]
• JSON data automatically saved to gemini_audio.json in same directory
• JSON contains has_music flag and music_segments array for downstream processing
• Purpose: Understand what's in the audio beyond speech, detect if music is present, translate foreign languages

3c. Music Identification (Shazam - Conditional)

If Gemini detects music, identify songs using Shazam:

${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/identify_music.py   [output_markdown]

• `: Path to the geminiaudio.json file created by analyzeaudio_gemini.py`
• Only runs if Gemini detected music (checks has_music flag in JSON)
• Extracts music segments based on Gemini's timestamps using FFmpeg
• Identifies each segment with Shazam API
• Returns: Song title, artist, album, genre, year
• Purpose: Identify specific songs in the video

Sequential Workflow Summary:

1. Whisper (local)     → Speech transcription
2. Gemini Audio (API)  → Detect music + timestamps, analyze non-speech audio
3. FFmpeg              → Extract music segments (if music detected)
4. Shazam (API)        → Identify songs (if music detected)

This approach ensures:

• Whisper focuses on speech (what it does best, runs locally)
• Gemini provides overall audio understanding and music detection
• Shazam only processes clean music segments (better accuracy)
• No unnecessary API calls if no music is present

Gemini vs Shazam - Complementary Capabilities:

• Gemini Audio API: AI-powered audio understanding
• Recognizes music characteristics: genre, mood, instrumentation, tempo
• Example: "Rock/Alternative song with electric guitar and male vocals, energetic"
• May recognize well-known classical pieces (especially if mentioned in speech)
• Provides timestamps and musical context
• Does NOT identify specific tracks, artists, or albums

• Shazam: Audio fingerprint matching
• Identifies specific recordings: track title, artist, album, year
• Example: "Substitution - Silversun Pickups, Swoon (2009)"
• Requires at least 3 seconds of audio for reliable identification
• Provides Shazam URLs and metadata

Both work together: Gemini describes what the music sounds like, Shazam identifies which specific track it is.

3.5. Context Clarification with Intelligent Question Generation

CRITICAL STEP: After gathering initial data (frames, audio, music identification), ask the user context questions to understand the video's perspective and avoid misidentification.

Why this step is critical: Visual evidence alone can be ambiguous. For example, seeing streaming platform with a BRB screen could mean either:

• The user is streaming and stepped away (creator perspective)
• The user is watching someone else's stream (viewer perspective)

Only the user can clarify their relationship to the content.

---

Workflow: Analyze First, Then Ask

Step 1: Complete Initial Data Gathering

Before asking any questions, run the full analysis pipeline through Step 3c:

• ✓ Extract frames (Step 1)
• ✓ Extract and transcribe audio (Steps 2, 3a)
• ✓ Analyze audio with Gemini (Step 3b)
• ✓ Identify music with Shazam if detected (Step 3c)

Step 2: Build Contextual Evidence

Review what was gathered:

• Visual evidence: Sample 3-5 frames - what's visible? (UI, people, environments, screen content)
• Audio evidence: What's the audio content? (speech, music, ambient sounds)
• Music evidence: What songs were identified? With what timestamps?
• Conversation context: What has the user said about this video?
• User knowledge: What do you know about this user from conversation history and preferences?

Step 3: Ask Context Question (Q1) with Evidence

Present Q1 with context from your gathered evidence to help the user answer:

# Build context string from evidence
# Adapt summary based on what was found in the video

# Example 1: Stream Viewer
evidence_summary = """I've analyzed the video and found:
- Visual: Streaming platform interface with BRB screen, person visible, workspace shown
- Audio: Music playing ("Song Title" by Artist Name), minimal speech
- Duration: 1:57
"""

# Example 2: Tutorial/How-to
evidence_summary = """I've analyzed the video and found:
- Visual: Person's hands visible working with materials, step-by-step demonstrations
- Audio: Instructional narration explaining process, background music
- Duration: 8:34
"""

# Example 3: Screen Recording (Software Demo)
evidence_summary = """I've analyzed the video and found:
- Visual: VS Code editor with code visible, cursor movements, terminal commands
- Audio: Voiceover explaining code changes, keyboard typing sounds
- Duration: 5:12
"""

# Example 4: Screen Recording (Gameplay)
evidence_summary = """I've analyzed the video and found:
- Visual: Game interface with HUD elements, character movement, gameplay
- Audio: Game sounds, background music, occasional commentary
- Duration: 12:45
"""

# Example 5: Personal/Casual
evidence_summary = """I've analyzed the video and found:
- Visual: Person on camera in home setting, handheld/phone camera movement
- Audio: Person speaking directly to camera, ambient room sounds
- Duration: 2:18
"""

# Example 6: Personal/Casual (Phone Clip)
evidence_summary = """I've analyzed the video and found:
- Visual: Quick clip of outdoor scene, vertical/portrait orientation, casual framing
- Audio: Ambient sounds, brief speech, wind noise
- Duration: 0:43
"""

# Example 7: Event/Performance
evidence_summary = """I've analyzed the video and found:
- Visual: Stage with performers, audience visible, concert venue setting
- Audio: Live music performance, crowd noise, applause
- Duration: 3:56
"""

# Example 8: Presentation/Slides
evidence_summary = """I've analyzed the video and found:
- Visual: Slide deck with bullet points and diagrams, presenter occasionally visible
- Audio: Presenter speaking, slide transition sounds
- Duration: 18:24
"""

# Example 9: Casual Documentation (Workspace)
evidence_summary = """I've analyzed the video and found:
- Visual: Desk setup with monitors and equipment, informal camera angles
- Audio: Background music, ambient office sounds, no narration
- Duration: 1:34
"""

# Example 10: Behind-the-Scenes
evidence_summary = """I've analyzed the video and found:
- Visual: Production equipment, camera setup, people working on set
- Audio: Technical discussions, equipment sounds, music playing
- Duration: 4:27
"""

# Template for implementation:
# Select the most relevant description based on gathered evidence
def build_evidence_summary(frames_analysis, audio_analysis, music_data, duration):
    """
    Generate evidence summary based on analyzed content

    Returns contextual summary that helps user identify video type
    """
    visual_desc = describe_visual_content(frames_analysis)
    audio_desc = describe_audio_content(audio_analysis, music_data)

    return f"""I've analyzed the video and found:
- Visual: {visual_desc}
- Audio: {audio_desc}
- Duration: {format_duration(duration)}
"""

# Present Q1 with evidence context
AskUserQuestion(
    questions=[
        {
            "question": f"{evidence_summary}\n\nWhat type of video is this?",
            "header": "Video Type",
            "multiSelect": false,
            "options": [
                {
                    "label": "Tutorial/How-to",
                    "description": "Teaching or demonstrating something step-by-step"
                },
                {
                    "label": "Personal",
                    "description": "Personal video, daily life content, a moment or activity, clip from phone"
                },
                {
                    "label": "Screen recording",
                    "description": "Recording your own screen (software demo, debugging video, gameplay, etc.)"
                },
                {
                    "label": "Watching something",
                    "description": "Recording yourself viewing someone else's content"
                },
                {
                    "label": "Event/Performance",
                    "description": "Concert, presentation, ceremony, or live event"
                },
                {
                    "label": "Other/Not sure",
                    "description": "Type a custom description of what this vid

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [emdashcodes](https://github.com/emdashcodes)
- **Source:** [emdashcodes/claude-code-plugins](https://github.com/emdashcodes/claude-code-plugins)
- **License:** MIT
- **Homepage:** https://emdash.codes/claude-code/

Install and usage instructions live in the source repository linked above.