Faster Whisper

Faster Whisper

Local speech-to-text using faster-whisper — a CTranslate2 reimplementation of OpenAI's Whisper that runs 4-6x faster with identical accuracy. With GPU acceleration, expect ~20x realtime transcription (a 10-minute audio file in ~30 seconds).

When to Use

Use this skill when you need to:

• Transcribe audio/video files — meetings, interviews, podcasts, lectures, YouTube videos
• Generate subtitles — SRT, VTT, ASS, LRC, or TTML broadcast-standard subtitles
• Identify speakers — diarization labels who said what (--diarize)
• Transcribe from URLs — YouTube links and direct audio URLs (auto-downloads via yt-dlp)
• Transcribe podcast feeds — --rss fetches and transcribes episodes
• Batch process files — glob patterns, directories, skip-existing support; ETA shown automatically
• Convert speech to text locally — no API costs, works offline (after model download)
• Translate to English — translate any language to English with --translate
• Do multilingual transcription — supports 99+ languages with auto-detection
• Transcribe a batch of files in different languages — --language-map assigns a different language per file
• Transcribe multilingual audio — --multilingual for mixed-language audio
• Transcribe audio with specific terms — use --initial-prompt for jargon-heavy content or any other terms to look out for
• Preprocess noisy audio (before transcription) — --normalize and --denoise before transcription
• Stream output — --stream shows segments as they're transcribed
• Clip time ranges — --clip-timestamps to transcribe specific sections
• Search the transcript — --search "term" finds all timestamps where a word/phrase appears
• Detect chapters — --detect-chapters finds section breaks from silence gaps
• Export speaker audio — --export-speakers DIR saves each speaker's turns as separate WAV files
• Spreadsheet output — --format csv produces a properly-quoted CSV with timestamps

Trigger phrases: "transcribe this audio", "convert speech to text", "what did they say", "make a transcript", "audio to text", "subtitle this video", "who's speaking", "translate this audio", "translate to English", "find where X is mentioned", "search transcript for", "when did they say", "at what timestamp", "add chapters", "detect chapters", "find breaks in the audio", "table of contents for this recording", "TTML subtitles", "DFXP subtitles", "broadcast format subtitles", "Netflix format", "ASS subtitles", "aegisub format", "advanced substation alpha", "mpv subtitles", "LRC subtitles", "timed lyrics", "karaoke subtitles", "music player lyrics", "HTML transcript", "confidence-colored transcript", "color-coded transcript", "separate audio per speaker", "export speaker audio", "split by speaker", "transcript as CSV", "spreadsheet output", "transcribe podcast", "podcast RSS feed", "different languages in batch", "per-file language", "transcribe in multiple formats", "srt and txt at the same time", "output both srt and text", "remove filler words", "clean up ums and uhs", "strip hesitation sounds", "remove you know and I mean", "transcribe left channel", "transcribe right channel", "stereo channel", "left track only", "wrap subtitle lines", "character limit per line", "max chars per subtitle", "detect paragraphs", "paragraph breaks", "group into paragraphs", "add paragraph spacing"

⚠️ Agent guidance — keep invocations minimal:

CORE RULE: default command (./scripts/transcribe audio.mp3) is the fastest path — add flags only when the user explicitly asks for that capability.

Transcription:

• Only add --diarize if the user asks "who said what" / "identify speakers" / "label speakers"
• Only add --format srt/vtt/ass/lrc/ttml if the user asks for subtitles/captions in that format
• Only add --format csv if the user asks for CSV or spreadsheet output
• Only add --word-timestamps if the user needs word-level timing
• Only add --initial-prompt if there's domain-specific jargon to prime
• Only add --translate if the user wants non-English audio translated to English
• Only add --normalize/--denoise if the user mentions bad audio quality or noise
• Only add --stream if the user wants live/progressive output for long files
• Only add --clip-timestamps if the user wants a specific time range
• Only add --temperature 0.0 if the model is hallucinating on music/silence
• Only add --vad-threshold if VAD is aggressively cutting speech or including noise
• Only add --min-speakers/--max-speakers when you know the speaker count
• Only add --hf-token if the token is not cached at ~/.cache/huggingface/token
• Only add --max-words-per-line for subtitle readability on long segments
• Only add --filter-hallucinations if the transcript contains obvious artifacts (music markers, duplicates)
• Only add --merge-sentences if the user asks for sentence-level subtitle cues
• Only add --clean-filler if the user asks to remove filler words (um, uh, you know, I mean, hesitation sounds)
• Only add --channel left|right if the user mentions stereo tracks, dual-channel recordings, or asks for a specific channel
• Only add --max-chars-per-line N when the user specifies a character limit per subtitle line (e.g., "Netflix format", "42 chars per line"); takes priority over --max-words-per-line
• Only add --detect-paragraphs if the user asks for paragraph breaks or structured text output; --paragraph-gap (default 3.0s) only if they want a custom gap
• Only add --speaker-names "Alice,Bob" when the user provides real names to replace SPEAKER_1/2 — always requires --diarize
• Only add --hotwords WORDS when the user names specific rare terms not well served by --initial-prompt; prefer --initial-prompt for general domain jargon
• Only add --prefix TEXT when the user knows the exact words the audio starts with
• Only add --detect-language-only when the user only wants to identify the language, not transcribe
• Only add --stats-file PATH if the user asks for performance stats, RTF, or benchmark info
• Only add --parallel N for large CPU batch jobs; GPU handles one file efficiently on its own — don't add for single files or small batches
• Only add --retries N for unreliable inputs (URLs, network files) where transient failures are expected
• Only add --burn-in OUTPUT only when user explicitly asks to embed/burn subtitles into the video; requires ffmpeg and a video file input
• Only add --keep-temp when the user may re-process the same URL to avoid re-downloading
• Only add --output-template when user specifies a custom naming pattern in batch mode
• Multi-format output (--format srt,text): only when user explicitly wants multiple formats in one pass; always pair with -o
• Any word-level feature auto-runs wav2vec2 alignment (~5-10s overhead)
• --diarize adds ~20-30s on top of that

Search:

• Only add --search "term" when the user asks to find/locate/search for a specific word or phrase in audio
• --search replaces the normal transcript output — it prints only matching segments with timestamps
• Add --search-fuzzy only when the user mentions approximate/partial matching or typos
• To save search results to a file, use -o results.txt

Chapter detection:

• Only add --detect-chapters when the user asks for chapters, sections, a table of contents, or "where does the topic change"
• Default --chapter-gap 8 (8-second silence = new chapter) works for most podcasts/lectures; tune down for dense content
• --chapter-format youtube (default) outputs YouTube-ready timestamps; use json for programmatic use
• Always use --chapters-file PATH when combining chapters with a transcript output — avoids mixing chapter markers into the transcript text
• If the user only wants chapters (not the transcript), pipe stdout to a file with -o /dev/null and use --chapters-file
• Batch mode limitation: --chapters-file takes a single path — in batch mode, each file's chapters overwrite the previous. For batch chapter detection, omit --chapters-file (chapters print to stdout under === CHAPTERS (N) ===) or use a separate run per file

Speaker audio export:

• Only add --export-speakers DIR when the user explicitly asks to save each speaker's audio separately
• Always pair with --diarize — it silently skips if no speaker labels are present
• Requires ffmpeg; outputs SPEAKER_1.wav, SPEAKER_2.wav, etc. (or real names if --speaker-names is set)

Language map:

• Only add --language-map in batch mode when the user has confirmed different languages across files
• Inline format: "interview*.mp3=en,lecture*.mp3=fr" — fnmatch globs on filename
• JSON file format: @/path/to/map.json where the file is {"pattern": "lang_code"}

RSS / Podcast:

• Only add --rss URL when the user provides a podcast RSS feed URL
• Default fetches 5 newest episodes; --rss-latest 0 for all; --skip-existing to resume safely
• Always use -o with --rss — without it, all episode transcripts print to stdout concatenated, which is hard to use; each episode gets its own file when -o is set

Output format for agent relay:

• Search results (--search) → print directly to user; output is human-readable
• Chapter output → if no --chapters-file, chapters appear in stdout under === CHAPTERS (N) === header after the transcript; with --format json, chapters are also embedded in the JSON under "chapters" key
• Subtitle formats (SRT, VTT, ASS, LRC, TTML) → always write to -o file; tell the user the output path, never paste raw subtitle content
• Data formats (CSV, HTML, TTML, JSON) → always write to -o file; tell the user the output path, don't paste raw XML/CSV/HTML
• ASS format → for Aegisub, VLC, mpv; write to file and tell user they can open it in Aegisub or play it in VLC/mpv
• LRC format → timed lyrics for music players (Foobar2000, AIMP, VLC); write to file
• Multi-format (--format srt,text) → requires -o ; each format goes to a separate file; tell user all paths written
• JSON format → useful for programmatic post-processing; not ideal to paste in full to user
• Text/transcript → safe to show directly to user for short files; summarise for long ones
• Stats output (--stats-file) → summarise key fields (duration, processing time, RTF) for the user rather than pasting raw JSON
• Language detection (--detect-language-only) → print the result directly; it's a single line
• ETA is printed automatically to stderr for batch jobs; no action needed

When NOT to use:

• Cloud-only environments without local compute
• Files need_accuracy;

needaccuracy -> largev3 [label="yes"]; needaccuracy -> multilingual [label="no"]; multilingual -> largeturbo [label="yes"]; multilingual -> resourceconstrained [label="no (English)"]; resourceconstrained -> distilsmall [label="mobile/edge"]; resourceconstrained -> distilmedium [label="some limits"]; resourceconstrained -> distil_large [label="no"]; }

### Model Table

#### Standard Models (Full Whisper)

| Model                  | Size  | Speed     | Accuracy  | Use Case                           |
| ---------------------- | ----- | --------- | --------- | ---------------------------------- |
| `tiny` / `tiny.en`     | 39M   | Fastest   | Basic     | Quick drafts                       |
| `base` / `base.en`     | 74M   | Very fast | Good      | General use                        |
| `small` / `small.en`   | 244M  | Fast      | Better    | Most tasks                         |
| `medium` / `medium.en` | 769M  | Moderate  | High      | Quality transcription              |
| `large-v1/v2/v3`       | 1.5GB | Slower    | Best      | Maximum accuracy                   |
| `large-v3-turbo`       | 809M  | Fast      | Excellent | High accuracy (slower than distil) |

#### Distilled Models (~6x Faster, ~1% WER difference)

| Model                   | Size | Speed vs Standard | Accuracy  | Use Case                           |
| ----------------------- | ---- | ----------------- | --------- | ---------------------------------- |
| **`distil-large-v3.5`** | 756M | ~6.3x faster      | 7.08% WER | **Default, best balance**          |
| `distil-large-v3`       | 756M | ~6.3x faster      | 7.53% WER | Previous default                   |
| `distil-large-v2`       | 756M | ~5.8x faster      | 10.1% WER | Fallback                           |
| `distil-medium.en`      | 394M | ~6.8x faster      | 11.1% WER | English-only, resource-constrained |
| `distil-small.en`       | 166M | ~5.6x faster      | 12.1% WER | Mobile/edge devices                |

`.en` models are English-only and slightly faster/better for English content.

> **Note for distil models:** HuggingFace recommends disabling `condition_on_previous_text` for all distil models to prevent repetition loops. The script **auto-applies** `--no-condition-on-previous-text` whenever a `distil-*` model is detected. Pass `--condition-on-previous-text` to override if needed.

## Custom & Fine-tuned Models

WhisperModel accepts local CTranslate2 model directories and HuggingFace repo names — no code changes needed.

### Load a local CTranslate2 model

```bash
./scripts/transcribe audio.mp3 --model /path/to/my-model-ct2

Convert a HuggingFace model to CTranslate2

pip install ctranslate2
ct2-transformers-converter \
  --model openai/whisper-large-v3 \
  --output_dir whisper-large-v3-ct2 \
  --copy_files tokenizer.json preprocessor_config.json \
  --quantization float16
./scripts/transcribe audio.mp3 --model ./whisper-large-v3-ct2

Load a model by HuggingFace repo name (auto-downloads)

./scripts/transcribe audio.mp3 --model username/whisper-large-v3-ct2

Custom model cache directory

By default, models are cached in ~/.cache/huggingface/. Use --model-dir to override:

./scripts/transcribe audio.mp3 --model-dir ~/my-models

Setup

Linux / macOS / WSL2

# Base install (creates venv, installs deps, auto-detects GPU)
./setup.sh

# With speaker diarization support
./setup.sh --diarize

Requirements:

• Python 3.10+
• ffmpeg is not required for basic transcription — PyAV (bundled with faster-whisper) handles audio decoding. ffmpeg is only needed for --burn-in, --normalize, and --denoise.
• Optional: yt-dlp (for URL/YouTube input)
• Optional: pyannote.audio (for --diarize, installed via setup.sh --diarize)

Platform Support

| Platform | Acceleration | Speed | | ---------------------- | ------------ | ---------------- | | Linux + NVIDIA GPU | CUDA | ~20x realtime 🚀 | | WSL2 + NVIDIA GPU | CUDA | ~20x realtime 🚀 | | macOS Apple Silicon | CPU\* | ~3-5x realtime | | macOS Intel | CPU | ~1-2x realtime | | Linux (no GPU) | CPU | ~1x realtime |

\*faster-whisper uses CTranslate2 which is CPU-only on macOS, but Apple Silicon is fast enough for practical use.

GPU Support (IMPORTANT!)

The setup script auto-detects your GPU and installs PyTorch with CUDA. Always use GPU if available — CPU transcription is extremely slow.

| Hardware | Speed | 9-min video | | -------------- | -------------- | ----------- | | RTX 3070 (GPU) | ~20x realtime | ~27 sec | | CPU (int8) | ~0.3x realtime | ~30 min |

> RTX 3070 tip: Use --compute-type int8_float16 for hybrid quantization — saves ~1GB VRAM with minimal quality loss. Ideal for running diarization alongside transcription.

If setup didn't detect your GPU, manually install PyTorch with CUDA:

# For CUDA 12.x
uv pip install --python .venv/bin/python torch --index-url https://download.pytorch.org/whl/cu121

# For CUDA 11.x
uv pip install --python .venv/bin/python torch --index-url ht

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [ThePlasmak](https://github.com/ThePlasmak)
- **Source:** [ThePlasmak/faster-whisper](https://github.com/ThePlasmak/faster-whisper)
- **License:** MIT
- **Homepage:** https://clawhub.ai/ThePlasmak/faster-whisper

Install and usage instructions live in the source repository linked above.