Video Dubbing

Video Dubbing & Subtitle Sync

Complete workflow: ASR → translate → TTS → merge audio → burn subtitles

Pipeline

视频 → 抽音轨(16kHz) → ASR(whisper) → AI翻译 → TTS(dub_segments) → 合并音轨 → 烧录硬字幕

Workflow

0. Prerequisite Check

Ensure whisper + moviepy + mlx-audio are installed:

pip install openai-whisper moviepy 2>/dev/null
# mlx_audio for TTS:
pip install mlx-audio 2>/dev/null  # Apple Silicon

1. Extract Audio & Transcribe (ASR)

Extract 16kHz mono audio, then run whisper for accurate per-segment timestamps:

ffmpeg -y -i input.mp4 -ar 16000 -ac 1 -c:a pcm_s16le audio16k.wav

whisper audio16k.wav \
  --model large-v3-turbo \
  --language  \
  --output_format srt \
  --output_dir .

Output: audio16k.srt with native timestamps matching the video.

> Use large-v3-turbo for best accuracy. If you want even better Chinese recognition, install Qwen3-ASR (pip install qwen-asr) and use scripts/qwen3_asr.py instead.

2. Translate Subtitles (AI does this)

Read the SRT, translate each segment's text into the target language. Output one line per SRT segment, preserving order exactly.

• Count SRT segments first (e.g. 27 segments → 27 translated lines)
• Save as translated.txt in the working directory
• Verify line count matches SRT segment count before proceeding

3. Generate Dubbed Audio

python3 scripts/dub_segments.py audio16k.srt translated.txt dubbing.wav subtitle_synced.srt --lang 

What it does:

• Generates TTS per segment with voice consistency (first segment = voice reference)
• Preserves original timestamps — subtitles stay synced with video
• Smart speed adjustment: only adjusts segments that overflow their slot (atempo 0.88–1.20)
• Bridges adjacent gaps <1s for natural flow

4. Merge Dubbed Audio into Video

ffmpeg -y \
  -i input.mp4 \
  -i dubbing.wav \
  -map 0:v:0 -map 1:a:0 \
  -c:v copy -c:a aac -b:a 192k \
  -shortest \
  output_temp.mp4

Verify audio replaced:

ffprobe -v error -show_entries stream=codec_type -of csv=p=0 output_temp.mp4
# Should show: video, audio

5. Burn Hard Subtitles into Video

python3 scripts/burn_subtitles.py output_temp.mp4 subtitle_synced.srt output_final.mp4

This renders translated subtitles permanently into the video frame (hardcoded).

6. Cleanup (Optional)

rm -f audio16k.wav audio16k.srt translated.txt dubbing.wav subtitle_synced.srt output_temp.mp4

Output Files

| File | Description | |------|-------------| | output_final.mp4 | Final video with dubbed audio + hardcoded subtitles | | subtitle_synced.srt | Synced subtitles (original timestamps + translated text) | | dubbing.wav | Per-segment aligned dubbing track |

Key Design Decisions

| Principle | Why | |-----------|-----| | Preserve original ASR timestamps | They match the video's visual cues natively. Re-ASR on dubbing drifts. | | Per-segment speed adjustment (0.88–1.20x) | Global atempo on the whole dubbing sounds robotic. Per-segment is natural. | | Hard subtitles (burned in) | Soft subtitles don't work on all platforms. Burn them so they always show. | | 16kHz mono for ASR | Whisper expects this. Higher sample rate wastes compute without improving accuracy. | | Voice consistency via first segment reference | mlx-tts supports --ref_audio. First segment sets the voice for all others. | | No Demucs by default | For typical narration/speech videos, background music removal is unnecessary overhead. | | large-v3-turbo model | Best accuracy/speed tradeoff. Significantly better than base for proper nouns and fast speech. |

Dependencies

| Component | Install | Purpose | |-----------|---------|---------| | ffmpeg | brew install ffmpeg | Audio extraction, merging | | openai-whisper | pip install openai-whisper | Speech-to-text (ASR) | | moviepy | pip install moviepy | Burn subtitles into video | | mlx-audio | pip install mlx-audio | TTS on Apple Silicon | | qwen-asr | pip install qwen-asr (optional) | Better Chinese ASR |

Common Mistakes to Avoid

| Mistake | Why It Fails | Correct Approach | |---------|-------------|------------------| | ffmpeg without -map | Picks original audio (video source's audio stream), dubbing ignored | Always use -map 0:v:0 -map 1:a:0 | | Global atempo on entire dubbing | All speech slows uniformly (EN→ZH ~0.8x), sounds robotic | Per-segment alignment (dub_segments.py) | | Re-ASR on dubbed audio for timestamps | Timestamps drift from visual cues | Reuse original Step 1 timestamps | | Translation line count ≠ SRT segments | dub_segments.py requires strict 1:1 mapping | Count SRT segments first, match exactly | | Use whisper base/tiny model | Proper nouns wrong, fast speech missed | Use large-v3-turbo or turbo | | Burn subtitles without checking FFmpeg libass | subtitles= filter silently fails | Use scripts/burn_subtitles.py (moviepy) | | Translate segments out of order | Subtitles play at wrong times | Keep segment order: 1 translated line per SRT segment | | Skip verification | Audio may not have been replaced | Run ffprobe to check both audio + video streams exist |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: renky1025
• Source: renky1025/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.