Automatic Speech Recognition (ASR)

Automatic Speech Recognition (ASR)

Overview

After speaker diarization, you need to transcribe each speech segment to text. Whisper is the current state-of-the-art for ASR, with multiple model sizes offering different trade-offs between accuracy and speed.

When to Use

• After speaker diarization is complete
• Need to generate speaker-labeled transcripts
• Creating subtitles from audio segments
• Converting speech segments to text

Whisper Model Selection

Model Size Comparison

| Model | Size | Speed | Accuracy | Best For | |-------|------|-------|----------|----------| | tiny | 39M | Fastest | Lowest | Quick testing, low accuracy needs | | base | 74M | Fast | Low | Fast processing with moderate accuracy | | small | 244M | Medium | Good | Recommended balance | | medium | 769M | Slow | Very Good | High accuracy needs | | large-v3 | 1550M | Slowest | Best | Maximum accuracy |

Recommended: Use small or large-v3

For best accuracy (recommended for this task):

import whisper

model = whisper.load_model("large-v3")  # Best accuracy
result = model.transcribe(audio_path)

For balanced performance:

import whisper

model = whisper.load_model("small")  # Good balance
result = model.transcribe(audio_path)

Faster-Whisper (Optimized Alternative)

For faster processing with similar accuracy, use faster-whisper:

from faster_whisper import WhisperModel

# Use small model with CPU int8 quantization
model = WhisperModel("small", device="cpu", compute_type="int8")

# Transcribe
segments, info = model.transcribe(audio_path, beam_size=5)

# Process segments
for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

Advantages:

• Faster than standard Whisper
• Lower memory usage with quantization
• Similar accuracy to standard Whisper

Aligning Transcriptions with Diarization Segments

After diarization, you need to map Whisper transcriptions to speaker segments:

# After diarization, you have turns with speaker labels
turns = [
    {'start': 0.8, 'duration': 0.86, 'speaker': 'SPEAKER_01'},
    {'start': 5.34, 'duration': 0.21, 'speaker': 'SPEAKER_01'},
    # ...
]

# Run Whisper transcription
model = whisper.load_model("large-v3")
result = model.transcribe(audio_path)

# Map transcriptions to turns
transcripts = {}
for i, turn in enumerate(turns):
    turn_start = turn['start']
    turn_end = turn['start'] + turn['duration']
    
    # Find overlapping Whisper segments
    overlapping_text = []
    for seg in result['segments']:
        seg_start = seg['start']
        seg_end = seg['end']
        
        # Check if Whisper segment overlaps with diarization turn
        if seg_start  turn_start:
            overlapping_text.append(seg['text'].strip())
    
    # Combine overlapping segments
    transcripts[i] = ' '.join(overlapping_text) if overlapping_text else '[INAUDIBLE]'

Handling Empty or Inaudible Segments

# If no transcription found for a segment
if not overlapping_text:
    transcripts[i] = '[INAUDIBLE]'
    
# Or skip very short segments
if turn['duration'] < 0.3:
    transcripts[i] = '[INAUDIBLE]'

Language Detection

Whisper can auto-detect language, but you can also specify:

# Auto-detect (recommended)
result = model.transcribe(audio_path)

# Or specify language for better accuracy
result = model.transcribe(audio_path, language="en")

Best Practices

1. Use larger models for better accuracy: small minimum, large-v3 for best results
2. Align timestamps carefully: Match Whisper segments with diarization turns
3. Handle overlaps: Multiple Whisper segments may overlap with one diarization turn
4. Handle gaps: Some diarization turns may have no corresponding transcription
5. Post-process text: Clean up punctuation, capitalization if needed

Common Issues

1. Low transcription accuracy: Use larger model (small → medium → large-v3)
2. Slow processing: Use faster-whisper or smaller model
3. Misaligned timestamps: Check time alignment between diarization and transcription
4. Missing transcriptions: Check for very short segments or silence

Integration with Subtitle Generation

After transcription, combine with speaker labels for subtitles:

def generate_subtitles_ass(turns, transcripts, output_path):
    # ... header code ...
    
    for i, turn in enumerate(turns):
        start_time = format_time(turn['start'])
        end_time = format_time(turn['start'] + turn['duration'])
        speaker = turn['speaker']
        text = transcripts.get(i, "[INAUDIBLE]")
        
        # Format: SPEAKER_XX: text
        f.write(f"Dialogue: 0,{start_time},{end_time},Default,,0,0,0,,{speaker}: {text}\n")

Performance Tips

1. For accuracy: Use large-v3 model
2. For speed: Use faster-whisper with small model
3. For memory: Use faster-whisper with int8 quantization
4. Batch processing: Process multiple segments together if possible

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: benchflow-ai
• Source: benchflow-ai/skillsbench
• License: Apache-2.0
• Homepage: https://www.skillsbench.ai

Install and usage instructions live in the source repository linked above.