Feature Request: Improved Multi-Language Audio Transcription

[ilhamsyahids]

Problem

Whisper.cpp excels at transcribing audio in a single, specified language. However, when an audio input contains speech in multiple languages, the current transcription sometimes quality degrades significantly. This often manifests as:

• Skipping speech segments: Portions of the audio, particularly when language switches occur, are not transcribed at all.
• Unrelated results: The model may attempt to transcribe in the primary identified language, leading to nonsensical or incorrect text for segments in other languages.

Proposed Solutions

I propose adding functionality to whisper.cpp to handle multi-language audio inputs. I've explored a couple of approaches and believe they offer promising avenues for improvement:

1. Segment-level Language Detection (VAD + Auto-Detect):

   • Concept: Utilize a Voice Activity Detection (VAD) mechanism to segment the audio into smaller, speech-containing chunks. For each segment, independently perform language detection and then apply the transcription with the detected language.
   • Current Efforts: I have already experimented with this approach externally by using VAD to split segments and then performing auto-language detection on each segment before passing it to whisper.cpp. While this provides a workaround, integrating it directly into whisper.cpp would offer a more streamlined and optimized solution.

2. Constrained Language Probability for a List of Known Languages:

   • Concept: Instead of full auto-detection on every segment, allow the user to provide a predefined list of possible languages present in the audio (e.g., ["en", "fr", "de"]). For each speech segment, the model would then calculate probabilities only for these specified languages, selecting the most probable one for transcription. This could be particularly efficient if the number of potential languages is small.
   • Initial Scope: For an initial implementation, focusing on bilingual audio (e.g., providing two possible languages) would be a good starting point to demonstrate feasibility.

Benefits

• Significantly improved transcription accuracy for multilingual audio.
• Wider applicability in diverse real-world use cases.
• Reduced need for external pre-processing steps.

I believe these approaches are technically feasible.

Happy to contribute a PR to implement this feature. I would appreciate any guidance or feedback on the preferred implementation strategy and integration points within the codebase.

[mrfragger]

Concept: Utilize a Voice Activity Detection (VAD) mechanism to segment the audio into smaller, speech-containing chunks. For each segment, independently perform language detection and then apply the transcription with the detected language.

yea I've been removing silence with ffmpeg and always processing 30 second chunks at a time....do 50+ hour audiobooks and it works well. You are correct doing English only it's a much easier task to avoid hallucinations and other stuff. Usually do English and Arabic. Sometimes choose English and let it pick up Arabic but that doesn't work that great when the 30 second chunk is all Arabic though. So last few months just do auto and let it detect language.

[ilhamsyahids]

always processing 30 second chunks at a time

I prefer VAD (Voice Activity Detection) or silence-based splitting over fixed 30-second chunks. Fixed chunks can often cut sentences mid-way, losing context, and they struggle when a single chunk has multiple languages, leading to skipped speech or poor results. However depending on your use-case it might be enough using that.

Another key optimization is providing a list of possible languages. If we know the audio is only, say, English and Indonesian, telling the model that can prevent it from misidentifying similar languages, like Malay for Indonesian. Its similar but still having conflicted words. This approach improves accuracy by narrowing down the language choices.

[ilhamsyahids]

Hello, did you guys have a chance to look at this? @ggerganov @slaren @danbev

Thanks