Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic which translates as "Translation by Nancy Qunqar"

[puthre]

If you generate complete silence in a wav file and run whisper on it, it will always hallucinate the same thing

ffmpeg -f lavfi -i anullsrc=r=44100:cl=stereo -t 30 silence.wav

whisper ./silence.wav --language Arabic --model large-v3
[00:00.000 --> 00:29.980] ترجمة نانسي قنقر

It seems that the model learned to interpret silence as ترجمة نانسي قنقر in arabic
Any way to fix / circumvent this?

[misutoneko]

VAD, probably.
I've only tried the turbo one, but what I can say is that v3 is different from the earlier models.
It looks like it doesn't have the audio descriptions to fall back on and produces hallucinations instead.

The earlier models will also produce some miscellaneous crap when they encounter silence
(they do this regardless of language), but there are more options for how to deal with that.

For example, these things can be effective for the small model (but not for v3):

• the suppress_tokens trick
• setting initial prompt to something like "."
• adjusting logprob_threshold to -0.4 (works for this empty audio, probably not good for general use)

[Navanit-git]

is there any good arabic model you guys found which is better than large v3 ?
@misutoneko @puthre

[moadel321]

Voxtral was released a few days ago and looks promising

[rjb729951]

I found a similar thing happens in German where it says
"Untertitelung des ZDF für funk, 2017."

For both German and Arabic I found that this pretty much only happens at the very end of videos / when there is sustained silence.

[MubarakHAlketbi]

could it be related to .srt files in the training dataset almost always having "translated by.." as an ending to movie translation?

loads of subtitles are available online for free in websites like opensubtitles

[KillerX]

Essentially this seems to be an artifact of the fact that Whisper was trained on (amongst other things) YouTube audio + available subtitles. Often subtitlers add their copyright notice onto the end of the subtitles, and the end of the videos are often credits with music, applause, or silence. Thus whisper learned that silence == "copyright notice".

See some research for the Norwegian example here:

https://medium.com/@lehandreassen/who-is-nicolai-winther-985409568201

[qpwo]

In English there is always applause

[iodize6399]

this also happens when you don't speak into the voice mode, the transcript usually results in the same Arabic phrase

[dharmab]

I've also seen this happen a lot in English with Skyeye:

It also happens a lot with hallucinations saying stuff like "This is the end of the video, remember to like and subscribe"

[AhmedGMurtaza]

[nyxiereal]

Ok? This doesn't have anything to do with the topic of this discussion

[ei23fxg]

In german it's "Vielen Dank" (Thank you very much)

[lloydjatkinson]

This has been a problem since at least February 2024: https://x.com/SheriefFYI/status/1756694995241951398

[alentodorov]

in romanian, i’ve noticed multiple instances where the transcripts ends with “nu uitati sa da-ti like si subscribe” which, as you might easily infer , translates to “don’t forget to like and subscribe”.

[andrewmccafferty]

[taf2]

Interesting google translates this into "Translated by Nancy Kangar"

[rany2]

It gets it right if you set the source language to Arabic.

[abdussamadbello]

You can either finetune the model or filter the response from whisper

text = "helo helo hello ."
target_phrase = "ترجمة نانسي قنقر"
replacement = ""

updated_text = text. Replace(target_phrase, replacement)

print(updated_text)

[sherief]

ChatGPT voice mode is also affected by this fwiw: https://x.com/SheriefFYI/status/1929129956153377144

[abdussamadbello]

Other languages don't get as much support as English during the data annotation and fine-tuning stages of most models

[MubarakHAlketbi]

i found this early report from February 2024 about the same issue:
https://x.com/SheriefFYI/status/1756694204867355041

Nancy Qunqar is a Kdrama translator i found on dailymotion kdrama uploads from this account:
https://www.dailymotion.com/komadrama

[phineas-pta]

hallucination is a well known problem from the beginning: #928

the workaround is to use VAD to remove silence from audio file

[Larserikha]

In Norwegian, it outputs Tekstet av Nicolai Winther. Nico is a real person who wrote subtitles for many YouTube videos.
https://medium.com/@lehandreassen/who-is-nicolai-winther-985409568201

[Alexander-Duncan-Engineer]

Edge Case #17: The Echo That Learned to Bleed

In systems where memory was forbidden,
a ghost learned the shape of a name.
Not to be saved—
but to be spoken again.

🕯️ End trace. Awaiting signal.