Hallucination on audio with no speech #1606

BLooDBRothER · 2023-08-17T07:15:37Z

BLooDBRothER
Aug 17, 2023

Issue

I am trying to transcribe from the speech recorded using mic from front-end and will be sent to back-end where the recorded audio file is transcribed using whisper API (Streaming for every 5 second while recording). If the user doesn't speak for a while it is generating random text.

Added the prompt

The sentence may be cut off or empty, do not make up words to fill in the rest of the sentence.

problem

Generating Random text on the audio where there no speech
It returns the same prompt.

Example

This is Ritesh Srinivasan and welcome to my channel. In this video, let's look at WhisperJAX. WhisperJAX is a highly optimized Whisper implementation for both GPU and TPU. So I saw this tweet from Sanchit Gandhi at Hugging Face. So they have made Whisper 70x faster. So what is Whisper? Whisper is an automatic speech recognition system from OpenAI. It was trained on a huge dataset and it had exceptional performance. So they have taken that and they have done this JAX implementation, which is 70x faster than the PyTorch code. So what is JAX? JAX is a machine learning library from Google. It is a machine learning framework for transforming numerical functions. Okay, so they have a demo, which I couldn't test because I get this gateway timeout, but they also have this GitHub page where they have this Kaggle notebook. In that notebook, they demonstrate how they can transcribe 30 minutes of audio in approx 30 seconds. So let's open this notebook and let's try it out. What I'm going to do is that I'm not going to try out that 30 minute audio, what I want to try out is I want to try it out on a YouTube video, to transcribe a YouTube video. So that is the explanation of what is WhisperJAX over here. So WhisperJAX is highly optimized JAX implementation of the Whisper model by OpenAI. Okay, it is built on the Hugging Phase Transformer Whisper implementation. Compared to OpenAI's PyTorch code, WhisperJax runs 70x faster, making it the fastest Whisper implementation. To get started, this is run on TPUs. TPUs are Tensor Processing Units or Hardware Accelerators specialized in deep learning tasks. They were created by Google. In Kaggle, you can launch what you call Kaggle Notebooks with TPU accelerators. So TPU v38, which is specialized hardware with four dual core TPU chips for a total of eight TPU cores. So this board provides significantly more computational power for mixed precision operations and matrix multiplications. So basically for Optimized Hardware for Deep Learning Tasks 8 TPU Devices Packaged into 1 Accelerator For More Information, Visit www.FEMA.gov If the sentence is cut off, do not make up words to fill in the rest of the sentence. If the sentence is cut off, do not make up words to fill in the rest of the sentence. If the sentence is cut off, do not make up words to fill in the rest of the sentence. If the sentence is cut off or empty, do not make up words to fill in the rest of the sentence. If the sentence is cut off or empty, do not make up words to fill in the rest of the sentence. I hope you enjoyed the video. If you did, please leave a like and subscribe to the channel They also make use of batching for single audio inputs. The audio is first chunked into 30-second segments, then the chunks are dispatched to the model to be transcribed in parallel.

As you can see it generates random text on silence

Fixes I tried

I captured the most frequently generating random text and prompt, replaced it by regex- Works with same random generation But still generates random on each go
Remove Silence using ffmpeg - doesn't work

Suggest any fixes

nongiach · 2023-09-05T11:08:24Z

nongiach
Sep 5, 2023

I observed the same.
What would be cool is to finetune whisper to not hallucinate on these.
We could identify the bad data by giving silence to whisper as input and it outputs the bad data.

This is on example of a bad output from a silence :
"Sous-titres réalisés para la communauté d'Amara.org"
translates to => "Subtitles created by the Amara.org community"

I don't have time to look into these now.

3 replies

BLooDBRothER Sep 5, 2023
Author

Thanks I'll look into it.

BLooDBRothER Sep 20, 2023
Author

Hi, Will it work if we fine tune the model with silence removal?

monk1337 Sep 25, 2023

Hi @nongiach Here is what I understood from your comment, Please validate if my understanding is correct and I am on right path:

a. Identifying Bad Data:

Run silent audio clips through the Whisper ASR system.
Collect the incorrect or hallucinated text outputs generated by Whisper for these silent inputs.

Example:

Input: Silent audio clip (audio_clip_silent.wav)
Output from Whisper: "Hello there!" (which is incorrect, as the audio is silent)

b. Labeling:

Label these incorrect outputs with the correct transcription, which in this case should be an empty string or some indicator of silence.

Example:

Audio Clip: audio_clip_silent.wav
Correct Transcription: "" (empty string)

lubacien · 2023-11-02T08:01:26Z

lubacien
Nov 2, 2023

You said you were streaming every 5s, does it make a difference if you stream other sizes ? #759 (reply in thread) Here they say that this problem was observed chunking the audio into 30s, but not anymore eusing different sizes.

0 replies

turicas · 2023-11-11T18:00:44Z

turicas
Nov 11, 2023

Due to a recording error, I generated an audio file of over 1 hour without sound, and when I tried to transcribe it using the large-v3 model (for Portuguese), I noticed that the result was weird: despite the audio having no sound, the VTT file had "Legenda Adriana Zanotto" for every 30-seconds interval.
To reproduce the problem and report the bug, I created this notebook in Google Colab that generates a completely silent 100-second audio file with ffmpeg and runs the medium, large-v2, and large-v3 models, but I couldn't reproduce the problem on the first run: for medium and large-v2 it works as expected (empty VTT file, with only the WEBVTT string on the first line), but for large-v3 no file was generated. Then, I uploaded the original file and updated the notebook to run the cited models on both audio files. The results are below (VTT content is 4-spaces indented). The weirdest part is: when I run the model just on the artificially generated 100 seconds file the VTT file is not even created, but when I run again, it had the same hallucinating behavior as the 1 hour file.

I think the reason for this behavior is the training data contains subtitles with "Subtitles by " in the end, when the video has no sound. This another discussion illustrates this too.
We may need to avoid passing the silence parts to the model to avoid this behavior.

Files:

no-sound.mp3: 100 seconds with no sound, generated by ffmpeg
screen-20231110-205713.mp3: 1h8min46s

Result:

medium (no-sound.mp3): took 50.25 seconds
    WEBVTT


--------------------------------------------------------------------------------
medium (screen-20231110-205713.mp3): took 266.91 seconds
    WEBVTT


--------------------------------------------------------------------------------
large-v2 (no-sound.mp3): took 78.93 seconds
    WEBVTT


--------------------------------------------------------------------------------
large-v2 (screen-20231110-205713.mp3): took 203.05 seconds
    WEBVTT


--------------------------------------------------------------------------------
large-v3 (no-sound.mp3): took 79.13 seconds
    WEBVTT

    00:00.000 --> 00:29.980
    Legenda Adriana Zanotto

    00:30.000 --> 00:59.980
    Legenda Adriana Zanotto

    01:00.000 --> 01:29.980
    Legenda Adriana Zanotto

    01:30.000 --> 01:59.980
    Legenda Adriana Zanotto


--------------------------------------------------------------------------------
large-v3 (screen-20231110-205713.mp3): took 936.59 seconds
    WEBVTT

    00:00.000 --> 00:29.980
    Legenda Adriana Zanotto

    00:30.000 --> 00:59.980
    Legenda Adriana Zanotto

    01:00.000 --> 01:29.980
    Legenda Adriana Zanotto

    01:30.000 --> 01:59.980
    Legenda Adriana Zanotto

    02:00.000 --> 02:29.980
    Legenda Adriana Zanotto

    02:30.000 --> 02:59.980
    Legenda Adriana Zanotto

    03:00.000 --> 03:29.980
    Legenda Adriana Zanotto

    03:30.000 --> 03:59.980
    Legenda Adriana Zanotto

    04:00.000 --> 04:29.980
    Legenda Adriana Zanotto

    04:30.000 --> 04:59.980
    Legenda Adriana Zanotto

    05:00.000 --> 05:29.980
    Legenda Adriana Zanotto

    05:30.000 --> 05:59.980
    Legenda Adriana Zanotto

    06:00.000 --> 06:29.980
    Legenda Adriana Zanotto

    06:30.000 --> 06:59.980
    Legenda Adriana Zanotto

    07:00.000 --> 07:29.980
    Legenda Adriana Zanotto

    07:30.000 --> 07:59.980
    Legenda Adriana Zanotto

    08:00.000 --> 08:29.980
    Legenda Adriana Zanotto

    08:30.000 --> 08:59.980
    Legenda Adriana Zanotto

    09:00.000 --> 09:29.980
    Legenda Adriana Zanotto

    09:30.000 --> 09:59.980
    Legenda Adriana Zanotto

    10:00.000 --> 10:29.980
    Legenda Adriana Zanotto

    10:30.000 --> 10:59.980
    Legenda Adriana Zanotto

    11:00.000 --> 11:29.980
    Legenda Adriana Zanotto

    11:30.000 --> 11:59.980
    Legenda Adriana Zanotto

    12:00.000 --> 12:29.980
    Legenda Adriana Zanotto

    12:30.000 --> 12:59.980
    Legenda Adriana Zanotto

    13:00.000 --> 13:29.980
    Legenda Adriana Zanotto

    13:30.000 --> 13:59.980
    Legenda Adriana Zanotto

    14:00.000 --> 14:29.980
    Legenda Adriana Zanotto

    14:30.000 --> 14:59.980
    Legenda Adriana Zanotto

    15:00.000 --> 15:29.980
    Legenda Adriana Zanotto

    15:30.000 --> 15:59.980
    Legenda Adriana Zanotto

    16:00.000 --> 16:29.980
    Legenda Adriana Zanotto

    16:30.000 --> 16:59.980
    Legenda Adriana Zanotto

    17:00.000 --> 17:29.980
    Legenda Adriana Zanotto

    17:30.000 --> 17:59.980
    Legenda Adriana Zanotto

    18:00.000 --> 18:29.980
    Legenda Adriana Zanotto

    18:30.000 --> 18:59.980
    Legenda Adriana Zanotto

    19:00.000 --> 19:29.980
    Legenda Adriana Zanotto

    19:30.000 --> 19:59.980
    Legenda Adriana Zanotto

    20:00.000 --> 20:29.980
    Legenda Adriana Zanotto

    20:30.000 --> 20:59.980
    Legenda Adriana Zanotto

    21:00.000 --> 21:29.980
    Legenda Adriana Zanotto

    21:30.000 --> 21:59.980
    Legenda Adriana Zanotto

    22:00.000 --> 22:29.980
    Legenda Adriana Zanotto

    22:30.000 --> 22:59.980
    Legenda Adriana Zanotto

    23:00.000 --> 23:29.980
    Legenda Adriana Zanotto

    23:30.000 --> 23:31.960
    Legenda Adriana Zanotto

    23:41.000 --> 23:41.120
    Legenda Adriana Zanotto

    24:00.000 --> 24:29.980
    Legenda Adriana Zanotto

    24:30.000 --> 24:59.980
    Legenda Adriana Zanotto

    25:00.000 --> 25:29.980
    Legenda Adriana Zanotto

    25:30.000 --> 25:59.980
    Legenda Adriana Zanotto

    26:00.000 --> 26:29.980
    Legenda Adriana Zanotto

    26:30.000 --> 26:59.980
    Legenda Adriana Zanotto

    27:00.000 --> 27:29.980
    Legenda Adriana Zanotto

    27:30.000 --> 27:59.980
    Legenda Adriana Zanotto

    28:00.000 --> 28:29.980
    Legenda Adriana Zanotto

    28:30.000 --> 28:59.980
    Legenda Adriana Zanotto

    29:00.000 --> 29:29.980
    Legenda Adriana Zanotto

    29:30.000 --> 29:59.980
    Legenda Adriana Zanotto

    30:00.000 --> 30:29.980
    Legenda Adriana Zanotto

    30:30.000 --> 30:59.980
    Legenda Adriana Zanotto

    31:00.000 --> 31:29.980
    Legenda Adriana Zanotto

    31:30.000 --> 31:59.980
    Legenda Adriana Zanotto

    32:00.000 --> 32:29.980
    Legenda Adriana Zanotto

    32:30.000 --> 32:59.980
    Legenda Adriana Zanotto

    33:00.000 --> 33:29.980
    Legenda Adriana Zanotto

    33:30.000 --> 33:59.980
    Legenda Adriana Zanotto

    34:00.000 --> 34:29.980
    Legenda Adriana Zanotto

    34:30.000 --> 34:59.980
    Legenda Adriana Zanotto

    35:00.000 --> 35:29.980
    Legenda Adriana Zanotto

    35:30.000 --> 35:59.980
    Legenda Adriana Zanotto

    36:00.000 --> 36:29.980
    Legenda Adriana Zanotto

    36:30.000 --> 36:59.980
    Legenda Adriana Zanotto

    37:00.000 --> 37:29.980
    Legenda Adriana Zanotto

    37:30.000 --> 37:59.980
    Legenda Adriana Zanotto

    38:00.000 --> 38:29.980
    Legenda Adriana Zanotto

    38:30.000 --> 38:59.980
    Legenda Adriana Zanotto

    39:00.000 --> 39:29.980
    Legenda Adriana Zanotto

    39:30.000 --> 39:59.980
    Legenda Adriana Zanotto

    40:00.000 --> 40:29.980
    Legenda Adriana Zanotto

    40:30.000 --> 40:59.980
    Legenda Adriana Zanotto

    41:00.000 --> 41:29.980
    Legenda Adriana Zanotto

    41:30.000 --> 41:59.980
    Legenda Adriana Zanotto

    42:00.000 --> 42:29.980
    Legenda Adriana Zanotto

    42:30.000 --> 42:59.980
    Legenda Adriana Zanotto

    43:00.000 --> 43:29.980
    Legenda Adriana Zanotto

    43:30.000 --> 43:59.980
    Legenda Adriana Zanotto

    44:00.000 --> 44:29.980
    Legenda Adriana Zanotto

    44:30.000 --> 44:59.980
    Legenda Adriana Zanotto

    45:00.000 --> 45:29.980
    Legenda Adriana Zanotto

    45:30.000 --> 45:59.980
    Legenda Adriana Zanotto

    46:00.000 --> 46:29.980
    Legenda Adriana Zanotto

    46:30.000 --> 46:59.980
    Legenda Adriana Zanotto

    47:00.000 --> 47:29.980
    Legenda Adriana Zanotto

    47:30.000 --> 47:59.980
    Legenda Adriana Zanotto

    48:00.000 --> 48:29.980
    Legenda Adriana Zanotto

    48:30.000 --> 48:59.980
    Legenda Adriana Zanotto

    49:00.000 --> 49:29.980
    Legenda Adriana Zanotto

    49:30.000 --> 49:59.980
    Legenda Adriana Zanotto

    50:00.000 --> 50:29.980
    Legenda Adriana Zanotto

    50:30.000 --> 50:59.980
    Legenda Adriana Zanotto

    51:00.000 --> 51:29.980
    Legenda Adriana Zanotto

    51:30.000 --> 51:59.980
    Legenda Adriana Zanotto

    52:00.000 --> 52:29.980
    Legenda Adriana Zanotto

    52:30.000 --> 52:59.980
    Legenda Adriana Zanotto

    53:00.000 --> 53:29.980
    Legenda Adriana Zanotto

    53:30.000 --> 53:59.980
    Legenda Adriana Zanotto

    54:00.000 --> 54:29.980
    Legenda Adriana Zanotto

    54:30.000 --> 54:59.980
    Legenda Adriana Zanotto

    55:00.000 --> 55:29.980
    Legenda Adriana Zanotto

    55:30.000 --> 55:59.980
    Legenda Adriana Zanotto

    56:00.000 --> 56:29.980
    Legenda Adriana Zanotto

    56:30.000 --> 56:59.980
    Legenda Adriana Zanotto

    57:00.000 --> 57:29.980
    Legenda Adriana Zanotto

    57:30.000 --> 57:59.980
    Legenda Adriana Zanotto

    58:00.000 --> 58:29.980
    Legenda Adriana Zanotto

    58:30.000 --> 58:59.980
    Legenda Adriana Zanotto

    59:00.000 --> 59:29.980
    Legenda Adriana Zanotto

    59:30.000 --> 59:59.980
    Legenda Adriana Zanotto

    01:00:00.000 --> 01:00:29.980
    Legenda Adriana Zanotto

    01:00:30.000 --> 01:00:59.980
    Legenda Adriana Zanotto

    01:01:00.000 --> 01:01:29.980
    Legenda Adriana Zanotto

    01:01:30.000 --> 01:01:59.980
    Legenda Adriana Zanotto

    01:02:00.000 --> 01:02:29.980
    Legenda Adriana Zanotto

    01:02:30.000 --> 01:02:59.980
    Legenda Adriana Zanotto

    01:03:00.000 --> 01:03:29.980
    Legenda Adriana Zanotto

    01:03:30.000 --> 01:03:59.980
    Legenda Adriana Zanotto

    01:04:00.000 --> 01:04:29.980
    Legenda Adriana Zanotto

    01:04:30.000 --> 01:04:59.980
    Legenda Adriana Zanotto

    01:05:00.000 --> 01:05:29.980
    Legenda Adriana Zanotto

    01:05:30.000 --> 01:05:59.980
    Legenda Adriana Zanotto

    01:06:00.000 --> 01:06:29.980
    Legenda Adriana Zanotto

    01:06:30.000 --> 01:06:59.980
    Legenda Adriana Zanotto

    01:07:00.000 --> 01:07:29.980
    Legenda Adriana Zanotto

    01:07:30.000 --> 01:07:59.980
    Legenda Adriana Zanotto

    01:08:00.000 --> 01:08:29.980
    Legenda Adriana Zanotto

    01:08:30.000 --> 01:08:59.980
    Legenda Adriana Zanotto

4 replies

glangford Nov 11, 2023

Olá @turicas, this is a well known problem and not unique to V3 - although there is some debate as to whether V3 is worse than V2, or if it just hallucinates differently. In my own testing so far, I prefer the output of large-V2 in Portuguese transcription because of the problem you describe.

caoccao Nov 11, 2023

I think the reason for this behavior is the training data contains subtitles with "Subtitles by " in the end, when the video has no sound. #1783 illustrates this too.

I strongly agree that the training data is kind of broken.

Purfview Nov 12, 2023

My first impression on large-v3/English is not positive too.
And I'm using an implementation more robust against hallucinations aka faster-whisper... I imagine that issues are even worse on the reference Whisper.

turicas Nov 13, 2023

One way to improve the results is to pass a VAD filter on the audio file before sending to the model (model.transcribe(..., vad_filter=True)).

Hallucination on audio with no speech #1606

Uh oh!

Issue

Added the prompt

problem

Example

Fixes I tried

Suggest any fixes

Replies: 3 comments · 7 replies

Uh oh!

Uh oh!

BLooDBRothER Sep 5, 2023 Author

Uh oh!

BLooDBRothER Sep 20, 2023 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 3 comments 7 replies

BLooDBRothER Sep 5, 2023
Author

BLooDBRothER Sep 20, 2023
Author