Increasing WER & Validation Loss During Whisper Fine-Tuning

[monk1337]

Hi,
I've recently created a dataset using speech-to-text APIs on custom documents. The dataset consists of 1,000 audio samples, with 700 designated for training and 300 for testing. In total, this equates to about 4 hours of audio, where each clip is approximately 30 seconds long.

I'm attempting to fine-tune the Whisper small model with the help of HuggingFace's script, following the tutorial they've provided Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers.

Before diving into the fine-tuning, I evaluated the WER on OpenAI's pre-trained model, which stood at WER = 23.078%.

However, as my fine-tuning progresses, I'm observing some unexpected behavior:

As visible, the Validation Loss and WER are both on the rise during the fine-tuning phase. I'm at a bit of a loss here. Why might this be happening? Any insights or recommendations would be greatly appreciated.

Thank you in advance!

[imannalia22]

Hi, I'm facing a similar issue. Did you manage to find out why?

[gongouveia]

@monk1337 Did you select the correct language?
are you sure the wer is being computed correctly? you should normalise the tests.
are you padding the audios to 30 seconds, and sample rate of all audios to 16khz?

[abdulmughni-ahmad]

for me reducing the learning rate to a very low value worked to a degree.

learning rate = 1.75e-8

here are my other params

training_args = Seq2SeqTrainingArguments(
output_dir="./whisper-medium-arabic-finetuned",
per_device_train_batch_size=1,
per_device_eval_batch_size=1,
gradient_accumulation_steps=32,
learning_rate=1.75e-8,
warmup_steps=1000,
max_steps = 40000,
save_steps=1000,
eval_steps=1000,
logging_steps=50,
eval_strategy="steps",
save_total_limit=6,
bf16=True,
report_to="none",
load_best_model_at_end=True,
metric_for_best_model="wer",
greater_is_better=False,
eval_accumulation_steps=4,
predict_with_generate=True,
save_safetensors=True,
generation_max_length=120,
lr_scheduler_type="cosine",
max_grad_norm = 1,
)