Leveraging the Objective Intelligibility and Noise Estimation to Improve Conformer-Based MetricGAN

Abstract

Conformer-based MetricGAN (CMGAN) is a deep neural network (DNN)-based speech enhancement (SE) method that uses time-frequency (TF) domain features to learn a novel conformer-wise generative network, and it has demonstrated excellent SE performance in terms of various perceptual evaluation metrics.In this study, we propose to revise CMGAN along three directions. To begin, we incorporate phone-fortified perceptual loss (PFPL) into its loss function. The PFPL is calculated using latent representations of speech from the wav2vec module. With PFPL as part of the loss function can effectively use perceptual and linguistic speech information to direct CMGAN model training. Next, we revise the discriminator output by adding the STOI values. The original discriminator is trained to estimate the enhanced PESQ score by taking both clean and enhanced spectrum as inputs as well as the associated PESQ label. In other words, the initial discriminator only takes into account the PESQ score. By further considering STOI, we expect to improve the discriminator. Finally, we add noise label estimation to the entire CMGAN framework. The original CMGAN only calculates the disparity between the estimated value provided by the model and the clean target with clean labels. Instead, we further take into account noise estimation loss, which can show the discrepancy between the predicted noise and the noise label.The Voicebank-Demand dataset is used for the evaluation experiments. According to the experimental results, the revised CMGAN outperforms the original by gaining greater scores on objective perceptual metrics including PESQ and STOI. As a result, we confirm the success of the presented revisions in CMGAN.