RefineNet-Based Speech Enhancement with Composite Metric Loss Training

Speech enhancement is a task to improve the quality and intelligibility of degraded speech. Recent work shows convolutional neural network (CNN) with encoder-decoder architecture can achieve better performance with fewer parameters than the feed-forward neural network (FNN) and recurrent neural network (RNN). It inspires us to build a CNN model based on state-of-the-art encoder-decoder architecture, RefineNet, whose effectiveness has been proved for high-resolution semantic segmentation. In this work, RefineNet is used to exploit multi-level time-frequency features for generating high-level features. Furthermore, some works concern that the inconsistency between evaluation metrics and training loss may result in failing to obtain the optimal model by training. Therefore, we take metrics as loss and composite them. Furthermore, the waveform MSE loss, which contains phase information, is also composited to compensate for using noisy phase to reconstruct enhanced speech. Experiments show our model achieves high quality and intelligibility and outperform baselines, especially at a low signal-to-noise ratio.