Progressive Skip Connection Improves Consistency of Diffusion-Based Speech Enhancement

Abstract

Recent advancements in generative modeling have successfully integrated denoising diffusion probabilistic models (DDPMs) into the domain of speech enhancement (SE). Despite their considerable advantages in generalizability, ensuring semantic consistency of the generated samples with the condition signal remains a formidable challenge. Inspired by techniques addressing posterior collapse in variational autoencoders, we explore skip connections within diffusion-based SE models to improve consistency with condition signals. However, experiments reveal that simply adding skip connections is ineffective and even counterproductive. We argue that the independence between the predictive target and the condition signal causes this failure. To address this, we modify the training objective from predicting random Gaussian noise to predicting clean speech and propose a progressive skip connection strategy to mitigate the decrease in mutual information between the layer's output and the condition signal as network depth increases. Experiments on two standard datasets demonstrate the effectiveness of our approach in both seen and unseen scenarios.