T_SRNET: A multimodal model based on convolutional neural network for emotional speech enhancement

Abstract

Speech classification is a technology that can determine the emotional state conveyed by speech. It can support emotion-related applications and improve the human–computer interaction experience. However, the lack of high-quality speech annotation datasets makes it difficult for many models to provide sufficient data for training, resulting in poor model generalization performance. It is necessary to obtain more high-quality speech annotation datasets through the high-precision model. For example, there are many human emotional data in the image dataset that can be utilized to assist in speech emotional information recognition. In this study, a multimodal algorithm T_SRNET is proposed, which can assist speech emotion recognition by extracting image emotion features and converting them into spectrograms. Firstly, the face image data with emotions such as joy and sadness are transformed into the corresponding phonograms by the diffusion model. Secondly, the features can be extracted by using the speech feature extraction network SRNET based on the improved transform structure. Finally, the speech signal features are extracted, and the two features are fused before the decision is made to output the results. After ablation and contrast experiments, the accuracy of CREMA-D and IEMOCAP was improved by 2% and 1% respectively. Also it can be evaluated that the proposed model in this study can correlate image data with speech data, improve the quality of speech data tagging and enhance the performance of speech recognition.