Speech emotion recognition using multi resolution Hilbert transform based spectral and entropy features

Abstract

Speech emotion recognition (SER) is essential for addressing many personal and professional challenges in our everyday lives. The application of SER has shown potential in a number of domains, such as medical intervention, fortification of security systems, online marketing and educational platforms, personal communication, strengthening of devices and human interaction, and numerous other domains. Due to its extensive variety of applications, this subject has attracted the attention of several researchers for more than three decades. The performance of SER can be improved by adopting a suitable methodology for extracting the feature and using it to classify speech emotion. In our study, we used a novel technique known as the multi-resolution Hilbert transform (MRHT) method to extract the speech feature. We used the multi-resolution signal decomposition (MRSD) method to break down the speech signal frame (SSF) into a number of sub-frequency band signals, which are called modes or intrinsic mode functions (IMFs). Then, Hilbert transform (HT) is applied to each IMF signal to find the MRHT-based instantaneous amplitude (MRHIA) and MRHT-based instantaneous frequency (MRHIF) signal vectors. Features such as MRHT-based approximate entropy (MRHAE), MRHT-based permutation entropy (MRHPE), MRHT-based increment entropy (MRHIE), MRHT-based spectral entropy (MRHSE), and MRHT-based sample entropy (MRHSME) were calculated using each MRHIA and MRHIF signal vectors and the mel frequency cepstral coefficient (MFCC) feature extracted using the speech signals. The combinations of the proposed MRHT-based features (MRHAE + MRHPE + MRHIE + MRHSE + MRHSME) are known as the MRHT-based entropy feature (MRHEF). Subsequently, the MRHEF and MFCC features are used both alone and in conjunction to categorize speech emotion using a deep neural network (DNN) classifier. This results in emotion classification accuracies of 89.67%, 85.42%, and 83.48% for the EMO-DB, EMOVO, and SAVEE datasets, respectively. Comparing our experimental results with the other approaches, we found that the proposed feature combinations (MFCC + MRHEF) using a DNN classifier outperformed the state-of-the-art methods in SER.