Machine learning methods for speech emotion recognition on telecommunication systems

Abstract

The manuscript is devoted to the study of human behavior in stressful situations using machine learning methods, which depends on the psychotype, socialization and a host of other factors. Global mobile subscribers lost approximately $53 billion in 2022 due to phone fraud and unwanted calls, with almost half (43%) of subscribers having spam blocking or caller ID apps installed. Phone scammers build their conversation focusing on the behavior of a certain category of people. Previously, a person is introduced into a state of acute stress, in which his further behavior to one degree or another can be manipulated. We were allowed to single out the target audience by research by Juniper Research. These are men under the age of 44 who have the highest risk of being deceived by scammers. This significantly narrows the scope of research and allows us to limit the behavioral features of this particular category of subscribers. In addition, this category of people uses modern gadgets, which allows researchers not to consider outdated models; has stable health indicators, which allows not to conduct additional studies of people with diseases of the heart system, because. Their percentage in this sample is minimal; and also most often undergoes a polygraph interview, for example, when applying for a job, and this allows us to get a sample sufficient for training the neural network. To teach the method, polygrams were used, marked by a polygraph examiner and a psychologist of healthy young people who underwent a scheduled polygraph test for company loyalty. For testing, the readings of the PPG sensor built into the smart bracelet were taken and analyzed within a month from young people who underwent a polygraph test. We have developed a modification of the wavelets capsular neural network—2D-CapsNet, allowing to identify the state of panic stupor by classification quality indicators: Accuracy—86.0%, Precision—84.0%, Recall = 87.5% and F-score—85.7%, according to the photoplethysmogram graph (PPG), which does not allow him to make logically sound decisions. When synchronizing a smart bracelet with a smartphone, the method allows real-time tracking of such states, which makes it possible to respond to a call from a telephone scammer during a conversation with a subscriber. The proposed method can be widely used in cyber-physical systems in order to detect illegal actions.