Sensitive Data Classification of Imbalanced Short Text Based on Probability Distribution BERT in Electric power industry

Abstract

The exploitation of big data in industrial fields faces several challenges, such as data privacy and security, data integration and interoperability, and data analysis and visualization. Data privacy and security is a major concern, as the data collected from industrial fields often contain sensitive information. Due to the particularity of the industrial field, there are challenges in the utilization of big data. 1. The distribution of different categories data is extremely uneven; 2. There are a large number of industry terms in the short texts that constitute the metadata, which makes semantic representation difficult. These two challenges have a large impact on the application performance of existing models. In order to resolve the problems above, this paper proposes a pre-training model based on probability distribution, which for the classification of sensitive data in the power industry. The model consists of three modules: 1. The data enhancement module adopts the technology of synonym expansion and noise introduction, so that the model can extract the classification features of sensitive data with a small proportion; 2. The pre-training module adopts the BERT model, which can obtain the semantics of industry terms in short texts; 3. The probability prediction module is used to regularize the distribution of test data to meet the training data. Compared with the traditional classification model and the classification model based on deep learning, the F1-score can be improved by 36.68% and 6.39%.