Comprehensive Environmental Monitoring System for Industrial and Mining Enterprises Using Multimodal Deep Learning and CLIP Model

Abstract

Addressing the challenges of limited accuracy in anomaly detection within comprehensive environmental monitoring of industrial and mining enterprises, and the constraints posed by singular data modalities, this study proposes an integration of a multimodal Long Short-Term Memory (LSTM) model with the Contrastive Language-Image Pretraining (CLIP) model. The initial phase employs ResNet within the CLIP model for extracting image features, and a Transformer for encoding text features. Subsequently, feature vectors obtained from monitoring images and text are fused using a rudimentary concatenation method to generate a joint embedding representation. Principal Component Analysis (PCA) is then applied to diminish the dimensionality of the amalgamated features derived from environmental monitoring images, descriptive texts, and sensor data collected by industrial and mining enterprises. Finally, a multimodal LSTM model is leveraged to detect anomalies in the monitoring data by capturing long-term dependencies within time series information. The model was trained and evaluated using real-time data from a coal mining enterprise’s environmental monitoring system spanning March to September 2023. Results reveal that the multimodal LSTM-CLIP model achieved an anomaly detection accuracy of 0.98 in environmental monitoring, marking a 0.10 improvement over the unimodal LSTM model, with a response time of merely 110.25 milliseconds. These findings underscore the efficacy of the multimodal LSTM-CLIP model in integrating multimodal information, thereby significantly enhancing the accuracy of anomaly detection and the speed of environmental anomaly warnings, ultimately ensuring the safety of industrial and mining enterprises.