Multimodal Zero-Shot Shelf Deformation Detection Based on MEMS Sensors and Images

Abstract

As the variety and quantity of goods in modern warehouse management continue to increase, optimizing space utilization and ensuring the safe and orderly storage of goods have become critical challenges. High-rise shelving systems are increasingly favored by enterprises, but long-term use, collisions with stacker cranes, and overloading can lead to structural deformation of the shelves. If these deformations are not detected and addressed in a timely manner, they may result in serious safety incidents and significant property damage. To address this issue, this study proposes a zero-shot shelf deformation detection method based on multimodal data fusion. The proposed approach integrates Micro-Electro-Mechanical Systems (MEMS) sensors and image data to establish a real-time monitoring and alert mechanism. Specifically, MEMS sensors are employed for real-time acquisition of shelf status, with threshold values set to trigger an initial alert mechanism. Simultaneously, cameras capture shelf images, and multiple You Only Look Once (YOLO) models are used to detect and classify critical components of the shelf, such as beams and columns. YOLOv11n is ultimately selected as the optimal model for detecting these structural elements. Based on the detected beams and columns, further feature extraction is performed, and the sensor data is fused with these features. A K-Means clustering algorithm is then applied to conduct the clustering analysis. To address the issue of a lack of negative samples in the dataset, the study employs oversampling techniques, including SMOTE, ADASYN, and Borderline-SMOTE, combined with machine learning models such as Random Forest and Gradient Boosting Decision Trees (GBDT). The experimental results demonstrate that both Random Forest and GBDT achieved precision, recall, and F1 scores exceeding 95%, confirming the effectiveness and accuracy of the proposed method in shelf deformation detection. The multimodal detection method proposed in this study not only improves the accuracy and real-time performance of shelf deformation detection but also provides strong technical support for the safety management of warehouse operations.