Multi-sensor fusion for structural displacement estimation: Integrating vision and acceleration from mobile devices

Abstract

Structural displacement is a significant indicator for assessing structural safety conditions. Vision-based measurement methods offer a promising approach for non-contact structural displacement monitoring with substantial potential for practical applications. However, their accuracy is sensitive to lighting conditions and camera motion, which poses challenges for application in long-term structural health monitoring. In this study, a novel data fusion-based approach utilizing mobile phones for structural displacement measurement was proposed. In this method, the built-in sensors of mobile phones were employed to simultaneously capture structural vibration videos and corresponding acceleration data. A deep learning-based dense optical flow model was employed to estimate the full-field optical flow from the video, from which the structural motion is extracted. The rotation and translation of camera were inferred from the dense background motion using a least squares approach. Automatic unit conversion for visual measurements was then accomplished through variance-based fitting, and the measurement results were further refined by fusing the visual data with acceleration measurements via a Kalman filter, leading to accurate displacement estimation. The performance of this method was validated through inter-story displacement measurement within the structure and displacement measurement from outside the structure conducted in varying lighting conditions. The results indicate that the proposed method can achieve accurate displacement estimation across different lighting environments using mobile devices, with root mean square errors of fewer than 0.5 pixels. The proposed approach offers a low-cost and easily accessible solution for structural displacement measurement.