MVSRF: Point cloud semantic segmentation and optimization method for granular construction objects

Abstract

Identifying shapeless granular materials in complex construction scenarios is critical for achieving automation in engineering equipment such as wheel loaders. The challenges of segmenting point clouds for granular materials involve dealing with sparsity, real-time processing requirements, the lack of distinct shape representation, and the issue of different materials sharing similar shapes. This paper proposes MVSRF, a real-time multi-view based point cloud semantic segmentation method incorporating a single-frame re-segmentation component and a multi-frame semantic filter to enhance accuracy and robustness. First, the segmentation system generates a sparse pixel-depth grid map via semantic projection to encapsulate global points and their behaviors, while employing an edge detector to label boundary points around objects. Second, a zero-shot re-segmentation algorithm involving seed extension, novel one-dimensional DBSCAN, Delaunay triangulation, and semantic reassignment corrects mis-segmented points caused by mapping bias. Finally, a lightweight semantic filter is designed to suppress semantic noise during multiple observations. We have built a multi-sensor platform on a wheel loader and collected experimental data to verify the effectiveness of our method. Two optimization components illustrated exceptional performance on the annotated dataset. The MVSRF method possesses strong robustness against external calibration errors, camera pose estimation errors, and inaccurate image segmentation, providing a practical solution for real-time perception of granular materials.