Fusion of monocular camera and 3D LiDAR data for navigation line extraction under corn canopy

The navigation line serves as a datum for autonomous agricultural robots engaged in tasks such as monitoring, spraying, and fertilization, enabling them to traverse along crop rows. Although the above-canopy navigation line extraction methods for early-growth cornfields are advanced, they are difficult to apply to cornfields with tall plants and wide canopies in mid-to-late growth. Extracting navigation lines under the canopy using environment-aware sensors mounted at a lower position on robots is a viable option, but under-canopy complexities such as crossing leaves, varying exit crops, and light conditions pose a serious challenge to methods based on a single sensor. Therefore, this study proposes a feature-level fusion method by a monocular camera and 3D Light Detection and Ranging (LiDAR) using corn stems as references. This method includes three steps: (i) Semantic segmentation of the ground and corn stems in the image by the constructed StemFormer, which is a Transformer-based dual-branch network. (ii) After segmenting the ground and stem LiDAR point clouds based on the image semantic mask, the proposed adaptive radius filter is applied to filter the stem point cloud after dimensionality reduction based on the ground plane. (iii) The extraction of the navigation line under the corn canopy is achieved by clustering the stem point cloud using the Density-based Spatial Clustering of Applications with Noise (DBSCAN) algorithm and fitting the clustering centers by the Least Squares Method (LSM). The experimental results validate the accuracy and real-time performance of the fusion method, achieving a mean correct rate of navigation line extraction at 93.67 %, a mean absolute error of heading angle at 1.53° with a standard deviation of 1.46°, and an overall maximum running time of 80.58 ms. The navigation line extraction method offers a novel strategy for the automated navigation of agricultural robots in fields with tall crops such as corn.