What Helps to Detect What? Explainable AI and Multisensor Fusion for Semantic Segmentation of Simultaneous Crop and Land Cover Land Use Delineation

Abstract

This study introduces two novel explainable AI frameworks, Interclass-Grad-CAM and Spectral-Grad-CAM, designed to enhance the interpretability of semantic segmentation models for Crop and Land Cover Land Use (CLCLU) mapping. Interclass-Grad-CAM provides insights into interactions between land cover classes, revealing complex spatial arrangements, while Spectral-Grad-CAM quantifies the contributions of individual spectral bands to model predictions, optimizing spectral data use. These XAI methods significantly advance understanding of model behavior, particularly in heterogeneous landscapes, and ensure enhanced transparency in CLCLU mapping. To demonstrate the effectiveness of these innovations, we developed a framework that addresses data asymmetry between the United States and Mexico in the transboundary Middle Rio Grande region. Our approach integrates pixel-level multisensor fusion, combining dual-month moderate-resolution optical imagery (July and December 2023), synthetic aperture radar (SAR), and digital elevation model (DEM) data, processed using a Multi-Attention Network with a modified Mix Vision Transformer encoder to process multiple spectral inputs. Results indicate a uniform improvement in class-specific Intersection over Union by approximately 1% with multisensor integration compared to optical imagery alone. Optical bands proved most effective for crop classification, while SAR and DEM data enhanced predictions for nonagricultural types. This framework not only improves CLCLU mapping accuracy, but also offers a robust tool for broader environmental monitoring and resource management applications.