Hierarchical Self-Learning Knowledge Inference Based on Markov Random Field for Semantic Segmentation of Remote Sensing Images

Abstract

Semantic segmentation is one of the most important tasks in the field of remote sensing. As the spatial resolution increases, the remote sensing images can capture more detailed information and make hierarchical semantic interpretation possible. However, hierarchical semantic segmentation encounters high heterogeneity not only within the intra-layer classes but also among inter-layer classes. It brings challenges to semantic segmentation methods such as the convolutional neural network (CNN). In this article, a hierarchical self-learning knowledge inference model (HSKIM) based on the Markov random field (MRF) model is proposed for hierarchical semantic segmentation of remote sensing images. The HSKIM model introduces a new framework that integrates the advantages of CNN-based data feature learning and MRF-based hierarchical semantic inference. It contains three modules: data learning module ( $\boldsymbol {D}$ ), inference units generation module ( $\boldsymbol {I}$ ), and self-learning knowledge inference module ( $\boldsymbol {S}$ ). The module $\boldsymbol {D}$ uses CNN to learn specific data features layer by layer and extract preliminary geographical objects as the initial results. The module I refines the geographical objects using a novel boundary-preservation trick to generate more accurate inference units with clear geographical meaning. The module S introduces a hierarchical object-based MRF model to implement semantic inference among intra-layer and inter-layer inference units, guided by the spatial interactions and geographical criteria. This module can self-learn and update the relationship between classes iteratively and provide the final result. Experiments on the GID dataset with hierarchical classes, alongside 12 state-of-the-art CNN-based methods, validate the effectiveness and robustness of the proposed HSKIM model. The code of this article is available at https://github.com/iichengzi/HSKIM .