On the relevance of edge-conditioned convolution for GNN-based semantic image segmentation using spatial relationships

This paper addresses the fundamental task of semantic image segmentation by exploiting structural information (spatial relationships between image regions). To perform such task, we propose to combine a deep neural network (CNN) with inexact “many-to-one-or-none” graph matching where graphs encode efficiently class probabilities a nd structural information related to regions segmented by the CNN. In order to achieve node classification, a basic 2 -layer graph neural network (GNN) based on the edge-conditioned convolution operator (ECConv), managing both node and edge attributes, is considered. Prelim-inary experiments are performed on both a synthetic dataset and a public dataset of face images (FASSEG). Our approach is shown to be resilient to small training datasets that often limit the performance of deep learning thanks to a preprocessing task of graph coarsening. Results show that the proposal reaches a perfect accuracy on synthetic dataset and improves performance of the CNN by 6% (bounding box dice index) on FASSEG. Moreover, it enhances by 27% the initial Hausdorff distance (i.e. with CNN only) using the entire training dataset and by 41% with only 75% of training samples.