Performance Assessment of Convolutional Neural Networks for Semantic Image Segmentation

Convolutional neural networks are applied successfully for image classification and object detection. Recently, they have been adopted to semantic segmentation tasks and several new network architectures have been proposed. With respect to automotive applications, the Cityscapes dataset is often used as a benchmark. It is one of the biggest datasets in this field and consists of a training, a validation, and a test set. While training and validation allow the optimisation of these nets, the test dataset can be used to evaluate their performance. Our investigations have shown that while these networks perform well for images of the Cityscapes dataset, their segmentation quality significantly drops when applied to new data. It seems that they have limited generalisation abilities. In order to find out whether the image content itself or other image properties cause this effect, we have carried out systematic investigations with modified Cityscapes data. We have found that camera-dependent image properties like brightness, contrast, or saturation can significantly influence the segmentation quality. This papers presents the results of these tests including eight state-of-the-art CNNs. It can be concluded that the out-of-the-boxusage of CNNs in real-world environments is not recommended.