Domain adaptive semantic segmentation by optimal transport

Abstract

Scene segmentation is widely used in autonomous driving for environmental perception. Semantic scene segmentation has gained considerable attention owing to its rich semantic information. It assigns labels to the pixels in an image, thereby enabling automatic image labeling. Current approaches are based mainly on convolutional neural networks (CNN), however, they rely on numerous labels. Therefore, the use of a small amount of labeled data to achieve semantic segmentation has become increasingly important. In this study, we developed a domain adaptation framework based on optimal transport (OT) and an attention mechanism to address this issue. Specifically, we first generated the output space via a CNN owing to its superior of feature representation. Second, we utilized OT to achieve a more robust alignment of the source and target domains in the output space, where the OT plan defined a well attention mechanism to improve the adaptation of the model. In particular, the OT reduced the number of network parameters and made the network more interpretable. Third, to better describe the multiscale properties of the features, we constructed a multiscale segmentation network to perform domain adaptation. Finally, to verify the performance of the proposed method, we conducted an experiment to compare the proposed method with three benchmark and four SOTA methods using three scene datasets. The mean intersection-over-union (mIOU) was significantly improved, and visualization results under multiple domain adaptation scenarios also show that the proposed method performed better than semantic segmentation methods.