A New Generative Replay Approach for Incremental Class Learning of Medical Image for Semantic Segmentation

Abstract

Deep neural networks suffer from the notorious problem of catastrophic forgetting when the tasks keep increasing. The inaccessibility of previous data due to privacy limitations and other issues directly leads to a significant drop in performance in prior tasks. Existing incremental class learning (ICL) methods in semantic segmentation are mostly regularization-based. While in this work, we incorporate a generative replay-based approach to alleviating catastrophic forgetting for the first time. We introduce SegGAN to generate both previous images and the corresponding pixel-level labels to circumvent privacy limitations and replay them to retain learned knowledge in the subsequent learning steps. Furthermore, we propose a novel filtering mechanism to select high-quality generated data by the consistency constraint of the Pseudo-Labeling and generative replay method. Specifically, we use Pseudo-Labeling to obtain the pseudo-labels of the generated images and select reliable data with high confidence by comparing generated labels with pseudo-labels.