Convolutional Recurrent Neural Network with Multi-Scale Kernels on Dynamic Connectivity Network for AD Classification

Deep learning methods, including convolutional neural networks (CNNs) and recurrent neural network (RNN), have been used for analysis of brain network, e.g., dynamic functional connectivity (dFC) network. However, CNN usually extract local features of brain network, ignoring the temporal information of dFC network. In addition, diversity feature representations of brain network can be obtained using convolutional kernels with different scales, these representations may contain complementary information that could be used for further improving the diagnosis performance of brain disease (e.g., Alzheimer’s Disease, AD). To address this problem, in this paper, we propose a convolutional recurrent neural network with multi-scale kernels (MSK-CRNN) learning framework for brain disease classification with fMRI data. Specifically, we build a convolutional layer with multi-scale kernels to extract different-yet-complementary features from constructed dFC networks, and use a long short-term memory (LSTM) layer to further extract temporal information of dFC networks. The experimental results on 174 subjects with 563 scans from Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset demonstrate that, compared with the existing methods, the proposed MSK-CRNN method can further improve the performance of AD classification.