In high-speed milling, chatter detection plays an important role in ensuring surface quality and safe machining. Traditionally, chatter detection is performed by manually setting the feature threshold, which is unreliable. In this paper, an intelligent chatter detection method is proposed based on deep learning. The proposed method is featured by automatic chatter detection based on multi-channel features, and it is applicable in different milling conditions. To adaptively obtain the chatter signal and avoid the problem of modal mixing, the successive variational mode decomposition method is first used to extract the chatter frequency components without selecting parameters. Then, multi-channel features are extracted from the reconstructed chatter signal, and sensitive features strongly related to the milling chatter are selected based on mutual information metric. Next, a novel multi-channel feature fusion network, composed of the gated attention mechanism, ResNet module, CapsNet module, and classification module, is constructed to mine feature information and implement chatter detection. Finally, the signal data are acquired through a series of milling experiments. The identification performance of the model is evaluated in three scenarios, and an average accuracy of 0.9887 is achieved. In addition, ablation experiments and comparative studies with other detection methods are performed. The results show that the proposed method can improve the accuracy and generalization of chatter detection.