Multimodal fish maw type recognition based on Wasserstein generative adversarial network combined with gradient penalty and spectral fusion

There are many types of fish maw with significantly varying prices. The specific type directly affects its market value and medicinal efficacy. This paper proposes a fish maw type recognition method based on Wasserstein generative adversarial network combined with gradient penalty (WGAN-GP) and spectral fusion. By collecting Raman and near-infrared (NIR) spectral data of four types of fish maw (Beihai Male Fish Maw, Beihai Female Fish Maw, Yellow Croaker Fish Maw, and Red Mouth Croaker Fish Maw), we used WGAN-GP for data augmentation. The performance of three spectral fusion strategies (data layer, feature layer, and decision layer) was explored based on two one-dimensional convolutional neural network (1D-CNN) models. The results indicate that, after applying data augmentation and expanding the training set to 3,600 samples, the performances of the 1D-VGG (NIR), 1D-VGG (Raman), 1D-ResNet (NIR), and 1D-ResNet (Raman) models all reach optimal levels. The accuracies on the test set are improved by 15.48%, 13.10%, 1.19%, and 5.95%, respectively. Under different fusion strategies, the 1D-VGG (Raman)-1D-VGG (NIR) model at the feature layer and 1D-ResNet (Raman)(1.0)-1D-ResNet (NIR)(1.0) model at the decision layer achieved the same classification results. They exceeded other models in accuracy (98.21%), precision (98.27%), recall (98.21%), and F1-score (98.21%) on the test set. In summary, this study demonstrated the great potential of data enhancement and multimodal spectral data fusion in fish maw type identification, providing analytical tools for the development of fish maw detection equipment based on multimodal techniques.