Pine nuts are highly valued for their nutritional content but are prone to mold contamination during transportation and storage, potentially leading to the production of carcinogenic aflatoxins.To address this issue, we propose a rapid, non-destructive mold detection method using hyperspectral imaging combined with a lightweight three-dimensional convolutional neural network (Light3DCNN). A novel differentiable band-selection layer—feature selection (FS)—is integrated directly into the network’s training pipeline. Leveraging the Gumbel-Softmax relaxation technique and a straight-through estimator, FS enables gradient back-propagation through discrete spectral-band selections.This end-to-end learning mechanism dynamically selects the ten most informative bands by aligning the selection strategy with the final classification objective. In parallel, a lightweight feature-extraction unit combines deep convolution with point-wise convolution to independently capture spectral signatures and fuse multi-channel information with minimal computational overhead. This research compares the performance of the feature-selection-based FS-Light3DCNN model with other models. While FS-Light3DCNN achieves 99.34% accuracy—slightly lower than the full-band Light3DCNN (99.69%)—it significantly reduces training parameters and processing time by nearly 70%. FS-Light3DCNN outperforms models using SVM with UVE and CARS feature-selection algorithms, GLCM texture features, and traditional 1DCNN, 2DCNN, HybridSN, and 3DCNN models. Additionally, compared to PCA-Light3DCNN and RF-Light3DCNN, which use ten key bands selected by PCA and RF, FS-Light3DCNN improves accuracy by 6% and 5%, respectively. Experimental results demonstrate that FS-Light3DCNN excels in both accuracy and efficiency, effectively distinguishing between healthy and varying levels of mold contamination. This model provides a fast, reliable, and non-destructive method for assessing mold contamination in pine nuts and offers potential for broader applications in food quality testing.