Cost-efficient training of hyperspectral deep learning models for the detection of contaminating grains in bulk oats by fluorescent tagging

Abstract

Computer vision based on instance segmentation deep learning models offers great potential for automating many visual inspection tasks, such as the detection of contaminating grains in bulk oats, a nutrient rich grain which is well-tolerated by people suffering from gluten intolerance. Whereas distinguishing foreign objects is often relatively easy with the naked eye, it is much more difficult to distinguish highly similar products, e.g. different grain species or varieties. The subtle differences between such products may be captured by deep learning models combining the spectral and spatial features that are acquired with spectral cameras, measuring a spectral fingerprint for each pixel in an image. However, the training of supervised hyperspectral deep learning models requires large amounts of labelled data. As manual labelling is a tedious job and may induce labelling errors, we propose an alternative approach involving ‘tagging’ of the targets with fluorescent labels that make the targets ‘light up’ under UV illumination to efficiently generate ground truth segmentation masks. As these fluorescent labels are only visible in the UV range of the spectrum, the spectra in the SWIR range can still be used to discriminate grains from each other, making it a cost-efficient labeling technique for hyperspectral data, where labeled datasets are scarce. The primary objective of this study was to determine whether a hyperspectral deep learning segmentation model to detect uncoated spelt kernels in a bulk of oats could be trained more efficiently by coating the spelt kernels in the training images with a fluorescent paint. To this end, both a classical pixel classifier, as a benchmark model, and a deep learning segmentation model were trained on a bulk mixture of oats contaminated with coated spelt kernels and evaluated on bulk mixtures of oats and non-coated spelt kernels to assess their ability to generalize to uncoated samples. The deep learning model (RMSE = 1.34 %) outperformed the pixel classifier (RMSE = 1.91 %) in predicting the mass percentage of spelt without coating in a bulk mixture of oats, because it was more successful in segmenting the kernel edges. This indicates that the traditional pixel classification analysis could be bypassed in future research by efficiently generating the ground truth labels required for training hyperspectral deep learning models through the use of a fluorescent coating.