Developing tongue coating status assessment using image recognition with deep learning.

Purpose: To build an image recognition network to evaluate tongue coating status.

Methods: Two image recognition networks were built: one for tongue detection and another for tongue coating classification. Digital tongue photographs were used to develop both networks; images from 251 (178 women, 74.7±6.6 years) and 144 older adults (83 women, 73.8±7.3 years) who volunteered to participate were used for the tongue detection network and coating classification network, respectively. The learning objective of the tongue detection network is to extract a rectangular region that includes the tongue. You-Only-Look-Once (YOLO) v2 was used as the detection network, and transfer learning was performed using ResNet-50. The accuracy was evaluated by calculating the intersection over the union. For tongue coating classification, the rectangular area including the tongue was divided into a grid of 7×7. Five experienced panelists scored the tongue coating in each area using one of five grades, and the tongue coating index (TCI) was calculated. Transfer learning for tongue coating grades was performed using ResNet-18, and the TCI was calculated. Agreement between the panelists and network for the tongue coating grades in each area and TCI was evaluated using the kappa coefficient and intraclass correlation, respectively.

Results: The tongue detection network recognized the tongue with a high intersection over union (0.885±0.081). The tongue coating classification network showed high agreement with tongue coating grades and TCI, with a kappa coefficient of 0.826 and an intraclass correlation coefficient of 0.807, respectively.

Conclusions: Image recognition enables simple and detailed assessment of tongue coating status.