Multi-Task Learning Frameworks to Classify Food and Estimate Weight From a Single Image

Usually, elderly patients in hospitals suffer from malnutrition because they are unable to consume food as prescribed by doctors or nutritionists. Analyzing food intake is labor-intensive and time-consuming. Therefore, machine learning is used to analyze the food intake. Major food analysis tasks include food classification and food weight estimation. The basic machine learning approach to this problem is to combine a food classification model with a food weight estimate model sequentially. When we deployed it, we found that a large amount of memory and models were required. One solution is to use multi-task learning. In this study, we proposed multi-task learning frameworks that could recognize food and predict weight based on a single image. The performance of our frameworks was compared to the baseline models, which only utilized either regression or classification. Although baseline accuracy is higher, our framework has MAPE values that are lower than the baseline. To improve the performance, we explored different approaches for weighting loss, including manual weighting and auto weighting, using uncertainty and auxiliary tasks. From the experiment, our results showed that our multi-task learning framework that adjusted the weight of loss using auxiliary tasks outperformed the baseline models in terms of MAPE and Accuracy. Moreover, we demonstrate our framework when scaling up the backbone from ResNet50 to ResNet101 and ResNet152.