Indonesian Street Food Calorie Estimation Using Mask R-CNN and Multiple Linear Regression

Indonesian people need to know about the calorie information of the street food. One of effective ways to get that is using image-based calorie estimation technologies. However, there are two limitations. First, the problem of occluded food and second is the low R Squared value in measurement using linear regression model with area feature. This research proposed the Mask R-CNN model for amodal instance segmentation task to get the complete object shape and multiple linear regression model with area, perimeter, length, and width to predict the food weight. This research proposed Indonesian street food dataset that has six classes. There are 1646 images of the dataset and total instance of each food are 644 bakwan, 812 bolu, 918 cireng, 679 serabi, 711 tahu, and 766 tempe. The number of data point in multiple linear regression model is 230 bakwan, 200 bolu, 250 cireng, 240 serabi, 230 tahu, and 230 tempe. The proposed multiple linear regression model has the highest R Squared score in all classes with the average R Squared 0.80425. Mask R-CNN ResNeXt-101-FPN in amodal instance segmentation task reaches the best F1 Score. In occluded scenario this model gets F1 Score 0.821 in IoU threshold 0.85. In non-occluded scenario the model gets F1 Score 0.994 in IoU threshold 0.9. Even though the F1 Score is high, there are some false detections and the bad segmentation quality. In calorie prediction, the proposed model is not reducing MAE score in some classes due to the segmentation quality and food characteristic.