Unseen Food Segmentation

Food image segmentation is important for detailed analysis on food images, especially for classification of multiple food items and calorie amount estimation. However, there is a costly problem in training a semantic segmentation model because it requires a large number of images with pixel-level annotations. In addition, the existence of a myriad of food categories causes the problem of insufficient data in each category. Although several food segmentation datasets such as the UEC-FoodPix Complete has been released so far, the number of food categories is still limited to a small number. In this study, we propose an unseen class segmentation method with high accuracy by using both zero-shot and few-shot segmentation methods for any unseen classes. we make the following contributions: (1) we propose a UnSeen Food Segmentation method (USFoodSeg) that uses the zero-shot model to infer the segmentation mask from the class label words of unseen classes and those images, and uses the few-shot model to refine the segmentation masks. (2) We generate segmentation masks for 156 categories of the unseen class UEC-Food256, totaling 17,000 images, and 85 categories in the Food-101 dataset, totaling 85,000 images, with an accuracy of over 90%. Our proposed method is able to solve the problem of insufficient food segmentation data.