KUSK dataset: toward a direct understanding of recipe text and human cooking activity

In this paper, we provide a multimodal dataset for understanding cooking activities. To build the dataset, we instructed the subjects to perform cooking according to instructional texts shown on a display one by one. The instructional texts were generated from flow graphs, which were automatically extracted from recipes sampled from a Web site. The main identity of this dataset is the correspondence between the steps automatically extracted from recipes, and real human activities. Typical uses of our dataset are to construct classifiers for understanding human activities in the kitchen, text generation through observing the activities, and so on.