A dataset of egocentric and exocentric view hands in interactive senses

Abstract

The dataset presents raw data on the egocentric (first-person view) and exocentric (third-person view) perspectives, including 47166 frame images. Egocentric and exocentric frame images are recorded from original iPhone videos simultaneously. The egocentric view captures the details of proximity hand gestures and attentiveness of the iPhone wearer, while the exocentric view captures the hand gestures in the top-down view of all participants. The data provides frame images of two, three, and four people engaged in interactive games such as Poker, Checkers, and Dice. Furthermore, the data was collected in the real environment under natural, white, yellow, and dim light conditions. The dataset contains diverse hand gestures, including remarkable instances such as motion blur, extremely deformed, sharp shadows, and extremely dim light. Moreover, researchers working on artificial intelligence (AI) interaction games in extended reality can create sub-datasets from the metadata for one or both perspectives in the egocentric or exocentric views, facilitating the AI understanding of hand gestures in human interactive games. Furthermore, researchers can extract hand gestures considered relevant studies for hand-object interaction, such as hands deformed by holding a chess piece, blurred hand gripping containers at Dice, and hands obscured by playing cards. Researchers can annotate rectangular boxes, and hand edges for semi-supervised and supervised hand detection, hand segmentation, and hand classification to improve the ability of the AI to distinguish between each player's hand gestures. Unsupervised, self-supervised research can also be done directly using this dataset.