Early Prediction of Human Action by Deep Reinforcement Learning

Abstract

Early action prediction in video is a challenging task where the action of a human performer is expected to be predicted using only the initial few frames. We propose a novel technique for action prediction based on Deep Reinforcement learning, employing a Deep Q-Network (DQN) and the ResNext as the basic CNN architecture. The proposed DQN can predict the actions in videos from features extracted from the first few frames of the video, and the basic CNN model is adjusted by tuning the hyperparameters of the CNN network. The ResNext model is adjusted based on the reward provided by the DQN, and the hyperparameters are updated to predict actions. The agent's stopping criteria is higher or equal to the validation accuracy value. The DQN is rewarded based on the sequential input frames and the transition of action states (i.e., prediction of action class for an incremental 10 percent of the video). The visual features extracted from the first 10 percent of the video is forwarded to the next 10 percent of the video for each action state. The proposed method is tested on the UCF101 dataset and has outperformed the state-of-the-art in action prediction.