Jungin Park, Sangryul Jeon, Seungryong Kim, Jiyoung Lee, Sunok Kim, K. Sohn
While recognizing human actions and surrounding scenes addresses different aspects of video understanding, they have strong correlations that can be used to complement the singular information of each other. In this paper, we propose an approach for joint action and scene recognition that is formulated in an end-to-end learning framework based on temporal attention techniques and the fusion of them. By applying temporal attention modules to the generic feature network, action and scene features are extracted efficiently, and then they are composed to a single feature vector through the proposed fusion module. Our experiments on the CoVieW18 dataset show that our model is able to detect temporal attention with only weak supervision, and remarkably improves multi-task action and scene classification accuracies.
{"title":"Learning to Detect, Associate, and Recognize Human Actions and Surrounding Scenes in Untrimmed Videos","authors":"Jungin Park, Sangryul Jeon, Seungryong Kim, Jiyoung Lee, Sunok Kim, K. Sohn","doi":"10.1145/3265987.3265989","DOIUrl":"https://doi.org/10.1145/3265987.3265989","url":null,"abstract":"While recognizing human actions and surrounding scenes addresses different aspects of video understanding, they have strong correlations that can be used to complement the singular information of each other. In this paper, we propose an approach for joint action and scene recognition that is formulated in an end-to-end learning framework based on temporal attention techniques and the fusion of them. By applying temporal attention modules to the generic feature network, action and scene features are extracted efficiently, and then they are composed to a single feature vector through the proposed fusion module. Our experiments on the CoVieW18 dataset show that our model is able to detect temporal attention with only weak supervision, and remarkably improves multi-task action and scene classification accuracies.","PeriodicalId":151362,"journal":{"name":"Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128567373","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Session details: Session 1: Regular Track","authors":"K. Sohn","doi":"10.1145/3286585","DOIUrl":"https://doi.org/10.1145/3286585","url":null,"abstract":"","PeriodicalId":151362,"journal":{"name":"Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130895653","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Session details: Keynote & Invited Talks","authors":"K. Sohn","doi":"10.1145/3286584","DOIUrl":"https://doi.org/10.1145/3286584","url":null,"abstract":"","PeriodicalId":151362,"journal":{"name":"Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130458222","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper, we present a new end-to-end convolutional neural network architecture for video classification, and apply the model to action and scene recognition in untrimmed videos for the Challenge on Comprehensive Video Understanding in the Wild. The proposed architecture takes densely sampled video frames as inputs, and apply a temporal pooling operator inside the network to capture temporal context of the input video. As a result, our architecture outputs distinct video-level features with a set of different temporal pooling operators. Furthermore, we design a multimodal feature fusion model by concatenating our video-level features with those given in the challenge dataset. Experimental results on the challenge dataset demonstrate that the proposed architecture and the multimodal feature fusion approach together achieve outstanding performance in action and scene recognition.
{"title":"Video Understanding via Convolutional Temporal Pooling Network and Multimodal Feature Fusion","authors":"Heeseung Kwon, Suha Kwak, Minsu Cho","doi":"10.1145/3265987.3265991","DOIUrl":"https://doi.org/10.1145/3265987.3265991","url":null,"abstract":"In this paper, we present a new end-to-end convolutional neural network architecture for video classification, and apply the model to action and scene recognition in untrimmed videos for the Challenge on Comprehensive Video Understanding in the Wild. The proposed architecture takes densely sampled video frames as inputs, and apply a temporal pooling operator inside the network to capture temporal context of the input video. As a result, our architecture outputs distinct video-level features with a set of different temporal pooling operators. Furthermore, we design a multimodal feature fusion model by concatenating our video-level features with those given in the challenge dataset. Experimental results on the challenge dataset demonstrate that the proposed architecture and the multimodal feature fusion approach together achieve outstanding performance in action and scene recognition.","PeriodicalId":151362,"journal":{"name":"Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128544383","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild","authors":"","doi":"10.1145/3265987","DOIUrl":"https://doi.org/10.1145/3265987","url":null,"abstract":"","PeriodicalId":151362,"journal":{"name":"Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116957071","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Object tracking and segmentation are important research topics in computer vision. They provide the trajectory and boundary of an object based on their appearance and shape features. Most studies on tracking and segmentation focus on encoding methods for the feature of an object. However, the tracking trajectory and segmentation mask are acquired separately, although similar visual information is required for both methods. Therefore, in this paper, we propose a CNN-based joint object tracking and segmentation framework that provides a segmentation mask while improving the performance of object tacker. In our model, the tracking model determines the trajectory of the target object as a bounding box in each frame. Given the bounding box at each frame, the segmentation model predicts a dense mask of the target object in the bounding box. Then, the segmentation mask is used to refine the bounding box for the tracking model. We evaluate the performance of our algorithm on DAVIS benchmark dataset by AUC score and mean IoU. We showed that the performance of original tracker was improved by our proposed framework.
{"title":"Joint Object Tracking and Segmentation with Independent Convolutional Neural Networks","authors":"Hakjin Lee, Jongbin Ryu, Jongwoo Lim","doi":"10.1145/3265987.3265992","DOIUrl":"https://doi.org/10.1145/3265987.3265992","url":null,"abstract":"Object tracking and segmentation are important research topics in computer vision. They provide the trajectory and boundary of an object based on their appearance and shape features. Most studies on tracking and segmentation focus on encoding methods for the feature of an object. However, the tracking trajectory and segmentation mask are acquired separately, although similar visual information is required for both methods. Therefore, in this paper, we propose a CNN-based joint object tracking and segmentation framework that provides a segmentation mask while improving the performance of object tacker. In our model, the tracking model determines the trajectory of the target object as a bounding box in each frame. Given the bounding box at each frame, the segmentation model predicts a dense mask of the target object in the bounding box. Then, the segmentation mask is used to refine the bounding box for the tracking model. We evaluate the performance of our algorithm on DAVIS benchmark dataset by AUC score and mean IoU. We showed that the performance of original tracker was improved by our proposed framework.","PeriodicalId":151362,"journal":{"name":"Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132144016","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
CoVieW 2018 is a new challenge which aims at simultaneous scene and action recognition for untrimmed video [1]. In the challenge, frame-level video features extracted by pre-trained deep convolutional neural network (CNN) are provided for video-level classification. In this paper, a new approach for the video-level classification method is proposed. The proposed method focuses on the analysis in temporal domain and the temporal attention model is developed. To compensate for the differences in the lengths of various videos, temporal padding method is also developed to unify the lengths of videos. Further, data augmentation is performed to enhance some validation accuracy. Finally, for the train/validation in CoView 2018 dataset we recorded the performance of 95.53% accuracy in the scene and 87.17% accuracy in the action using temporal attention model, nonzero padding and data augmentation. The top-1 hamming score is the standard metric in the CoVieW 2018 challenge and 91.35% is obtained by the proposed method.
{"title":"New Feature-level Video Classification via Temporal Attention Model","authors":"Hongje Seong, Junhyuk Hyun, Suhyeon Lee, Suhan Woo, Hyunbae Chang, Euntai Kim","doi":"10.1145/3265987.3265990","DOIUrl":"https://doi.org/10.1145/3265987.3265990","url":null,"abstract":"CoVieW 2018 is a new challenge which aims at simultaneous scene and action recognition for untrimmed video [1]. In the challenge, frame-level video features extracted by pre-trained deep convolutional neural network (CNN) are provided for video-level classification. In this paper, a new approach for the video-level classification method is proposed. The proposed method focuses on the analysis in temporal domain and the temporal attention model is developed. To compensate for the differences in the lengths of various videos, temporal padding method is also developed to unify the lengths of videos. Further, data augmentation is performed to enhance some validation accuracy. Finally, for the train/validation in CoView 2018 dataset we recorded the performance of 95.53% accuracy in the scene and 87.17% accuracy in the action using temporal attention model, nonzero padding and data augmentation. The top-1 hamming score is the standard metric in the CoVieW 2018 challenge and 91.35% is obtained by the proposed method.","PeriodicalId":151362,"journal":{"name":"Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131381760","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Deterioration of image due to haze is one of the factors that degrade the performance of computer vision algorithm. The haze component absorbs and reflects the reflected light from the object, distorting the original irradiance. The more the distance from the camera is, the more deteriorated it tends to be. Therefore, studies have been conducted to remove haze by estimating the distribution of haze along the distance. In this paper, we use convolution neural network to simultaneously perform depth estimation and haze removal based on stereo image, and depth information to help improve performance of haze removal. We propose a multitasking network in which the encoder learns depth information and dehazing features simultaneously by performing depth estimation and dehazing using two decoders. The learning of the network is based on a stereo image, and a large amount of left and right hazy images are required. However, existing hazy image data sets are inferior in reality because they are added to fog components in indoor images. Therefore, a data set composed of a haze component corresponding to the distance information was constructed and used in the KITTI road data set composed of a large amount of stereo outdoor driving images. Experimental results show that the proposed network has robust dehazing performance compared to existing methods for various levels of hazy images and improves the visibility by strengthening the contrast of boundaries in faint areas due to haze.
{"title":"Stereo Vision aided Image Dehazing using Deep Neural Network","authors":"Jeong-Yun Na, Kuk-jin Yoon","doi":"10.1145/3265987.3265993","DOIUrl":"https://doi.org/10.1145/3265987.3265993","url":null,"abstract":"Deterioration of image due to haze is one of the factors that degrade the performance of computer vision algorithm. The haze component absorbs and reflects the reflected light from the object, distorting the original irradiance. The more the distance from the camera is, the more deteriorated it tends to be. Therefore, studies have been conducted to remove haze by estimating the distribution of haze along the distance. In this paper, we use convolution neural network to simultaneously perform depth estimation and haze removal based on stereo image, and depth information to help improve performance of haze removal. We propose a multitasking network in which the encoder learns depth information and dehazing features simultaneously by performing depth estimation and dehazing using two decoders. The learning of the network is based on a stereo image, and a large amount of left and right hazy images are required. However, existing hazy image data sets are inferior in reality because they are added to fog components in indoor images. Therefore, a data set composed of a haze component corresponding to the distance information was constructed and used in the KITTI road data set composed of a large amount of stereo outdoor driving images. Experimental results show that the proposed network has robust dehazing performance compared to existing methods for various levels of hazy images and improves the visibility by strengthening the contrast of boundaries in faint areas due to haze.","PeriodicalId":151362,"journal":{"name":"Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129080194","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Session details: Session 2: Challenge Track","authors":"K. Sohn","doi":"10.1145/3286586","DOIUrl":"https://doi.org/10.1145/3286586","url":null,"abstract":"","PeriodicalId":151362,"journal":{"name":"Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114141736","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Analyzing videos is one of the fundamental problems of computer vision and multimedia analysis for decades. The task is very challenging as video is an information-intensive media with large variations and complexities. Thanks to the recent development of deep learning techniques, researchers in both computer vision and multimedia communities are now able to boost the performance of video analysis significantly and initiate new research directions to analyze video content. This talk will cover recent advances under the umbrella of video understanding, which start from basic networks that are widely adopted in state-of-the-art deep learning pipelines, to fundamental challenges of video representation learning and video classification/recognition, finally to an emerging area of video and language.
{"title":"Deep Video Understanding: Representation Learning, Action Recognition, and Language Generation","authors":"Tao Mei","doi":"10.1145/3265987.3265994","DOIUrl":"https://doi.org/10.1145/3265987.3265994","url":null,"abstract":"Analyzing videos is one of the fundamental problems of computer vision and multimedia analysis for decades. The task is very challenging as video is an information-intensive media with large variations and complexities. Thanks to the recent development of deep learning techniques, researchers in both computer vision and multimedia communities are now able to boost the performance of video analysis significantly and initiate new research directions to analyze video content. This talk will cover recent advances under the umbrella of video understanding, which start from basic networks that are widely adopted in state-of-the-art deep learning pipelines, to fundamental challenges of video representation learning and video classification/recognition, finally to an emerging area of video and language.","PeriodicalId":151362,"journal":{"name":"Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127605191","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：