Computer Vision and Image Understanding最新文献

In the intelligent transportation system (ITS), detecting vehicles and pedestrians in low-light conditions is challenging due to the low contrast between objects and the background. Recently, many works have enhanced low-light images using deep learning-based methods, but these methods require paired images during training, which are impractical to obtain in real-world traffic scenarios. Therefore, we propose a self-supervised network (SSN) for low-light traffic image enhancement that can be trained without paired images. To avoid amplifying noise and artifacts in the processed image during enhancement, we first proposed a denoising net to reduce the noise and artifacts in the input image. Then the processed image can be enhanced by the enhancement net. Considering the compression of the traffic image, we designed an artifacts removal net to improve the quality of the enhanced image. We proposed several effective and differential losses to make SSN trainable with low-light images only. To better integrate the extracted features from different levels in the network, we also proposed an attention module named the multi-head non-local block. In experiments, we evaluated SSN and other low-light image enhancement methods on two low-light traffic image sets: the Berkeley Deep Drive (BDD) dataset and the Hong Kong night-time multi-class vehicle (HK) dataset. The results indicated that SSN significantly improves upon other methods in visual comparison and some blind image quality metrics. We also conducted comparisons on classical ITS tasks like vehicle detection on the images enhanced by SSN and other methods, which further verified its effectiveness.

Recently, 3D point cloud classification has made significant progress with the help of many datasets. However, these datasets do not reflect the incomplete nature of real-world point clouds caused by occlusion, which limits the practical application of current methods. To bridge this gap, we propose ModelNet-O, a large-scale synthetic dataset of 123,041 samples that emulates real-world point clouds with self-occlusion caused by scanning from monocular cameras. ModelNet-O is 10 times larger than existing datasets and offers more challenging cases to evaluate the robustness of existing methods. Our observation on ModelNet-O reveals that well-designed sparse structures can preserve structural information of point clouds under occlusion, motivating us to propose a robust point cloud processing method that leverages a critical point sampling (CPS) strategy in a multi-level manner. We term our method PointMLS. Through extensive experiments, we demonstrate that our PointMLS achieves state-of-the-art results on ModelNet-O and competitive results on regular datasets such as ModelNet40 and ScanObjectNN, and we also demonstrate its robustness and effectiveness. Code available: https://github.com/fanglaosi/ModelNet-O_PointMLS.

3D human pose and shape estimation is often impacted by distribution bias in real-world scenarios due to factors such as bone length, camera parameters, background, and occlusion. To address this issue, we propose the Confidence Sharing Adaptation (CSA) algorithm, which corrects model bias using unlabeled images from the test domain before testing. However, the lack of annotation constraints in the adaptive training process poses a significant challenge, making it susceptible to model collapse. CSA utilizes a decoupled dual-branch learning framework to provide pseudo-labels and remove noise samples based on the confidence scores of the inference results. By sharing the most confident prior knowledge between the dual-branch networks, CSA effectively mitigates distribution bias. CSA is also remarkably adaptable to severely occluded scenes, thanks to two auxiliary techniques: a self-attentive parametric regressor that ensures robustness to occlusion of local body parts and a rendered surface texture loss that regulates the relationship between occlusion of human joint positions. Evaluation results show that CSA successfully adapts to scenarios beyond the training domain and achieves state-of-the-art performance on both occlusion-specific and general benchmarks. Code and pre-trained models are available for research at https://github.com/bodymapper/csa.git

Large pre-trained models based on Vision Transformers (ViTs) contain nearly billions of parameters, demanding substantial computational resources and storage space. This restricts their transferability across different tasks. Recent approaches try to use adapter fine-tuning to address this drawback. However, there is still potential to improve the number of tunable parameters and the accuracy in these methods. To address this challenge, we propose an adapter fine-tuning module called Lv-Adapter, which consists of a linear layer and vector. This module enables targeted parameter fine-tuning of pretrained models by learning both the prior knowledge of pre-trained task and the information from downstream specific task, to adapt to various downstream tasks in image and video tasks while transfer learning. Compared to full fine-tuning methods, Lv-Adapter has several appealing advantages. Firstly, by adding only about 3% extra parameters to ViT, Lv-Adapter achieves comparable accuracy to full fine-tuning methods and even significantly surpasses them on action recognition benchmarks. Secondly, Lv-Adapter is a lightweight module that can be plug-and-play in different transformer models due to its simplicity. Finally, to validate these claims, extensive experiments were conducted on five image and video datasets in this study, providing evidence for the effectiveness of Lv-Adapter. When only 3.5% of the extra parameters are updated, it respectively achieves a relative boost of about 13% and 24% compared to the fully fine-tuned model on SSv2 and HMDB51.

The multi-view 3D human pose estimation task relies on 2D human pose estimation for each view; however, severe occlusion, truncation, and human interaction lead to incorrect 2D human pose estimation for some views. The traditional “Matching-Lifting-Tracking” paradigm amplifies the incorrect 2D human pose into an incorrect 3D human pose, which significantly challenges the robustness of multi-view 3D human pose estimation. In this paper, we propose a novel method that tackles the inherent difficulties of the traditional paradigm. This method is rooted in the newly devised “Skeleton Pooling-Clustering-Tracking (SPCT)” paradigm. It initiates a 2D human pose estimation for each perspective. Then a symmetrical dilated network is created for skeleton pool estimation. Upon clustering the skeleton pool, we introduce and implement an innovative tracking method that is explicitly designed for the SPCT paradigm. The tracking method refines and filters the skeleton clusters, thereby enhancing the robustness of the multi-person 3D human pose estimation results. By coupling the skeleton pool with the tracking refinement process, our method obtains high-quality multi-person 3D human pose estimation results despite severe occlusions that produce erroneous 2D and 3D estimates. By employing the proposed SPCT paradigm and a computationally efficient network architecture, our method outperformed existing approaches regarding robustness on the Shelf, 4D Association, and CMU Panoptic datasets, and could be applied in practical scenarios such as markerless motion capture and animation production.

Sign Language Production (SLP) aims to translate spoken language into visual sign language sequences. The most challenging process in SLP is the transformation of a sequence of sign glosses into corresponding sign poses (G2P). Existing approaches on G2P mainly focus on constructing mappings of sign language glosses to frame-level sign pose representations, while neglecting gloss is just a weak annotation of the sequence of sign poses. To address this problem, this paper proposes the semantic-driven diffusion model with gloss-pose latent spaces alignment (SDD-GPLA) for G2P. G2P is divided into two phases. In the first phase, we design the gloss-pose latent spaces alignment (GPLA) to model the sign pose latent representations with glosses dependency. In the second phase, we propose semantic-driven diffusion (SDD) with supervised pose reconstruction guidance as a mapping between the gloss and sign poses latent features. In addition, we propose the sign pose decoder (${\text{Decoder}}^p$) to progressively generate high-resolution sign poses from latent sign pose features and to guide the SDD training process. We evaluated SDD-GPLA on a self-collected dataset of Daily Chinese Sign Language (DCSL) and a public dataset called RWTH-Phoenix-Weather-2014T. Compared with the state-of-the-art G2P methods, we obtain at least 22.9% and 2.3% improvement in WER scores on the above two datasets, respectively.

Semantic facial attribute editing using pre-trained Generative Adversarial Networks (GANs) has attracted a great deal of attention and effort from researchers in recent years. Due to the high quality of face images generated by StyleGANs, much work has focused on the StyleGANs’ latent space and the proposed methods for facial image editing. Although these methods have achieved satisfying results for manipulating user-intended attributes, they have not fulfilled the goal of preserving the identity, which is an important challenge. We present ID-Style, a new architecture capable of addressing the problem of identity loss during attribute manipulation. The key components of ID-Style include a Learnable Global Direction (LGD) module, which finds a shared and semi-sparse direction for each attribute, and an Instance-Aware Intensity Predictor (IAIP) network, which finetunes the global direction according to the input instance. Furthermore, we introduce two losses during training to enforce the LGD and IAIP to find semi-sparse semantic directions that preserve the identity of the input instance. Despite reducing the size of the network by roughly 95% as compared to similar state-of-the-art works, ID-Style outperforms baselines by 10% and 7% in identity preserving metric (FRS) and average accuracy of manipulation (mACC), respectively.

Monocular depth estimation (MDE) is crucial in a wide range of applications, including robotics, autonomous driving and virtual reality. Self-supervised monocular depth estimation has emerged as a promising MDE approach without requiring hard-to-obtain depth labels during training, and multi-scale photometric loss is widely used for self-supervised monocular depth estimation as the self-supervised signal. However, multi-photometric loss is a weak training signal and might disturb the good intermediate features representation. In this paper, we propose a successive depth map self-distillation(SDM-SD) loss, which combines with the single-scale photometric loss to replace the multi-scale photometric loss. Moreover, considering that multi-stage feature representations are essential for dense prediction tasks such as depth estimation, we also propose a dense skip connection, which can efficiently fuse the intermediate features of the encoder and fully utilize them in each stage of the decoder in our encoder–decoder architecture. By applying successive depth map self-distillation loss and dense skip connection, our proposed method can achieve state-of-the-art performance on the KITTI benchmark, and exhibit the best generalization ability on the challenging indoor dataset NYUv2 dataset.

Image super resolution involves enhancing the spatial resolution of low-quality images and improving their visual quality. As in many real-life situations, the image degradation process is unknown, performing the task of image super resolution in a blind manner is of paramount importance. Deep neural networks provide high performances for the task of blind image super resolution, in view of their end-to-end learning capability between the low-resolution images and their ground truth versions. Generally speaking, deep blind image super resolution networks initially estimate the parameters of the image degradation process, such as blurring kernel, and then use them for super-resolving the low-resolution images. In this paper, we develop a novel deep learning-based scheme for the task of blind image super resolution, in which the idea of leveraging the hybrid representations is utilized. Specifically, we employ the deterministic and stochastic representations of the blurring kernel parameters to train a deep blind super resolution network in an effective manner. The results of extensive experiments prove the effectiveness of various ideas used in the development of the proposed deep blind image super resolution network.

This paper tackles the challenge of point-supervised temporal action detection, wherein only a single frame is annotated for each action instance in the training set. Most of the current methods, hindered by the sparse nature of annotated points, struggle to effectively represent the continuous structure of actions or the inherent temporal and semantic dependencies within action instances. Consequently, these methods frequently learn merely the most distinctive segments of actions, leading to the creation of incomplete action proposals. This paper proposes POTLoc, a Pseudo-label Oriented Transformer for weakly-supervised Action Localization utilizing only point-level annotation. POTLoc is designed to identify and track continuous action structures via a self-training strategy. The base model begins by generating action proposals solely with point-level supervision. These proposals undergo refinement and regression to enhance the precision of the estimated action boundaries, which subsequently results in the production of ‘pseudo-labels’ to serve as supplementary supervisory signals. The architecture of the model integrates a transformer with a temporal feature pyramid to capture video snippet dependencies and model actions of varying duration. The pseudo-labels, providing information about the coarse locations and boundaries of actions, assist in guiding the transformer for enhanced learning of action dynamics. POTLoc outperforms the state-of-the-art point-supervised methods on THUMOS’14 and ActivityNet-v1.2 datasets.