Computer Vision and Image Understanding最新文献

Brain imaging plays a crucial role in the diagnosis and treatment of various neurological disorders, providing valuable insights into the structure and function of the brain. Techniques such as magnetic resonance imaging (MRI) and computed tomography (CT) enable non-invasive visualization of the brain, aiding in the understanding of brain anatomy, abnormalities, and functional connectivity. However, cost and radiation dose may limit the acquisition of specific image modalities, so medical image synthesis can be used to generate required medical images without actual addition. CycleGAN and other GANs are valuable tools for generating synthetic images across various fields. In the medical domain, where obtaining labeled medical images is labor-intensive and expensive, addressing data scarcity is a major challenge. Recent studies propose using transfer learning to overcome this issue. This involves adapting pre-trained CycleGAN models, initially trained on non-medical data, to generate realistic medical images. In this work, transfer learning was applied to the task of MR-CT image translation and vice versa using 18 pre-trained non-medical models, and the models were fine-tuned to have the best result. The models’ performance was evaluated using four widely used image quality metrics: Peak-signal-to-noise-ratio, Structural Similarity Index, Universal Quality Index, and Visual Information Fidelity. Quantitative evaluation and qualitative perceptual analysis by radiologists demonstrate the potential of transfer learning in medical imaging and the effectiveness of the generic pre-trained model. The results provide compelling evidence of the model’s exceptional performance, which can be attributed to the high quality and similarity of the training images to actual human brain images. These results underscore the significance of carefully selecting appropriate and representative training images to optimize performance in brain image analysis tasks.

This survey provides a comprehensive evaluation of various deep learning-based segmentation architectures. It covers a wide range of models, from traditional ones like FCN and PSPNet to more modern approaches like SegFormer and FAN. In addition to assessing the methods in terms of segmentation accuracy, we propose to also evaluate the methods in terms of temporal consistency and corruption vulnerability. Most of the existing surveys on semantic segmentation focus on outdoor datasets. In contrast, this survey focuses on indoor scenarios to enhance the applicability of segmentation methods in this specific domain. Furthermore, our evaluation consists of a performance analysis of the methods in prevalent real-world segmentation scenarios that pose particular challenges. These complex situations involve scenes impacted by diverse forms of noise, blur corruptions, camera movements, optical aberrations, among other factors. By jointly exploring the segmentation accuracy, temporal consistency, and corruption vulnerability in challenging real-world situations, our survey offers insights that go beyond existing surveys, facilitating the understanding and development of better image segmentation methods for indoor scenes.

Images and videos often suffer from issues such as motion blur, video discontinuity, or rolling shutter artifacts. Prior studies typically focus on designing specific algorithms to address individual issues. In this paper, we highlight that these issues, albeit differently manifested, fundamentally stem from sub-optimal exposure processes. With this insight, we propose a paradigm termed re-exposure, which resolves the aforementioned issues by performing exposure simulation. Following this paradigm, we design a new architecture, which constructs visual content representation from images and event camera data, and performs exposure simulation in a controllable manner. Experiments demonstrate that, using only a single model, the proposed architecture can effectively address multiple visual issues, including motion blur, video discontinuity, and rolling shutter artifacts, even when these issues co-occur.

Photometric Stereo (PS) addresses the challenge of reconstructing a three-dimensional (3D) representation of an object by estimating the 3D normals at all points on the object’s surface. This is achieved through the analysis of at least three photographs, all taken from the same viewpoint but with distinct lighting conditions. This paper introduces a novel approach for Universal PS, i.e., when both the active lighting conditions and the ambient illumination are unknown. Our method employs a multi-scale encoder–decoder architecture based on Transformers that allows to accommodates images of any resolutions as well as varying number of input images. We are able to scale up to very high resolution images like 6000 pixels by 8000 pixels without losing performance and maintaining a decent memory footprint. Moreover, experiments on publicly available datasets establish that our proposed architecture improves the accuracy of the estimated normal field by a significant factor compared to state-of-the-art methods. Code and dataset available at: https://clement-hardy.github.io/Uni-MS-PS/index.html.

Most of the existing face photo de-meshing methods have accomplished promising results; there are certain quality problems with these methods like the inpainted regions would appear blurry and unpleasant boundaries becoming visible. Such artifacts cause generated face photos unreal. Therefore, we propose an effective image-to-image translation framework called Face De-meshing Using Generative Adversarial Networks (De-mesh GANs). The De-mesh GANs is a two-stage model: (i) binary mask generating module, is a three convolution layers-based encoder–decoder network architecture that automatically generates a binary mask for the meshed region, and (ii) face photo de-meshing module, is a GANs-based network that eliminates the mesh mask and synthesizes the meshed area. An arrangement of careful losses (reconstruction loss, adversarial loss, and perceptual loss) is used to reassure the better quality of the de-mesh face photos. To facilitate the training of the proposed model, we have designed a dataset of clean/corrupted photo pairs using the CelebA dataset. Qualitative and quantitative evaluations of the De-mesh GANs on real-world corrupted face photo images show better performance than the previously proposed face photo de-meshing models. Furthermore, we also offer the ablation study for performance assessment of the additional network i.e., perceptual network.

Hand gesture is one of the most efficient and natural interfaces in current human–computer interaction (HCI) systems. Despite the great progress achieved in hand gesture-based HCI, perceiving or tracking the hand pose from images remains challenging. In the past decade, several challenges have been indicated and explored, such as incomplete data issue, the requirement of large-scale annotated dataset, and 3D hand pose estimation from monocular RGB image; however, there is a lack of surveys to provide comprehensive collection and analysis for these challenges and corresponding solutions. To this end, this paper devotes effort to the general challenges of hand gesture interpretation techniques in HCI systems based on visual sensors and elaborates on the corresponding solutions in current state-of-the-arts, which can provide a systematic reminder for practical problems of hand gesture interpretation. Moreover, this paper provides informative cues for recent datasets to further point out the inherent differences and connections among them, such as the annotation of objects and the number of hands, which is important for conducting research yet ignored by previous reviews. In retrospect of recent developments, this paper also conjectures what the future work will concentrate on, from the perspectives of both hand gesture interpretation and dataset construction.

The similar distractor problem is one of the most difficult challenges for Siamese-based trackers. Since they formulate the visual tracking task as a similar matching problem, these trackers involve an essential problem that they are sensitive to the intra-class and inter-class instances with similar appearance confusion. To solve the problem, we propose an object discriminability re-extraction network (ODR-Net) for distractor-aware visual object tracking. The network first mines similar distractors from existing tracking information with a distractor capture module, and then re-extracts discriminative features to redetect the target from distractors with a discriminative feature re-extraction module. It solves the distractor problem in the decoding phase of a tracker and can be considered as a general block that applied to existing Siamese trackers to tackle the similar distractor problem. To demonstrate the effectiveness of the proposed method, extensive experiments and comparisons with state-of-the-art trackers are conducted on a variety of large-scale benchmark datasets, including GOT-10k, LaSOT, OTB-2015, TrackingNet, VOT2020, VOT2021, and VOT2022. Without bells and whistles, our ODR-Net achieves leading performance with a real-time speed.

Previous work has shown that CNN-based local distribution learning can efficiently reconstruct high-resolution images, but with limited performance improvement against complex degraded images. In this paper, we propose an enhanced local distribution learning framework, called ELDRN, which successfully generalizes local distribution learning to realistic images whose degradation process is complex and unknowable. The cores of our ELDRN are the parallel attention block and dilated neighborhood sampling. The former mines discriminative features at both spatial and channel levels, that is, parameters for constructing local distributions, thus improving the robustness of distributions to real degradation patterns. To deal with the fact that the reference range of the target sub-pixel is not exactly equal to its neighborhood, we explicitly increase the sampling density, i.e., fusing more sampled pixels to produce the target sub-pixel. Experiments conducted on RealSR dataset illustrate that our ELDRN outperforms recent learning-based SISR methods and reconstructs visually-pleasant high-quality images.

Image dehazing under deficient data is an ill-posed and challenging problem. Most existing methods tackle this task by developing either CycleGAN-based hazy-to-clean translation or physical-based haze decomposition. However, geometric structure is often not effectively incorporated in their straightforward hazy-clean projection framework, which might incur inaccurate estimation in distant areas. In this paper, we rethink the image dehazing task and propose a depth-aware perception framework, DehazeDP, for robust haze decomposition on deficient data. Our DehazeDP is insthe pired by Diffusion Probabilistic Model to form an end-to-end training pipeline that seamlessly ines the hazy image generation with haze disentanglement. Specifically, in the forward phase, the haze is added to a clean image step-by-step according to the depth distribution. Then, in the reverse phase, a unified U-Net is used to predict the haze and recover the clean image progressively. Extensive experiments on public datasets demonstrate that the proposed DehazeDP performs favorably against state-of-the-art approaches. We release the code and models at https://github.com/stallak/DehazeDP.

This paper focuses on Human–Object Interaction (HOI) detection, addressing the challenge of identifying and understanding the interactions between humans and objects within a given image or video frame. Spearheaded by Detection Transformer (DETR), recent developments lead to significant improvements by replacing traditional region proposals by a set of learnable queries. However, despite the powerful representation capabilities provided by Transformers, existing Human–Object Interaction (HOI) detection methods still yield low confidence levels when dealing with complex interactions and are prone to overlooking interactive actions. To address these issues, we propose a novel approach UAHOI, Uncertainty-aware Robust Human–Object Interaction Learning that explicitly estimates prediction uncertainty during the training process to refine both detection and interaction predictions. Our model not only predicts the HOI triplets but also quantifies the uncertainty of these predictions. Specifically, we model this uncertainty through the variance of predictions and incorporate it into the optimization objective, allowing the model to adaptively adjust its confidence threshold based on prediction variance. This integration helps in mitigating the adverse effects of incorrect or ambiguous predictions that are common in traditional methods without any hand-designed components, serving as an automatic confidence threshold. Our method is flexible to existing HOI detection methods and demonstrates improved accuracy. We evaluate UAHOI on two standard benchmarks in the field: V-COCO and HICO-DET, which represent challenging scenarios for HOI detection. Through extensive experiments, we demonstrate that UAHOI achieves significant improvements over existing state-of-the-art methods, enhancing both the accuracy and robustness of HOI detection.