Computer Vision and Image Understanding最新文献

Synthesizing complex images from text presents challenging. Compared to autoregressive and diffusion model-based methods, Generative Adversarial Network-based methods have significant advantages in terms of computational cost and generation efficiency yet remain two limitations: first, these methods often refine all features output from the previous stage indiscriminately, without considering these features are initialized gradually during the generation process; second, the sparse semantic constraints provided by the text description are typically ineffective for refining fine-grained features. These issues complicate the balance between generation quality, computational cost and inference speed. To address these issues, we propose a Multi-granularity Feature Aware Enhancement GAN (MFAE-GAN), which allows the refinement process to match the order of different granularity features being initialized. Specifically, MFAE-GAN (1) samples category-related coarse-grained features and instance-level detail-related fine-grained features at different generation stages based on different attention mechanisms in Coarse-grained Feature Enhancement (CFE) and Fine-grained Feature Enhancement (FFE) to guide the generation process spatially, (2) provides denser semantic constraints than textual semantic information through Multi-granularity Features Adaptive Batch Normalization (MFA-BN) in the process of refining fine-grained features, and (3) adopts a Global Semantics Preservation (GSP) to avoid the loss of global semantics when sampling features continuously. Extensive experimental results demonstrate that our MFAE-GAN is competitive in terms of both image generation quality and efficiency.

Self-supervised learning methods are increasingly important for monocular depth estimation since they do not require ground-truth data during training. Although existing methods have achieved great success for better monocular depth estimation based on Convolutional Neural Networks (CNNs), the limited receptive field of CNNs usually is insufficient to effectively model the global information, e.g., relationship between foreground and background or relationship among objects, which are crucial for accurately capturing scene structure. Recently, some studies based on Transformers have attracted significant interest in computer vision. However, duo to the lack of spatial locality bias, they may fail to model the local information, e.g., fine-grained details with an image. To tackle these issues, we propose a novel self-supervised learning framework by incorporating the advantages of both the CNNs and Transformers so as to model the complete contextual information for high-quality monocular depth estimation. Specifically, the proposed method mainly includes two branches, where the Transformer branch is considered to capture the global information while the Convolution branch is exploited to preserve the local information. We also design a rectangle convolution module with pyramid structure to perceive the semi-global information, e.g. thin objects, along the horizontal and vertical directions within an image. Moreover, we propose a shape refinement module by learning the affinity matrix between pixel and its neighborhood to obtain accurate geometrical structure of scenes. Extensive experiments evaluated on KITTI, Cityscapes and Make3D dataset demonstrate that the proposed method achieves the competitive result compared with the state-of-the-art self-supervised monocular depth estimation methods and shows good cross-dataset generalization ability.

The images captured under adverse weather conditions suffer from poor visibility and contrast problems. Such images are not suitable for computer vision analysis and similar applications. Therefore, image defogging/dehazing is one of the most intriguing topics. In this paper, a new, fast, and robust defogging/de-hazing algorithm is proposed by combining the Retinex theory with independent component analysis, which performs better than existing algorithms. Initially, the foggy image is decomposed into two components: reflectance and luminance. The former is computed using the Retinex theory, while the latter is obtained by decomposing the foggy image into parallel and perpendicular components of air-light. Finally, the defogged image is obtained by applying Koschmieder’s law. Simulation results demonstrate the absence of halo effects and the presence of high-resolution images. The simulation results also confirm the effectiveness of the proposed method when compared to other conventional techniques in terms of NIQE, FADE, SSIM, PSNR, AG, CIEDE2000, $\bar{r}$, and implementation time. All foggy and defogged results are available in high quality at the following link: https://drive.google.com/file/d/1OStXrfzdnF43gr6PAnBd8BHeThOfj33z/view?usp=drive_link.

Recovering clear structures from severely blurry inputs is a huge challenge due to the detail loss and ambiguous semantics. Although segmentation maps can help deblur facial images, their effectiveness is limited in complex natural scenes because they ignore the detailed structures necessary for deblurring. Furthermore, direct segmentation of blurry images may introduce error propagation. To alleviate the semantic confusion and avoid error propagation, we propose utilizing high-level vision tasks, such as classification, to learn a comprehensive prior for severe blur removal. We propose a feature learning strategy based on knowledge distillation, which aims to learn the priors with global contexts and sharp local structures. To integrate the priors effectively, we propose a semantic prior embedding layer with multi-level aggregation and semantic attention. We validate our method on natural image deblurring benchmarks by introducing the priors to various models, including UNet and mainstream deblurring baselines, to demonstrate its effectiveness and generalization ability. The results show that our approach outperforms existing methods on severe blur removal with our plug-and-play semantic priors.

Object Re-Identification (Re-ID) aims to identify and retrieve specific objects from images captured at different places and times. Recently, object Re-ID has achieved great success with the advances of Vision Transformers (ViT). However, the effects of the global–local relation have not been fully explored in Transformers for object Re-ID. In this work, we first explore the influence of global and local features of ViT and then further propose a novel Global–Local Transformer (GLTrans) for high-performance object Re-ID. We find that the features from last few layers of ViT already have a strong representational ability, and the global and local information can mutually enhance each other. Based on this fact, we propose a Global Aggregation Encoder (GAE) to utilize the class tokens of the last few Transformer layers and learn comprehensive global features effectively. Meanwhile, we propose the Local Multi-layer Fusion (LMF) which leverages both the global cues from GAE and multi-layer patch tokens to explore the discriminative local representations. Extensive experiments demonstrate that our proposed method achieves superior performance on four object Re-ID benchmarks. The code is available at https://github.com/AWangYQ/GLTrans.

The challenge in multi-focus image fusion tasks lies in accurately preserving the complementary information from the source images in the fused image. However, existing datasets often lack ground truth images, making it difficult for some full-reference loss functions (such as SSIM) to effectively participate in model training, thereby further affecting the performance of retaining source image details. To address this issue, this paper proposes an unsupervised dual-channel dense convolutional method, DCD, for multi-focus image fusion. DCD designs Patch processing blocks specifically for the fusion task, which segment the source image pairs into equally sized patches and evaluate their information to obtain a reconstructed image and a set of adaptive weight coefficients. The reconstructed image is used as the reference image, enabling unsupervised methods to utilize full-reference loss functions in training and overcoming the challenge of lacking labeled data in the training set. Furthermore, considering that the human visual system (HVS) is more sensitive to brightness than color, DCD trains the dual-channel network using both RGB images and their luminance components. This allows the network to focus more on the brightness information while preserving the color and gradient details of the source images, resulting in fused images that are more compatible with the HVS. The adaptive weight coefficients obtained through the Patch processing blocks are also used to determine the degree of preservation of the brightness information in the source images. Finally, comparative experiments on different datasets also demonstrate the superior performance of DCD in terms of fused image quality compared to other methods.

This paper proposes a novel real-time method for high-quality view interpolation from light field. The proposal is a lightweight method, which can be used with consumer GPU, reaching same or better quality than existing methods, in a shorter time, with significantly smaller memory requirements. Light field belongs to image-based rendering methods that can produce realistic images without computationally demanding algorithms. The novel view is synthesized from multiple input images of the same scene, captured at different camera positions. Standard rendering techniques, such as rasterization or ray-tracing, are limited in terms of quality, memory footprint, and speed. Light field rendering methods often produce unwanted artifacts resembling ghosting or blur in certain parts of the scene due to unknown geometry of the scene. The proposed method estimates the geometry for each pixel as an optimal focusing distance to mitigate the artifacts. The focusing distance determines which pixels from the input images are mixed to produce the final view. State-of-the-art methods use a constant-step pixel matching scan that iterates over a range of focusing distances. The scan searches for a distance with the smallest color dispersion of the contributing pixels, assuming that they belong to the same spot in the scene. The paper proposes an optimal scanning strategy of the focusing range, an improved color dispersion metric, and other minor improvements, such as sampling block size adjustment, out-of-bounds sampling, and filtering. Experimental results show that the proposal uses less resources, achieves better visual quality, and is significantly faster than existing light field rendering methods. The proposal is $8\times$ faster than the methods in the same category. The proposal uses only four closest views from the light field data and reduces the necessary data transfer. Existing methods often require the full light field grid, which is typically 8 × 8 images large. Additionally, a new 4K light field dataset, containing scenes of various types, was created and published. An optimal novel method for light field acquisition is also proposed and used to create the dataset.

Deep generative models have shown impressive results in generating realistic images of faces. GANs managed to generate high-quality, high-fidelity images when conditioned on semantic masks, but they still lack the ability to diversify their output. Diffusion models partially solve this problem and are able to generate diverse samples given the same condition. This paper introduces a novel strategy for enhancing diffusion models through multi-conditioning, harnessing cross-attention mechanisms to utilize multiple feature sets, ultimately enabling the generation of high-quality and controllable images. The proposed method extends previous approaches by introducing conditioning on both attributes and semantic masks, ensuring finer control over the generated face images. In order to improve the training time and the generation quality, the impact of applying perceptual-focused loss weighting into the latent space instead of the pixel space is also investigated. The proposed solution has been evaluated on the CelebA-HQ dataset, and it can generate realistic and diverse samples while allowing for fine-grained control over multiple attributes and semantic regions. Experiments on the DeepFashion dataset have also been performed in order to analyze the capability of the proposed model to generalize to different domains. In addition, an ablation study has been conducted to evaluate the impact of different conditioning strategies on the quality and diversity of the generated images.

While the under-display camera (UDC) system provides an effective solution for notch-free full-screen displays, it inevitably causes severe image quality degradation due to the diffraction phenomenon. Recent methods have achieved decent performance with deep neural networks, yet the characteristic of the point spread function (PSF) is less studied. In this paper, considering the large support and spatial inconsistency of PSF, we propose De${}^2$Net for UDC image restoration with feature deconvolution and kernel decomposition. In terms of feature deconvolution, we introduce Wiener deconvolution as a preliminary process, which alleviates feature entanglement caused by the large PSF support. Besides, the deconvolution kernel can be learned from training images, eliminating the tedious PSF-obtaining process. As for kernel decomposition, we observe regular patterns for PSFs at different positions. Thus, with a kernel prediction network (KPN) deployed for handling the spatial inconsistency problem, we improve it from two aspects, i.e., (i) decomposing the predicted kernels into a set of bases and weights, (ii) decomposing kernels into groups with different dilation rates. These modifications largely improve the receptive field under certain memory limits. Extensive experiments on three commonly used UDC datasets show that De${}^2$Net outperforms existing methods both quantitatively and qualitatively. Source code and pre-trained models are available at https://github.com/HyZhu39/De2Net.

Generalizing to out-of-distribution (OOD) data is a challenging task for existing deep learning approaches. This problem largely comes from the common but often incorrect assumption of statistical learning algorithms that the source and target data come from the same i.i.d. distribution. To tackle the limited variability of domains available during training, as well as domain shifts at test time, numerous approaches for domain generalization have focused on generating samples from new domains. Recent studies on this topic suggest that feature statistics from instances of different domains can be mixed to simulate synthesized images from a novel domain. While this simple idea achieves state-of-art results on various domain generalization benchmarks, it ignores structural information which is key to transferring knowledge across different domains. In this paper, we leverage the ability of humans to recognize objects using solely their structural information (prominent region contours) to design a Structural-Aware Feature Stylization method for domain generalization. Our method improves feature stylization based on mixing instance statistics by enforcing structural consistency across the different style-augmented samples. This is achieved via a multi-task learning model which classifies original and augmented images while also reconstructing their edges in a secondary task. The edge reconstruction task helps the network preserve image structure during feature stylization, while also acting as a regularizer for the classification task. Through quantitative comparisons, we verify the effectiveness of our method upon existing state-of-the-art methods on PACS, VLCS, OfficeHome, DomainNet and Digits-DG. The implementation is available at this repository.