Computer Vision and Image Understanding最新文献

Methods based on class activation maps (CAM) provide a simple mechanism to interpret predictions of convolutional neural networks by using linear combinations of feature maps as saliency maps. By contrast, masking-based methods optimize a saliency map directly in the image space or learn it by training another network on additional data. In this work we introduce Opti-CAM, combining ideas from CAM-based and masking-based approaches. Our saliency map is a linear combination of feature maps, where weights are optimized per image such that the logit of the masked image for a given class is maximized. We also fix a fundamental flaw in two of the most common evaluation metrics of attribution methods. On several datasets, Opti-CAM largely outperforms other CAM-based approaches according to the most relevant classification metrics. We provide empirical evidence supporting that localization and classifier interpretability are not necessarily aligned.

Pedestrian trajectory prediction plays a key role in understanding human behavior and guiding autonomous driving. It is a difficult task due to the multi-modal nature of human motion. Recent advances have mainly focused on modeling this multi-modality, either by using implicit generative models or explicit pre-defined anchors. However, the former is limited by the sampling problem, while the latter introduces strong prior to the data, both of which require extra tricks to achieve better performance. To address these issues, we propose a simple yet effective framework called Efficient Multi-modal Predictors (EMP), which casts off the generative paradigm and predicts multi-modal trajectories in an end-to-end style. It is achieved by combining a set of parallel predictors with a model error based sparse selector. During training, the entire set of parallel multi-modal predictors will converge into disjoint subsets, with each subset specializing in one mode, thus obtaining multi-modal prediction with no human prior and reducing the problems of above two genres. Experiments on SDD/ETH-UCY/NBA datasets show that EMP achieves state-of-the-art performance with the highest inference speed. Additionally, we show that by replacing multi-modal modules with EMP, state-of-the-art works outperform their baselines, which further validate the versatility of EMP. Moreover, we formally prove that EMP can alleviate the problem of modal collapse and has a low test error bound.

Super-resolution image reconstruction techniques have advanced quickly, leading to the generation of a sizable number of super-resolution images using different super-resolution techniques. Nevertheless, accurately assessing the quality of super-resolution images remains a formidable challenge. This paper introduces a novel Multi-Frequency Cascade Transformers (MFCT) for evaluating super-resolution image quality (SR-IQA). In the first step, we develop a unique Frequency-Divided Module (FDM) to transform the super-resolution images into three different frequency bands. Subsequently, the Cascade Transformer Blocks (CAF) incorporating hierarchical self-attention mechanisms are employed to capture cross-window features for quality perception. Ultimately, the image quality scores from different frequency bands are fused to derive the overall image quality score. The experimental results show that, on the chosen SR-IQA databases, the proposed MFCT-based SR-IQA method can consistently outperforms all the compared Image Quality Assessment (IQA) models. Furthermore, a collection of thorough ablation studies demonstrates that, when compared to other earlier rivals, the newly proposed approach exhibits impressive generalization ability. The code will be available at https://github.com/kbzhang0505/MFCT.

Detection Transformer (DETR) and its variants have emerged a new paradigm to object detection, but their high computational cost hinders practical applications. By investigating their essential components, we found that the transformer-based head usually occupies a significant amount of computation. Through further comparing heavy and light transformer heads, we observed that both heads produced satisfactory results for easy images while showing a noticeable difference for hard images. Inspired by these findings, we propose a dynamic head switching (DHS) strategy to dynamically select the proper head for each image at inference for a better balance of efficiency and accuracy. Specifically, our DETR model incorporates multiple heads with different computational complexity and a lightweight module which selects proper heads for given images. This module is optimized to maximize detection accuracy while adhering to the overall computational budget limitations. To minimize the potential accuracy drop when executing the lighter heads, we propose online head distillation (OHD) to improve the accuracy of the lighter heads with the help of the heavier head. Extensive experiments on the MS COCO dataset validated the effectiveness of the proposed method, which demonstrated a better accuracy–efficiency trade-off compared to the baseline using static heads.

Multi-view 3D shape classification, which identifies a 3D shape based on its 2D views rendered from different viewpoints, has emerged as a promising method of shape understanding. A key building block in these methods is cross-view feature aggregation. However, existing methods dominantly follow the “extract-then-aggregate” pipeline for view-level global feature aggregation, leaving cross-view pixel-level feature interaction under-explored. To tackle this issue, we develop a “fuse-while-extract” pipeline, with a novel View-aligned Pixel-level Fusion (VPF) module to fuse cross-view pixel-level features originating from the same 3D part. We first reconstruct the 3D coordinate of each feature via the rasterization results, then match and fuse the features via spatial neighbor searching. Incorporating the proposed VPF module with ResNet18 backbone, we build a novel view-aligned multi-view network, which conducts feature extraction and cross-view fusion alternatively. Extensive experiments have demonstrated the effectiveness of the VPF module as well as the excellent performance of the proposed network.

Hyperspectral (HS) image always suffers from the deficiency of low spatial resolution, compared with conventional optical image types, which has limited its further applications in remote sensing areas. Therefore, HS image super-resolution (SR) techniques are broadly employed in order to observe finer spatial structures while preserving the spectra of ground covers. In this paper, a novel multi-dimensional attention-aided transposed convolutional long-short term memory (LSTM) network is proposed for single HS image super-resolution task. The proposed network employs the convolutional bi-directional LSTM for the purpose of local and non-local spatial–spectral feature explorations, and transposed convolution for the purpose of image amplification and reconstruction. Moreover, a multi-dimensional attention module is proposed, aiming to capture the salient features on spectral, channel, and spatial dimensions, simultaneously, to further improve the learning abilities of network. Experiments on four commonly-used HS images demonstrate the effectiveness of this approach, compared with several state-of-the-art deep learning-based SR methods.

The prominence of high-quality video services has become so substantial that by 2030, it is estimated that approximately 80% of internet traffic will consist of videos. On the contrary, video denoising remains a relatively unexplored and intricate field, presenting more substantial challenges compared to image denoising. Many published deep learning video denoising algorithms typically rely on simple, efficient single encoder–decoder networks, but they have inherent limitations in preserving intricate image details and effectively managing noise information propagation for noise residue modelling. In response to these challenges, the proposed work introduces an innovative approach; in terms of utilization of cascaded UNets for progressive noise residual prediction in video denoising. This multi-stage encoder–decoder architecture is meticulously designed to accurately predict noise residual maps, thereby preserving the locally fine details within video content as represented by SSIM. The proposed network has undergone extensive end-to-end training from scratch without explicit motion compensation to reduce complexity. In terms of the more rigorous SSIM metric, the proposed network outperformed all video denoising methods while maintaining a comparable PSNR.

Eye gaze provides valuable cues about human intent, making gaze estimation a hot topic. Extracting multi-scale information has recently proven effective for gaze estimation in complex scenarios. However, existing methods for estimating gaze based on multi-scale features tend to focus only on information from single-level feature maps. Furthermore, information across different scales may also lack relevance. To address these issues, we propose a novel joint pyramidal perceptual attention and hierarchical consistency constraint (PaCo) for gaze estimation. The proposed PaCo consists of two main components: pyramidal perceptual attention module (PPAM) and hierarchical consistency constraint (HCC). Specifically, PPAM first extracts multi-scale spatial features using a pyramid structure, and then aggregates information from coarse granularity to fine granularity. In this way, PPAM enables the model to simultaneously focus on both the eye region and facial region at multiple scales. Then, HCC makes constrains consistency on low-level and high-level features, aiming to enhance the gaze semantic consistency between different feature levels. With the combination of PPAM and HCC, PaCo can learn more discriminative features in complex situations. Extensive experimental results show that PaCo achieves significant performance improvements on challenging datasets such as Gaze360, MPIIFaceGaze, and RT-GENE,reducing errors to 10.27$°$, 3.23$°$, 6.46$°$, respectively.

While convolutional neural networks (CNN) have achieved remarkable performance in single image deraining tasks, it is still a very challenging task due to CNN’s limited receptive field and the unreality of the output image. In this paper, UC-former, an effective and efficient U-shaped architecture based on transformer for image deraining was presented. In UC-former, there are two core designs to avoid heavy self-attention computation and inefficient communications across encoder and decoder. First, we propose a novel channel across Transformer block, which computes self-attention between channels. It significantly reduces the computational complexity of high-resolution rain maps while capturing global context. Second, we propose a multi-scale feature fusion module between the encoder and decoder to combine low-level local features and high-level non-local features. In addition, we employ depth-wise convolution and H-Swish non-linear activation function in Transformer Blocks to enhance rain removal authenticity. Extensive experiments indicate that our method outperforms the state-of-the-art deraining approaches on synthetic and real-world rainy datasets.

The widespread use of various chemical gases in industrial processes necessitates effective measures to prevent their leakage during transportation and storage, given their high toxicity. Thermal infrared-based computer vision detection techniques provide a straightforward approach to identify gas leakage areas. However, the development of high-quality algorithms has been challenging due to the low texture in thermal images and the lack of open-source datasets. In this paper, we present the RGB-Thermal Cross Attention Network (RT-CAN), which employs an RGB-assisted two-stream network architecture to integrate texture information from RGB images and gas area information from thermal images. Additionally, to facilitate the research of invisible gas detection, we introduce Gas-DB, an extensive open-source gas detection database including about 1.3K well-annotated RGB-thermal images with eight variant collection scenes. Experimental results demonstrate that our method successfully leverages the advantages of both modalities, achieving state-of-the-art (SOTA) performance among RGB-thermal methods, surpassing single-stream SOTA models in terms of accuracy, Intersection of Union (IoU), and F2 metrics by 4.86%, 5.65%, and 4.88%, respectively. The code and data can be found at https://github.com/logic112358/RT-CAN.