IET Computer Vision最新文献

We introduce the shape-image multimodal network (SIM-Net), a novel 2D image classification architecture that integrates 3D point cloud representations inferred directly from RGB images. Our key contribution lies in a pixel-to-point transformation that converts 2D object masks into 3D point clouds, enabling the fusion of texture-based and geometric features for enhanced classification performance. SIM-Net is particularly well-suited for the classification of digitised herbarium specimens—a task made challenging by heterogeneous backgrounds, nonplant elements, and occlusions that compromise conventional image-based models. To address these issues, SIM-Net employs a segmentation-based preprocessing step to extract object masks prior to 3D point cloud generation. The architecture comprises a CNN encoder for 2D image features and a PointNet-based encoder for geometric features, which are fused into a unified latent space. Experimental evaluations on herbarium datasets demonstrate that SIM-Net consistently outperforms ResNet101, achieving gains of up to 9.9% in accuracy and 12.3% in F-score. It also surpasses several transformer-based state-of-the-art architectures, highlighting the benefits of incorporating 3D structural reasoning into 2D image classification tasks.

In recent years, the development of deep learning algorithms has significantly advanced the application of synthetic aperture radar (SAR) aircraft detection in remote sensing and military fields. However, existing methods face a dual dilemma: CNN-based models suffer from insufficient detection accuracy due to limitations in local receptive fields, whereas Transformer-based models improve accuracy by leveraging attention mechanisms but incur significant computational overhead due to their quadratic complexity. This imbalance between accuracy and efficiency severely limits the development of SAR aircraft detection. To address this problem, this paper propose a novel neural network based on state space models (SSM), termed the Mamba SAR detection network (MSAD). Specifically, we design a feature encoding module, MEBlock, that integrates CNN with SSM to enhance global feature modelling capabilities. Meanwhile, the linear computational complexity brought by SSM is superior to that of Transformer architectures, achieving a reduction in computational overhead. Additionally, we propose a context-aware feature fusion module (CAFF) that combines attention mechanisms to achieve adaptive fusion of multi-scale features. Lastly, a lightweight parameter-shared detection head (PSHead) is utilised to effectively reduce redundant parameters through implicit feature interaction. Experiments on the SAR-AirCraft-v1.0 and SADD datasets show that MSAD achieves higher accuracy than existing algorithms, whereas its GFLOPs are 2.7 times smaller than those of the Transformer architecture RT-DETR. These results validate the core role of SSM as an accuracy-efficiency balancer, reflecting MSAD's perceptual capability and performance in SAR aircraft detection in complex environments.

We present MRF (multi-attention fusion artistic radiance fields), a novel approach to 3D scene stylisation that synthesises artistic rendering by integrating stylised 2D images with neural radiance fields. Our method effectively incorporates high-frequency stylistic elements from 2D artistic representations while maintaining geometric consistency across multiple viewpoints. To address the challenges of view-dependent stylisation coherence and semantic fidelity, we introduce two key components: (1) a multi-scale attention module (MAM) that facilitates hierarchical feature extraction and fusion across different spatial resolutions and (2) a CLIP-guided semantic consistency module that preserves the underlying scene structure during style transfer. Through extensive experimentation, we demonstrate that MRF achieves superior stylisation quality and detail preservation compared to state-of-the-art methods, particularly in capturing fine artistic details while maintaining view consistency. Our approach represents a significant advancement in neural rendering-based artistic stylisation of 3D scenes.

High-accuracy point cloud (self-) reconstruction is crucial for point cloud editing, translation, and unsupervised representation learning. However, existing point cloud reconstruction methods often sacrifice many geometric details. Altough many techniques have proposed how to construct better point cloud decoders, only a few have designed point cloud encoders from a reconstruction perspective. We propose an autodecoder architecture to achieve detail-preserving point cloud reconstruction while bypassing the performance bottleneck of the encoder. Our architecture is theoretically applicable to any existing point cloud decoder. For training, both the weights of the decoder and the pre-initialised latent codes, corresponding to the input points, are updated simultaneously. Experimental results demonstrate that our autodecoder achieves an average reduction of 24.62% in Chamfer Distance compared to existing methods, significantly improving reconstruction quality on the ShapeNet dataset. Furthermore, we verify the effectiveness of our autodecoder in point cloud generation, upsampling, and unsupervised representation learning to demonstrate its performance on downstream tasks, which is comparable to the state-of-the-art methods. We will make our code publicly available after peer review.

As we all know, the emergence of a large number of adversarial samples reveals the vulnerability of deep neural networks. Attackers seriously affect the performance of models by adding imperceptible perturbations. Although adversarial samples have a high transferability success rate in white-box models, they are less effective in black-box models. To address this problem, this paper proposes a new transferability attack strategy, Gradient Related Variance and Input Transformation Attack (GRVT). First, the image is divided into small blocks, and random transformations are applied to each block to generate diversified images; then, in the gradient update process, the gradient of the neighbourhood area is introduced, and the current gradient is associated with the neighbourhood average gradient through Cosine Similarity. The current gradient direction is adjusted using the associated gradient combined with the previous gradient variance, and a step size reducer adjusts the gradient step size. Experiments on the ILSVRC 2012 dataset show that the transferability success rate of adversarial samples between convolutional neural network (CNN) and vision transformer (ViT) models is higher than that of currently advanced methods. Additionally, the adversarial samples generated on the ensemble model are practical against nine defence strategies. GRVT shows excellent transferability and broad applicability.

Weakly supervised semantic segmentation (WSSS) methods are extensively studied due to the availability of image-level annotations. Relying on class activation maps (CAMs) derived from original classification networks often suffers from issues such as inaccurate object localization, incomplete object regions, and the inclusion of confusing background pixels. To address these issues, we propose a two-stage method that enhances the foreground–background discriminative ability in a global context (FB-DGC). Specifically, a cross-domain feature calibration module (CFCM) is first proposed to calibrate foreground and background salient features using global spatial location information, thereby expanding foreground features while mitigating the impact of inaccurate localization in class activation regions. A class-specific distance module (CSDM) is further adopted to facilitate the separation of foreground–background features, thereby enhancing the activation of target regions, which alleviates the over-smoothing of features produced by the network and mitigates issues associated with confused features. In addition, an adaptive edge feature extraction (AEFE) strategy is proposed to identify target features in candidate boundary regions and capture missed features, compensating for drawbacks in recognising the co-occurrence of multiple targets. The proposed method is extensively evaluated on the challenging PASCAL VOC 2012 and MS COCO 2014 datasets, demonstrating its feasibility and superiority.

RGB and thermal salient object detection (RGB-T SOD) aims to accurately locate and segment salient objects in aligned visible and thermal image pairs. However, existing methods often struggle to produce complete masks and sharp boundaries in challenging scenarios due to insufficient exploration of complementary features from the dual modalities. In this paper, we propose a novel mamba-based fusion network for RGB-T SOD task, named Mamba4SOD, which integrates the strengths of Swin Transformer and Mamba to construct robust multi-modal representations, effectively reducing pixel misclassification. Specifically, we leverage Swin Transformer V2 to establish long-range contextual dependencies and thoroughly analyse the impact of features at various levels on detection performance. Additionally, we develop a novel Mamba-based fusion module with linear complexity, boosting multi-modal enhancement and fusion. Experimental results on VT5000, VT1000 and VT821 datasets demonstrate that our method outperforms the state-of-the-art RGB-T SOD methods.

Object detection is the most crucial and challenging task of computer vision and has been used in various fields in recent years, such as autonomous driving and industrial inspection. Traditional object detection methods are mainly based on the sliding windows and the handcrafted features, which have problems such as insufficient understanding of image features and low accuracy of detection. With the rapid advancements in deep learning, convolutional neural networks (CNNs) and vision transformers have become fundamental components in object detection models. These components are capable of learning more advanced and deeper image properties, leading to a transformational breakthrough in the performance of object detection. In this review, we comprehensively review the representative object detection models from deep learning periods, tracing their architectural shifts and technological breakthroughs. Furthermore, we discuss key challenges and promising research directions in the object detection. This review aims to provide a comprehensive foundation for practitioners to enhance their understanding of object detection technologies.

The visual dialogue task requires computers to comprehend image content and preceding question-and-answer history to accurately answer related questions, with each round of dialogue providing the necessary historical context for subsequent interactions. Existing research typically processes multiple questions related to a single image as independent samples, which results in redundant modelling of the images and their captions and substantially increases computational costs. To address the challenges above, we introduce a fast transformer for visual dialogue, termed FastVDT, which utilises novel attention masks and continuous positional encoding. FastVDT models multiple image-related questions as an integrated entity, accurately processing prior conversation history in each dialogue round while predicting answers to multiple questions. Our method effectively captures the interrelations among questions and significantly reduces computational overhead. Experimental results demonstrate that our method delivers outstanding performance on the VisDial v0.9 and v1.0 datasets. FastVDT achieves comparable performance to VD-BERT and VU-BERT while reducing computational costs by 80% and 56%, respectively.

Adverse weather conditions in real-world scenarios can degrade the performance of deep learning-based object detection models. A commonly used approach is to apply image restoration before object detection to improve degraded images. However, there is no direct correlation between the visual quality of image restoration and the object detection accuracy. Furthermore, image restoration and object detection have potential conflicting objectives, making joint optimisation difficult. To address this, we propose an end-to-end object detection network specifically designed for rainy and foggy conditions. Our approach cascades an image restoration subnetwork with a detection subnetwork and optimises them jointly through a shared objective. Specifically, we introduce an expanded dilated convolution block and a weather attention block to enhance the effectiveness and robustness of the restoration network under various weather degradations. Additionally, we incorporate an auxiliary alignment branch with feature alignment loss to align the features of restored and clean images within the detection backbone, enabling joint optimisation of both subnetworks. A novel training strategy is also proposed to further improve object detection performance under rainy and foggy conditions. Extensive experiments on the vehicle-rain-fog, VOC-fog and real-world fog datasets demonstrate that our method outperforms recent state-of-the-art approaches in image restoration quality and detection accuracy. The code is available at https://github.com/HappyPessimism/RainFog-Restoration-Detection.