IET Computer Vision最新文献

The increasing adoption of 3D point cloud data in various applications, such as autonomous vehicles, robotics and virtual reality, has brought about significant advancements in object recognition and scene understanding. However, this progress is accompanied by new security challenges, particularly in the form of backdoor attacks. These attacks involve inserting malicious information into the training data of machine learning models, potentially compromising the model's behaviour. In this paper, we propose CloudFort, a novel defence mechanism designed to enhance the robustness of 3D point cloud classifiers against backdoor attacks. CloudFort leverages spatial partitioning and ensemble prediction techniques to effectively mitigate the impact of backdoor triggers while preserving the model's performance on clean data. We evaluate the effectiveness of CloudFort through extensive experiments, demonstrating its strong resilience against the point cloud backdoor attack (PCBA). Our results show that CloudFort significantly enhances the security of 3D point cloud classification models without compromising their accuracy on benign samples. Furthermore, we explore the limitations of CloudFort and discuss potential avenues for future research in the field of 3D point cloud security. The proposed defence mechanism represents a significant step towards ensuring the trustworthiness and reliability of point-cloud-based systems in real-world applications.

Multiobject tracking of ships is crucial for various applications, such as maritime security and the development of ship autopilot systems. However, existing ship visual datasets primarily focus on ship detection tasks, lacking a fully open-source dataset for multiobject tracking research. Furthermore, current methods often struggle with extracting appearance features under complex sea conditions, varying scales and different ship types, affecting tracking precision. To address these issues, we propose ShipsMOT, a new benchmark dataset containing 121 video sequences with an average of 15.45 s per sequence, covering 15 distinct ship types and a total of 237,999 annotated bounding boxes. Additionally, we propose JDR-CSTrack, a ship multiobject tracking framework that improves feature extraction at different scales by optimising a joint detection and Re-ID network. JDR-CSTrack utilises the fusion of appearance and motion features for multilevel data association, thereby minimising track loss and ID switches. Experimental results confirm that ShipsMOT can serve as a benchmark for future research in ship multiobject tracking and validate the superiority of the proposed JDR-CSTrack framework. The dataset and code can be found on https://github.com/jpj0916/ShipsMOT.

Micro-expressions are involuntary facial movements that reveal a person's true emotions when attempting to conceal them. These expressions hold significant potential for various applications. However, due to their brief duration and subtle manifestation, detailed features are often obscured by redundant information, making micro-expression recognition challenging. Previous studies have primarily relied on convolutional neural networks (CNNs) to process high-resolution images or optical flow features, but the complexity of deep networks often introduces redundancy and leads to overfitting. In this paper, we propose EIRN, a novel method for micro-expression recognition. Unlike conventional approaches, EIRN explicitly separates facial features of different granularities, using shallow networks to extract sparse features from low-resolution greyscale images, while treating onset–apex pairs as Siamese samples and employing a Siamese neural network (SNN) to extract dense features from high-resolution counterparts. These multigranularity features are then integrated for accurate classification. To mitigate overfitting in fine-grained feature extraction by the SNN, we introduce an attention module tailored to enhance crucial feature representation from both onset and apex frames during training. Experimental results on single and composite datasets demonstrate the effectiveness of our approach and its potential for real-world applications.

Transformers have achieved remarkable success in computer vision, but their reliance on self-attention mechanisms poses challenges for small-scale datasets due to high computational demands and data requirements. This paper introduces the Multi-Head Adaptive Filter Frequency Vision Transformer (MAF-FViT), a Vision Transformer model that replaces self-attention with frequency-domain adaptive filters. MAF-FViT leverages multi-head adaptive filtering in the frequency domain to capture essential features with reduced computational complexity, providing an efficient alternative for vision tasks on limited data. Training is carried out from scratch without the need for pretraining on large-scale datasets. The proposed MAF-FViT model demonstrates strong performance on various image classification tasks, achieving competitive accuracy with a lower parameter count and faster processing times compared to self-attention-based models and other models employing alternative token mixers. The multi-head adaptive filters enable the model to capture complex image features effectively, preserving high classification accuracy while minimising computational load. The results demonstrate that frequency-domain adaptive filters offer an effective alternative to self-attention, enabling competitive performance on small-scale datasets while reducing training time and memory requirements. MAF-FViT opens avenues for resource-efficient transformer models in vision applications, making it a promising solution for settings constrained by data or computational resources.

Skeleton-based temporal action segmentation aims to segment and classify human actions in untrimmed skeletal sequences. Existing methods struggle with distinguishing transition poses between adjacent frames and fail to adequately capture semantic dependencies between joints and actions. To address these challenges, we propose a large language model-based spatio-temporal semantic enhancement (LLM-STSE) method, a novel framework that combines adaptive spatio-temporal axial attention (ASTA-Attention) and dynamic semantic-guided multimodal action segmentation (DSG-MAS). ASTA-Attention models spatial and temporal dependencies using axial attention, whereas DSG-MAS dynamically generates semantic prompts based on joint motion and fuses them with skeleton features for more accurate segmentation. Experiments on MCFS and PKU-MMD datasets show that LLM-STSE achieves state-of-the-art performance, significantly improving action segmentation, especially in complex transitions, with substantial F1 score gains across multiple public datasets.

The transformer's capabilities in natural language processing and computer vision are impressive, but interpretability is crucial in specific domain applications. The NesT model, with its pyramidal structure, demonstrates high accuracy and faster training speeds. Unlike other models, a unique aspect of NesT is its avoidance of the [CLS] token, which presents challenges when applying interpretability methods that rely on the model's internal structure. Instead, NesT divides the image into 16 blocks and processes them using 16 independent vision transformers. We propose the NesT-Shapley method, which utilises this structure to combine the Shapley value method (a self-interpretable approach) with the independently operating vision transformers within NesT, significantly reducing computational complexity. On the other hand, we introduced the feature weight augmentation (FWA) method to address the challenges of weight adjustment in the final interpretability results produced by interpretability methods without [CLS] token, markedly enhancing the performance of interpretability methods and providing a better understanding of the information flow during the prediction process in the NesT model. We conducted perturbation experiments on the NesT model using the ImageNet and CIFAR-100 datasets and segmentation experiments on the ImageNet-Segmentation dataset, achieving impressive experimental results.

Segmenting unlabelled medical images with a minimal amount of labelled data is a daunting task due to the complex feature landscapes and the prevalent noise and artefacts characteristic of medical imaging processes. The SAM has showcased the potential of large-scale image segmentation models for achieving zero-shot generalisation across previously unseen objects. However, directly applying SAM to medical image segmentation without incorporating prior knowledge of the target task can lead to unsatisfactory results. To address this, we enhance SAM by integrating prior knowledge of medical image segmentation tasks. This enables it to quickly adapt to few-shot medical image segmentation tasks while ensuring efficient parameter training. Our method employs an ensemble learning strategy to train a simple classifier, producing a coarse mask for each test image. Importantly, this coarse mask generates more accurate prompt points and boxes, thus improving SAM's capacity for prompt-driven segmentation. Furthermore, to refine SAM's ability to produce more precise masks, we introduce the Isolated Noise Removal (INR) module, which efficiently removes noise from the coarse masks. In addition, our novel Multi-point Automatic Prompt (MPAP) module is designed to independently generate multiple effective and evenly distributed point prompts based on these coarse masks. Additionally, we introduce an innovative knee joint dataset benchmark specifically for medical image segmentation, contributing further to the research field. Extensive evaluations on three benchmark datasets confirm the superior performance of our approach compared to existing methods, demonstrating its efficacy and significant progress in the domain of few-shot medical image segmentation.

The long-tailed distribution of training samples is a fundamental challenge in human-object interaction (HOI) detection, leading to extremely imbalanced performance on non-rare and rare classes. Existing works adopt the idea of compositional learning, in which object and action features are learnt individually and re-composed into new samples of rare HOI classes. However, most of these methods are proposed on traditional CNN-based frameworks which are weak in capturing image-wide context. Moreover, the simple feature integration mechanisms fail to aggregate effective semantics in re-composed features. As a result, these methods achieve only limited improvements on knowledge generalisation. We propose a novel transformer-based compositional learning framework for HOI detection. Human-object pair features and interaction features containing rich global context are extracted, and comprehensively integrated via the cross-attention mechanism, generating re-composed features containing more generalisable semantics. To further improve re-composed features and promote knowledge generalisation, we leverage the vision-language model CLIP in a computation-efficient manner to improve re-composition sampling and guide the interaction feature learning. Experiments on two benchmark datasets prove the effectiveness of our method in improving performance on both rare and non-rare HOI classes.

Sign language translation based on machine learning plays a crucial role in facilitating communication between deaf and hearing individuals. However, due to the complexity and variability of sign language, coupled with limited observation angles, single-view sign language translation models often underperform in real-world applications. Although some studies have attempted to improve translation efficiency by incorporating multiview data, challenges, such as feature alignment, fusion, and the high cost of capturing multiview data, remain significant barriers in many practical scenarios. To address these issues, we propose a multiview distillation transformer model (MVDT) for continuous sign language translation. The MVDT introduces a novel distillation mechanism, where a teacher model is designed to learn common features from multiview data, subsequently guiding a student model to extract view-invariant features using only single-view input. To evaluate the proposed method, we construct a multiview sign language dataset comprising five distinct views and conduct extensive experiments comparing the MVDT with state-of-the-art methods. Experimental results demonstrate that the proposed model exhibits superior view-invariant translation capabilities across different views.

Accurate perception and understanding of the three-dimensional environment is crucial for autonomous vehicles to navigate efficiently and make wise decisions. However, in complex real-world scenarios, the information obtained by a single-modal sensor is often incomplete, severely affecting the detection accuracy of occluded targets. To address this issue, this paper proposes a novel adaptive multi-scale attention aggregation strategy, efficiently fusing multi-scale feature representations of heterogeneous data to accurately capture the shape details and spatial relationships of targets in three-dimensional space. This strategy utilises learnable sparse keypoints to dynamically align heterogeneous features in a data-driven manner, adaptively modelling the cross-modal mapping relationships between keypoints and their corresponding multi-scale image features. Given the importance of accurately obtaining the three-dimensional shape information of targets for understanding the size and rotation pose of occluded targets, this paper adopts a shape prior knowledge-based constraint method and data augmentation strategy to guide the model to more accurately perceive the complete three-dimensional shape and rotation pose of occluded targets. Experimental results show that our proposed model achieves 2.15%, 3.24% and 2.75% improvement in 3D_R40 mAP score under the easy, moderate and hard difficulty levels compared to MVXNet, significantly enhancing the detection accuracy and robustness of occluded targets in complex scenarios.