IET Computer Vision最新文献

Convolutional sparse coding (CSC) using learnt convolutional dictionaries has recently emerged as an effective technique for emphasising discriminative structures in signal and image processing applications. In this paper, we propose a multilayer model for convolutional sparse networks (CSNs), based on hierarchical convolutional sparse coding and dictionary learning, as a competitive alternative to conventional deep convolutional neural networks (CNNs). In the proposed CSN architecture, each layer learns a convolutional dictionary from the feature maps of the preceding layer (if available), and then uses it to extract sparse representations. This hierarchical process is repeated to obtain high-level feature maps in the final layer, suitable for pattern recognition and classification tasks. One key advantage of the CSN framework is its reduced sensitivity to training set size and its significantly lower computational complexity compared to CNNs. Experimental results on image classification tasks show that the proposed model achieves up to 7% higher accuracy than CNNs when trained with only 150 samples, while reducing computational cost by at least 50% under similar conditions.

With urbanisation accelerating, predicting the heat release rate (HRR) of building fires using visual data has emerged as a pivotal research focus in the field of fire rescue. However, existing approaches face challenges, such as limited training data and complex models, which lead to suboptimal performance and slow inference speeds. To address these issues and adapt to the rapid morphological changes of smoke in dynamic fire environments, we propose a lightweight neural network prediction model based on adaptive pooling with channel information interaction (APCI). This model can achieve high precision while maintaining faster inference speed. Our approach employs simplified dense connections to propagate shallow smoke features, thereby effectively capturing the relationship between smoke textures and multiscale features to accommodate the variations of smoke morphologies. To mitigate the loss of smoke features caused by spatial misalignment and ventilation disturbances during downsampling, we introduce an adaptive weighted pooling mechanism that fully leverages the detailed information contained in the invoked smoke. Additionally, an enhanced channel shuffle operation in channel information interaction ensures effective cross-level transfer to detail-aware information exchange during sudden escalations in fire intensity in the hybrid feature fusion framework. Experiments on the smoke-heat release rate dataset we created demonstrate that the proposed method can achieve a coefficient of determination $��\left(R^2\right)�$ of 0.937, a root mean square error (RMSE) of 23.0 kW, a mean absolute error (MAE) of 17.4 kW and with an inference time of 4.13 ms per image.

Action Quality Assessment (AQA) has become an advanced technology applied in various domains. However, most existing datasets focus on sports events, such as the Olympics, whereas datasets tailored for daily exercise activities remain scarce. Additionally, many of these datasets are unsuitable for direct application in AQA tasks. To address these limitations, we constructed a new AQA dataset, TaiChi-AQA, which includes detailed scoring annotations. Our dataset comprises 1313 Tai Chi action videos and features a comprehensive set of fine-grained labels, including action labels, action descriptions and frame-level perspective information. To validate the effectiveness of TaiChi-AQA, we systematically evaluated it using a variety of popular AQA methods. We also propose a straightforward yet effective module that integrates a multi-head attention mechanism with a gated multilayer perceptron (gMLP). This module is combined with the distributed autoencoder (DAE) framework. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the TaiChi-AQA dataset. The dataset are publicly available at https://github.com/mlxger/TaiChi-AQA.

Few-shot learning (FSL) aims to classify novel categories using only a few labelled examples, which poses significant challenges for generalisation. Among existing approaches, distribution-based methods have shown promise by constructing class distributions for novel categories using statistical priors transferred from base classes. However, these methods often rely on nearest-neighbour visual similarity and assume equal contributions from selected base classes, which can lead to inaccurate priors. In this paper, we propose CAPE (contribution-aware prior estimation), a method that addresses this issue from two complementary perspectives. On the one hand, CAPE assigns adaptive weights to base class prototypes based on their relevance to the novel support set, mitigating the limitations of equal-contribution assumptions. On the other hand, to compensate for the ambiguity of visual features, especially in the 1-shot scenario, we incorporate semantic information from category labels to enhance prior selection. By jointly leveraging visual and semantic information, CAPE constructs more accurate and robust priors for the feature distributions of novel classes. Extensive experiments on four widely used FSL benchmarks, including mini ImageNet, tieredImageNet, CIFAR-FS and CUB datasets, demonstrate that our method consistently outperforms existing approaches, highlighting the effectiveness of contribution- and semantics-aware prior estimation.

Optimising both trajectory position information and identity information is a key challenge in multiple object tracking. Mainstream approaches ensure ID consistency by combining detection data with various additional information. However, many methods overlook the inherent spatio-temporal correlation of trajectory position information. We argue that additional modules are redundant, and that forecasting trajectories directly without the need for interframe association by utilising motion constraints is adequate. In this study, we introduce a novel end-to-end network called the spatio-temporal multiple object tracking with transformer (STMOTR), which employs motion constraints to establish binary matching within the reconstructed deformable-DETR network, heuristically learning object trajectories from the Video Swin backbone. This subtly constrained matching rule not only keeps the detection ID consistency but also significantly reduces the potential for tracking ID switch. We evaluated STMOTR on the UA-DETRAC and our proposed tunnel multiple object tracking dataset (T-MOT), achieving state-of-the-art performance with 39.8% PR-MOTA on the UA-DETRAC and 79.6% MOTA on the T-MOT. The source code is also available at https://github.com/Jade-Ray/STMOTR.

Hyperspectral imaging (HSI) plays a crucial role in urban area analysis from satellite data and supports the continuous advancement of intelligent cities. However, its practical deployment is hindered by two major challenges: the scarcity of reliable training annotations and the high spectral similarity among different land-cover classes. To address these issues, this paper introduces a novel meta-learning framework that synergistically combines knowledge transfer across domains with a dual-adjustment mode (comprising intracorrection (IC) and interalignment (IA)), while ensuring end-to-end trainability. Our contributions are twofold. (1) We refine the 3D attention network TGAN into TGAN2 (3D ghost attention network v2) by replacing the original ghost blocks with ghost-V2 modules and enlarging the receptive field to capture global context. (2) We propose a dual-adjustment mode (comprising intracorrection (IC) and interalignment (IA)) to generate robust class prototypes and mitigate domain shift. These innovations are integrated into our overarching framework, DMCM2 (dual-adjustment cross-domain meta-learning framework v2), which is unified by its end-to-end trainability and efficiency. The code and models will be publicly available at: https://github.com/YAO-JQ/DMCM2.

This paper introduces SwapDiffusion, a novel framework for content-style disentanglement in diffusion-based image generation. We advance the understanding of the extended textual conditioning ($��P�+�$) space in SDXL by identifying the $��4^{\text{th}}�$ and $��7^{\text{th}}�$ transformer block layers as primarily responsible for content and style, respectively. Building on this insight, we introduce a novel q-transformer architecture. It features a block-diagonal matrix masked self-attention layer that effectively isolates content and style embeddings by reducing inter-query interference. This design not only enhances disentanglement but also improves training efficiency. Crucially, the learnt image embeddings align well with textual ones, enabling flexible content and style control via images, text or their combinations. SwapDiffusion supports diverse applications such as style transfer (image- or text-driven), image variation, stylised text-to-image generation and multimodal-prompted image synthesis. Experimental results demonstrate that by aligning learnt image embeddings with the U-Net's pre-identified functional layers for content and style, SwapDiffusion achieves superior content-style separation and image quality while offering greater adaptability than existing approaches. The implementation code and pre-trained models will be released at https://github.com/lioo717/SwapDiffusion.

This paper presents a novel video captioning method guided by object semantic priors, aimed at improving the performance of vision-based video captioning models. The proposed approach leverages an object detection model to extract semantic representations of objects within image sequences, which are used as prior information to enhance the visual features of the video. During the encoding stage, this prior information is integrated with the video content, enabling a more comprehensive understanding of the visual context. In the decoding stage, the prior information guides the generation of more accurate and contextually appropriate captions. Extensive experiments on the MSVD and MSR-VTT datasets show that the proposed method significantly outperforms existing vision-based video captioning approaches in terms of caption accuracy and relevance. The results validate the effectiveness of incorporating object semantic priors into vision-based models for generating high-quality video captions.

The field of face forgery detection continues to encounter significant challenges in achieving generalisation, and the rapid advancement of generative models particularly diffusion models has further intensified this problem. To tackle these challenges, we propose a novel detection framework that integrates spatial central masking and frequency-enriched diffusion reconstruction (MFDR), thereby enhancing both local detail reconstruction accuracy and global structural recovery. Specifically, during data preprocessing, we apply central masking to reconstruct the original image. The detector learns pixel-level discrepancies between the reconstructed masked regions and the corresponding original regions, which improves sensitivity to reconstruction errors and guides the model to focus more effectively on localised artefact detection. At the frequency-domain level, our proposed frequency-enhanced diffusion module explicitly optimises residual reconstruction in both low- and high-frequency subbands, effectively improving global structural recovery and preserving high-frequency detail fidelity. This, in turn, strengthens the model's capacity to capture forgery traces. Furthermore, during training, we introduce a contrastive learning strategy in which real images processed through masked diffusion and frequency reconstruction are used as positive samples. This design enables the detector to jointly perceive spatial reconstruction errors and preserve frequency-domain texture fidelity, thereby significantly enhancing its ability to detect subtle forgery artefacts. Experimental results show that our method achieves superior performance in detecting face images generated by various diffusion models (e.g., DDPM, LDM) and surpasses the diffusion reconstruction contrastive training (DRCT) baseline.

This paper proposes a unified framework integrating enhanced neural radiance fields (NeRF) with causal feature fusion to tackle 3D reconstruction and 2D/3D alignment challenges in complex scenes. For example, in 3D reconstruction, explicit representations have low reconstruction quality, whereas implicit ones have slow reconstruction speed; 2D/3D matching lacks effective information fusion; furthermore, existing 3D reconstruction methods fail to provide complementary information for alignment and relying solely on 2D alignment is susceptible to background interference. In order to improve 2D/3D alignment accuracy, we propose a holistic alignment architecture, including a combined implicit and explicit 3D reconstruction method, capable of constructing higher-quality 3D scenes and, crucially, generating richer features, such as voxel density information and colour information, to provide better feature complements for background robustness. Meanwhile, we construct 2D causal features and utilise the features for fusion and achieve more robust alignment through multidimensional anti-interference feature computation. Extensive experiments validate our framework on both public benchmarks and specialised domains. In medical endoscopy, the system assists surgeons by providing real-time 3D contextual guidance, reducing procedural risks. Quantitative results show superior performance over state-of-the-art methods. The proposed technology demonstrates broad applicability in scenarios demanding robust scene understanding.