Computer Vision and Image Understanding最新文献

Non-nutritive sucking (NNS), which refers to the act of sucking on a pacifier, finger, or similar object without nutrient intake, plays a crucial role in assessing healthy early development. In the case of preterm infants, NNS behavior is a key component in determining their readiness for feeding. In older infants, the characteristics of NNS behavior offer valuable insights into neural and motor development. Additionally, NNS activity has been proposed as a potential safeguard against sudden infant death syndrome (SIDS). However, the clinical application of NNS assessment is currently hindered by labor-intensive and subjective finger-in-mouth evaluations. Consequently, researchers often resort to expensive pressure transducers for objective NNS signal measurement. To enhance the accessibility and reliability of NNS signal monitoring for both clinicians and researchers, we introduce a vision-based algorithm designed for non-contact detection of NNS activity using baby monitor footage in natural settings. Our approach involves a comprehensive exploration of optical flow and temporal convolutional networks, enabling the detection and amplification of subtle infant-sucking signals. We successfully classify short video clips of uniform length into NNS and non-NNS periods. Furthermore, we investigate manual and learning-based techniques to piece together local classification results, facilitating the segmentation of longer mixed-activity videos into NNS and non-NNS segments of varying duration. Our research introduces two novel datasets of annotated infant videos, including one sourced from our clinical study featuring 18 infant subjects and 183 h of overnight baby monitor footage. Additionally, we incorporate a second, shorter dataset obtained from publicly available YouTube videos. Our NNS action recognition algorithm achieves an impressive 95.8% accuracy in binary classification, based on 960 2.5-s balanced NNS versus non-NNS clips from our clinical dataset. We also present results for a subset of clips featuring challenging video conditions. Moreover, our NNS action segmentation algorithm achieves an average precision of 93.5% and an average recall of 92.9% across 30 heterogeneous 60-s clips from our clinical dataset.

Recently, although Transformer has widespread application in the field of computer vision, the quadratic complexity of its Self-Attention hindered the processing in large-scale image captioning task. Therefore, in this paper, we propose a Learnable Linear-Attention with Fast-Normalization for Large-Scale Image Captioning (dubbed as LLAFN-Generator). Firstly, it introduces a Learnable Linear-Attention (LLA) module to solve the weight score learning of large-scale images, which is simply implemented through two linear layers and greatly reduces the computation complexity. Meanwhile, the Fast-Normalization (FN) method is employed in the Learnable Linear-Attention instead of the original Softmax function to improve the computational speed. Additionally, the feature enhancement module be used to compensate for the shallow, fine-grained information in order to enhance the feature representation of the model. Finally, extensive experiments on the MS COCO dataset show that the computational complexity is reduced by 30% and the parameter is reduced by 20% on models of the same size, with the performance metrics BLEU_1 and CIDEr increasing by 1.2% and 3.6%, respectively.

In this paper, we tackle the problem of detailed hand-object 3D reconstruction from monocular video with unknown objects, for applications where the required accuracy and level of detail is important, e.g. object hand-over in human–robot collaboration, or manipulation and contact point analysis. While the recent literature on this topic is promising, the accuracy and generalization abilities of existing methods are still lacking. This is due to several limitations, such as the assumption of known object class or model for a small number of instances, or over-reliance on off-the-shelf keypoint and structure-from-motion methods for object-relative viewpoint estimation, prone to complete failure with previously unobserved, poorly textured objects or hand-object occlusions. To address previous method shortcomings, we present a 2-stage pipeline superseding state-of-the-art (SotA) performance on several metrics. First, we robustly retrieve viewpoints relying on a learned pairwise camera pose estimator trainable with a low data regime, followed by a globalized Shonan pose averaging. Second, we simultaneously estimate detailed 3D hand-object shapes and refine camera poses using a differential renderer-based optimizer. To better assess the out-of-distribution abilities of existing methods, and to showcase our methodological contributions, we introduce the new SHOWMe benchmark dataset with 96 sequences annotated with poses, millimetric textured 3D shape scans, and parametric hand models, introducing new object and hand diversity. Remarkably, we show that our method is able to reconstruct 100% of these sequences as opposed to SotA Structure-from-Motion (SfM) or hand-keypoint-based pipelines, and obtains reconstructions of equivalent or better precision when existing methods do succeed in providing a result. We hope these contributions lead to further research under harder input assumptions. The dataset can be downloaded at https://download.europe.naverlabs.com/showme.

In the era of AIGC, the fast development of visual content generation technologies, such as diffusion models, brings potential security risks to our society. Existing generated image detection methods suffer from performance drops when faced with out-of-domain generators and image scenes. To relieve this problem, we propose Artifact Purification Network (APN) to facilitate the artifact extraction from generated images through the explicit and implicit purification processes. For the explicit one, a suspicious frequency-band proposal method and a spatial feature decomposition method are proposed to extract artifact-related features. For the implicit one, a training strategy based on mutual information estimation is proposed to further purify the artifact-related features. The experiments are conducted in two settings. Firstly, we perform a cross-generator evaluation, wherein detectors trained using data from one generator are evaluated on data generated by other generators. Secondly, we conduct a cross-scene evaluation, wherein detectors trained for a specific domain of content (e.g., ImageNet) are assessed on data collected from another domain (e.g., LSUN-Bedroom). Results show that for cross-generator detection, the average accuracy of APN is $5.6\text{\%}\sim 16.4\text{\%}$ higher than the previous 11 methods on the GenImage dataset and $1.7\text{\%}\sim 50.1\text{\%}$ on the DiffusionForensics dataset. For cross-scene detection, APN maintains its high performance. Via visualization analysis, we find that the proposed method can extract diverse forgery patterns and condense the forgery information diluted in irrelated features. We also find that the artifact features APN focuses on across generators and scenes are global and diverse. The code will be available at https://github.com/RichardSunnyMeng/APN-official-codes.

In the rapidly evolving domain of image generation, the availability of sufficient data is crucial for effective model training. However, obtaining a large dataset is often challenging. Medical imaging, industrial monitoring, and self-driving cars are among the applications that require high-fidelity image generation from limited or single data points. The paper proposes a novel approach for increasing the diversity of images generated from a single input image by combining a Denoising Diffusion Probabilistic Model (DDPM) with the ConvNeXt-V2 architecture. This technique addresses the issue of limited data availability by utilizing single images using the BSD and Places365 datasets, significantly increasing the ability of the model through different conditions. The research greatly enhances the image quality by including Global Response Normalization (GRN) and Sigmoid-Weighted Linear Units (SiLU) in the DDPM. In-depth analyses and comparisons with the existing State-of-the-art (SOTA) models highlight the model’s effectiveness, which shows higher experimental results. Achievements include a Pixel Diversity score of 0.87±0.1, an LPIPS Diversity score of 0.42±0.03, and a SIFID for Patch Distribution of 0.046±0.02, along with notable NIQE and RECO scores. These findings indicate the exceptional ability of the model to generate a wide range of high-quality images, exhibiting significant advancement over existing State-of-the-art models in the field of image generation.

Skeleton-based action recognition is a key research area in video understanding, beneficial from its compact and efficient motion information. To relieve from the burden of expensive and laborious data annotation, unsupervised approaches, particularly contrastive learning, have been widely employed to extract action representations from unlabeled data. In this paper, we propose an Ensemble framework for Contrastive Learning of Representation (EnsCLR) to preform unsupervised skeleton-based action recognition. Concretely, Queue Extension method is devised to generate discriminative representation by aggregating the ensemble information from multiple pipelines. Furtherly, Ensemble Nearest Neighbors Mining (ENNM) method is utilized to excavate the most similar samples from the unlabeled data as positive samples, which alleviates the false-negative samples problem caused by the disregard of category label. The experiments with extensive evaluation protocols show that EnsCLR outperforms previous state-of-the-art methods on NTU60, NTU120, and PKU-MMD datasets.

We propose a very fast frame-level model for anomaly detection in video, which learns to detect anomalies by distilling knowledge from multiple highly accurate object-level teacher models. To improve the fidelity of our student, we distill the low-resolution anomaly maps of the teachers by jointly applying standard and adversarial distillation, introducing an adversarial discriminator for each teacher to distinguish between target and generated anomaly maps. We conduct experiments on three benchmarks (Avenue, ShanghaiTech, UCSD Ped2), showing that our method is over 7 times faster than the fastest competing method, and between 28 and 62 times faster than object-centric models, while obtaining comparable results to recent methods. Our evaluation also indicates that our model achieves the best trade-off between speed and accuracy, due to its previously unheard-of speed of 1480 FPS. In addition, we carry out a comprehensive ablation study to justify our architectural design choices. Our code is freely available at: https://github.com/ristea/fast-aed.

Low-light image enhancement algorithms play a crucial role in revealing details obscured by darkness in images and substantially improving overall image quality. However, existing methods often suffer from issues like color or lightness distortion and possess limited scalability. In response to these challenges, we introduce a novel low-light image enhancement algorithm leveraging a cell vibration energy model and lightness difference. Initially, a new low-light image enhancement framework is proposed, building upon a comprehensive understanding and analysis of the cell vibration energy model and its statistical properties. Subsequently, to achieve pixel-level multi-lightness difference adjustment and exert control over the lightness level of each pixel independently, a lightness difference adjustment strategy is introduced utilizing Weibull distribution and linear mapping. Furthermore, to expand the adaptive range of the algorithm, we consider the disparities between HSV space and RGB space. Two enhanced image output modes are designed, accompanied by a thorough analysis and deduction of the relevant image layer mapping formulas. Finally, to enhance the reliability of experimental results, certain image faults in the SICE database are rectified using the feature matching method. Experimental results showcase the superiority of the proposed algorithm over twelve state-of-the-art algorithms. The resource code of this article will be released at https://github.com/leixiaozhou/CDEGmethod.

The viral spread of massive deepfake videos over social networks has caused serious security problems. Despite the remarkable advancements achieved by existing deepfake detection algorithms, deepfake videos over social networks are inevitably influenced by compression factors. This causes deepfake detection performance to be limited by the following challenging issues: (a) interfering with compression artifacts, (b) loss of feature information, and (c) aliasing of feature distributions. In this paper, we analyze the common mechanism between compression artifacts and deepfake artifacts, revealing the structural similarity between them and providing a reliable theoretical basis for enhancing the robustness of deepfake detection models against compression. Firstly, based on the common mechanism between artifacts, we design a frequency domain adaptive notch filter to eliminate the interference of compression artifacts on specific frequency bands. Secondly, to reduce the sensitivity of deepfake detection models to unknown noise, we propose a spatial residual denoising strategy. Thirdly, to exploit the intrinsic correlation between feature vectors in the frequency domain branch and the spatial domain branch, we enhance deepfake features using an attention-based feature fusion method. Finally, we adopt a multi-task decision approach to enhance the discriminative power of the latent space representation of deepfakes, achieving deepfake detection with robustness against compression. Extensive experiments show that compared with the baseline methods, the detection performance of the proposed algorithm on compressed deepfake videos has been significantly improved. In particular, our model is resistant to various types of noise disturbances and can be easily combined with baseline detection models to improve their robustness.

Vision-Language Pretraining (VLP) and Foundation models have been the go-to recipe for achieving SoTA performance on general benchmarks. However, leveraging these powerful techniques for more complex vision-language tasks, such as cooking applications, with more structured input data, is still little investigated. In this work, we propose to leverage these techniques for structured-text based computational cuisine tasks. Our strategy, dubbed VLPCook, first transforms existing image-text pairs to image and structured-text pairs. This allows to pretrain our VLPCook model using VLP objectives adapted to the structured data of the resulting datasets, then finetuning it on downstream computational cooking tasks. During finetuning, we also enrich the visual encoder, leveraging pretrained foundation models (e.g. CLIP) to provide local and global textual context. VLPCook outperforms current SoTA by a significant margin (+3.3 Recall@1 absolute improvement) on the task of Cross-Modal Food Retrieval on the large Recipe1M dataset. We conduct further experiments on VLP to validate their importance, especially on the Recipe1M+ dataset. Finally, we validate the generalization of the approach to other tasks (i.e, Food Recognition) and domains with structured text such as the Medical domain on the ROCO dataset. The code will be made publicly available.