Journal of Visual Communication and Image Representation最新文献

IF 2.6 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2024-11-12 DOI: 10.1016/j.jvcir.2024.104340

Songhao Zhu, Yi Wang

Few-shot object detection method aims to learn novel classes through a small number of annotated novel class samples without having a catastrophic impact on previously learned knowledge, thereby expanding the trained model’s ability to detect novel classes. For existing few-shot object detection methods, there is a prominent false positive issue for the novel class samples due to the similarity in appearance features and feature distribution between the novel classes and the base classes. That is, the following two issues need to be solved: (1) How to detect these false positive samples in large-scale dataset, and (2) How to utilize the correlations between these false positive samples and other samples to improve the accuracy of the detection model. To address the first issue, an adaptive fusion data augmentation strategy is utilized to enhance the diversity of novel class samples and further alleviate the issue of false positive novel class samples. To address the second issue, a similarity transfer strategy is here proposed to effectively utilize the correlations between different categories. Experimental results demonstrate that the proposed method performs well in various settings of PASCAL VOC and MSCOCO datasets, achieving 48.7 and 11.3 on PASCAL VOC and MSCOCO under few-shot settings (shot = 1) in terms of nAP50 respectively.

少量对象检测方法旨在通过少量标注的新类别样本来学习新类别，而不会对先前学习的知识产生灾难性影响，从而扩大训练模型检测新类别的能力。对于现有的少量物体检测方法来说，由于新类别与基础类别在外观特征和特征分布上的相似性，新类别样本的假阳性问题非常突出。也就是说，需要解决以下两个问题：（1）如何在大规模数据集中检测出这些假阳性样本；（2）如何利用这些假阳性样本与其他样本之间的相关性来提高检测模型的准确性。针对第一个问题，我们采用了一种自适应融合数据增强策略，以增强新类别样本的多样性，进一步缓解新类别样本的假阳性问题。为解决第二个问题，本文提出了一种相似性转移策略，以有效利用不同类别之间的相关性。实验结果表明，所提出的方法在 PASCAL VOC 和 MSCOCO 数据集的各种设置下均表现良好，在 PASCAL VOC 和 MSCOCO 数据集的少镜头设置（镜头 = 1）下，nAP50 分别达到 48.7 和 11.3。

{"title":"Multi-level similarity transfer and adaptive fusion data augmentation for few-shot object detection","authors":"Songhao Zhu, Yi Wang","doi":"10.1016/j.jvcir.2024.104340","DOIUrl":"10.1016/j.jvcir.2024.104340","url":null,"abstract":"<div><div>Few-shot object detection method aims to learn novel classes through a small number of annotated novel class samples without having a catastrophic impact on previously learned knowledge, thereby expanding the trained model’s ability to detect novel classes. For existing few-shot object detection methods, there is a prominent false positive issue for the novel class samples due to the similarity in appearance features and feature distribution between the novel classes and the base classes. That is, the following two issues need to be solved: (1) How to detect these false positive samples in large-scale dataset, and (2) How to utilize the correlations between these false positive samples and other samples to improve the accuracy of the detection model. To address the first issue, an adaptive fusion data augmentation strategy is utilized to enhance the diversity of novel class samples and further alleviate the issue of false positive novel class samples. To address the second issue, a similarity transfer strategy is here proposed to effectively utilize the correlations between different categories. Experimental results demonstrate that the proposed method performs well in various settings of PASCAL VOC and MSCOCO datasets, achieving 48.7 and 11.3 on PASCAL VOC and MSCOCO under few-shot settings (shot = 1) in terms of nAP50 respectively.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"105 ","pages":"Article 104340"},"PeriodicalIF":2.6,"publicationDate":"2024-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142659528","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Color image watermarking using vector SNCM-HMT 使用矢量 SNCM-HMT 对彩色图像进行水印处理

IF 2.6 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2024-11-04 DOI: 10.1016/j.jvcir.2024.104339

Hongxin Wang, Runtong Ma, Panpan Niu

An image watermarking scheme is typically evaluated using three main conflicting characteristics: imperceptibility, robustness, and capacity. Developing a good image watermarking method is challenging because it requires a trade-off between these three basic characteristics. In this paper, we proposed a statistical color image watermarking based on robust discrete nonseparable Shearlet transform (DNST)-fast quaternion generic polar complex exponential transform (FQGPCET) magnitude and vector skew-normal-Cauchy mixtures (SNCM)-hidden Markov tree (HMT). The proposed watermarking system consists of two main parts: watermark inserting and watermark extraction. In watermark inserting, we first perform DNST on R, G, and B components of color host image, respectively. We then compute block FQGPCET of DNST domain color components, and embed watermark signal in DNST-FQGPCET magnitudes using multiplicative approach. In watermark extraction, we first analyze the robustness and statistical characteristics of local DNST-FQGPCET magnitudes of color image. We then observe that, vector SNCM-HMT model can capture accurately the marginal distribution and multiple strong dependencies of local DNST-FQGPCET magnitudes. Meanwhile, vector SNCM-HMT parameters can be computed effectively using variational expectation–maximization (VEM) parameter estimation. Motivated by our modeling results, we finally develop a new statistical color image watermark decoder based on vector SNCM-HMT and maximum likelihood (ML) decision rule. Experimental results on extensive test images demonstrate that the proposed statistical color image watermarking provides a performance better than that of most of the state-of-the-art statistical methods and some deep learning approaches recently proposed in the literature.

图像水印方案通常使用三个相互冲突的主要特征进行评估：不可感知性、鲁棒性和容量。开发一种好的图像水印方法具有挑战性，因为它需要在这三个基本特性之间进行权衡。在本文中，我们提出了一种基于鲁棒离散非可分剪切变换（DNST）-快速四元泛极性复指数变换（FQGPCET）幅度和矢量偏斜-正态-考奇混合物（SNCM）-隐藏马尔可夫树（HMT）的统计彩色图像水印。拟议的水印系统包括两个主要部分：水印插入和水印提取。在插入水印时，我们首先分别对彩色主图像的 R、G 和 B 分量执行 DNST。然后，我们计算 DNST 域彩色分量的块 FQGPCET，并使用乘法方法将水印信号嵌入 DNST-FQGPCET 幅值中。在提取水印时，我们首先分析了彩色图像局部 DNST-FQGPCET 幅值的鲁棒性和统计特征。结果表明，矢量 SNCM-HMT 模型能准确捕捉局部 DNST-FQGPCET 幅值的边际分布和多重强依赖关系。同时，矢量 SNCM-HMT 参数可通过变分期望最大化（VEM）参数估计法有效计算。在建模结果的激励下，我们最终开发出一种基于向量 SNCM-HMT 和最大似然 (ML) 决策规则的新型统计彩色图像水印解码器。在大量测试图像上的实验结果表明，所提出的统计彩色图像水印的性能优于大多数最先进的统计方法和最近在文献中提出的一些深度学习方法。

{"title":"Color image watermarking using vector SNCM-HMT","authors":"Hongxin Wang, Runtong Ma, Panpan Niu","doi":"10.1016/j.jvcir.2024.104339","DOIUrl":"10.1016/j.jvcir.2024.104339","url":null,"abstract":"<div><div>An image watermarking scheme is typically evaluated using three main conflicting characteristics: imperceptibility, robustness, and capacity. Developing a good image watermarking method is challenging because it requires a trade-off between these three basic characteristics. In this paper, we proposed a statistical color image watermarking based on robust discrete nonseparable Shearlet transform (DNST)-fast quaternion generic polar complex exponential transform (FQGPCET) magnitude and vector skew-normal-Cauchy mixtures (SNCM)-hidden Markov tree (HMT). The proposed watermarking system consists of two main parts: watermark inserting and watermark extraction. In watermark inserting, we first perform DNST on R, G, and B components of color host image, respectively. We then compute block FQGPCET of DNST domain color components, and embed watermark signal in DNST-FQGPCET magnitudes using multiplicative approach. In watermark extraction, we first analyze the robustness and statistical characteristics of local DNST-FQGPCET magnitudes of color image. We then observe that, vector SNCM-HMT model can capture accurately the marginal distribution and multiple strong dependencies of local DNST-FQGPCET magnitudes. Meanwhile, vector SNCM-HMT parameters can be computed effectively using variational expectation–maximization (VEM) parameter estimation. Motivated by our modeling results, we finally develop a new statistical color image watermark decoder based on vector SNCM-HMT and maximum likelihood (ML) decision rule. Experimental results on extensive test images demonstrate that the proposed statistical color image watermarking provides a performance better than that of most of the state-of-the-art statistical methods and some deep learning approaches recently proposed in the literature.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"105 ","pages":"Article 104339"},"PeriodicalIF":2.6,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142659380","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A memory access number constraint-based string prediction technique for high throughput SCC implemented in AVS3 在 AVS3 中实现基于内存访问数约束的字符串预测技术，以实现高吞吐量 SCC

IF 2.6 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2024-11-03 DOI: 10.1016/j.jvcir.2024.104338

Liping Zhao , Zuge Yan , Keli Hu , Sheng Feng , Jiangda Wang , Xueyan Cao , Tao Lin

String prediction (SP) is a highly efficient screen content coding (SCC) tool that has been adopted in international and Chinese video coding standards. SP exhibits a highly flexible and efficient ability to predict repetitive matching patterns. However, SP also suffers from low throughput of decoded display output pixels per memory access, which is synchronized with the decoder clock, due to the high number of memory accesses required to decode an SP coding unit for display. Even in state-of-the-art (SOTA) SP, the worst-case scenario involves two memory accesses for decoding each 4-pixel basic string unit across two memory access units, resulting in a throughput as low as two pixels per memory access (PPMA). To solve this problem, we are the first to propose a technique called memory access number constraint-based string prediction (MANC-SP) to achieve high throughput in SCC. First, a novel MANC-SP framework is proposed, a well-designed memory access number constraint rule is established on the basis of statistical data, and a constrained RDO-based string searching method is presented. Compared with the existing SOTA SP, the experimental results demonstrate that MANC-SP can improve the throughput from 2 to 2.67 PPMA, achieving a throughput improvement of 33.33% while maintaining a negligible impact on coding efficiency and complexity.

字符串预测（SP）是一种高效的屏幕内容编码（SCC）工具，已被国际和中国的视频编码标准所采用。SP 具有高度灵活和高效的预测重复匹配模式的能力。然而，SP 也存在每次内存访问（与解码器时钟同步）的解码显示输出像素吞吐量低的问题，这是因为对 SP 编码单元进行解码显示需要大量的内存访问。即使在最先进的（SOTA）SP 中，最糟糕的情况也是在两个存储器访问单元中对每个 4 像素基本字符串单元进行解码时需要两次存储器访问，导致每次存储器访问的吞吐量低至两个像素。为解决这一问题，我们首次提出了一种称为基于内存访问数约束的字符串预测（MANC-SP）的技术，以实现 SCC 的高吞吐量。首先，我们提出了一个新颖的 MANC-SP 框架，在统计数据的基础上建立了一个精心设计的内存访问数约束规则，并提出了一种基于 RDO 约束的字符串搜索方法。实验结果表明，与现有的 SOTA SP 相比，MANC-SP 可将吞吐量从 2 PPMA 提高到 2.67 PPMA，吞吐量提高了 33.33%，同时对编码效率和复杂度的影响几乎可以忽略不计。

{"title":"A memory access number constraint-based string prediction technique for high throughput SCC implemented in AVS3","authors":"Liping Zhao , Zuge Yan , Keli Hu , Sheng Feng , Jiangda Wang , Xueyan Cao , Tao Lin","doi":"10.1016/j.jvcir.2024.104338","DOIUrl":"10.1016/j.jvcir.2024.104338","url":null,"abstract":"<div><div>String prediction (SP) is a highly efficient screen content coding (SCC) tool that has been adopted in international and Chinese video coding standards. SP exhibits a highly flexible and efficient ability to predict repetitive matching patterns. However, SP also suffers from low throughput of decoded display output pixels per memory access, which is synchronized with the decoder clock, due to the high number of memory accesses required to decode an SP coding unit for display. Even in state-of-the-art (SOTA) SP, the worst-case scenario involves two memory accesses for decoding each 4-pixel basic string unit across two memory access units, resulting in a throughput as low as two pixels per memory access (PPMA). To solve this problem, we are the first to propose a technique called memory access number constraint-based string prediction (MANC-SP) to achieve high throughput in SCC. First, a novel MANC-SP framework is proposed, a well-designed memory access number constraint rule is established on the basis of statistical data, and a constrained RDO-based string searching method is presented. Compared with the existing SOTA SP, the experimental results demonstrate that MANC-SP can improve the throughput from 2 to 2.67 PPMA, achieving a throughput improvement of <strong>33.33%</strong> while maintaining a negligible impact on coding efficiency and complexity.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"105 ","pages":"Article 104338"},"PeriodicalIF":2.6,"publicationDate":"2024-11-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142659504","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Faster-slow network fused with enhanced fine-grained features for action recognition 快慢网络融合增强型细粒度特征进行动作识别

IF 2.6 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2024-10-30 DOI: 10.1016/j.jvcir.2024.104328

Xuegang Wu , Jiawei Zhu , Liu Yang

Two-stream methods, which separate human actions and backgrounds into temporal and spatial streams visually, have shown promising results in action recognition datasets. However, prior researches emphasize motion modeling but overlook the robust correlation between motion features and spatial information, causing restriction of the model’s ability to recognize behaviors entailing occlusions or rapid changes. Therefore, we introduce Faster-slow, an improved framework for frame-level motion features. It introduces a Behavioural Feature Enhancement (BFE) module based on a novel two-stream network with different temporal resolutions. BFE consists of two components: MM, which incorporates motion-aware attention to capture dependencies between adjacent frames; STC, which enhances spatio-temporal and channel information to generate optimized features. Overall, BFE facilitates the extraction of finer-grained motion information, while ensuring a stable fusion of information across both streams. We evaluate the Faster-slow on the Atomic Visual Actions dataset, and the Faster-AVA dataset constructed in this paper, yielding promising experimental results.

双流法将人的动作和背景以视觉方式分为时间流和空间流，这种方法在动作识别数据集中显示出良好的效果。然而，之前的研究强调运动建模，却忽视了运动特征与空间信息之间的强相关性，从而限制了模型识别包含遮挡或快速变化的行为的能力。因此，我们引入了帧级运动特征改进框架 Faster-slow。它基于具有不同时间分辨率的新型双流网络，引入了行为特征增强（BFE）模块。BFE 由两个部分组成：MM：结合运动感知注意力，捕捉相邻帧之间的依赖关系；STC：增强时空信息和信道信息，生成优化特征。总之，BFE 可帮助提取更精细的运动信息，同时确保两个数据流信息的稳定融合。我们在 Atomic Visual Actions 数据集和本文构建的 Faster-AVA 数据集上对 Faster-slow 进行了评估，取得了令人满意的实验结果。

{"title":"Faster-slow network fused with enhanced fine-grained features for action recognition","authors":"Xuegang Wu , Jiawei Zhu , Liu Yang","doi":"10.1016/j.jvcir.2024.104328","DOIUrl":"10.1016/j.jvcir.2024.104328","url":null,"abstract":"<div><div>Two-stream methods, which separate human actions and backgrounds into temporal and spatial streams visually, have shown promising results in action recognition datasets. However, prior researches emphasize motion modeling but overlook the robust correlation between motion features and spatial information, causing restriction of the model’s ability to recognize behaviors entailing occlusions or rapid changes. Therefore, we introduce Faster-slow, an improved framework for frame-level motion features. It introduces a Behavioural Feature Enhancement (BFE) module based on a novel two-stream network with different temporal resolutions. BFE consists of two components: MM, which incorporates motion-aware attention to capture dependencies between adjacent frames; STC, which enhances spatio-temporal and channel information to generate optimized features. Overall, BFE facilitates the extraction of finer-grained motion information, while ensuring a stable fusion of information across both streams. We evaluate the Faster-slow on the Atomic Visual Actions dataset, and the Faster-AVA dataset constructed in this paper, yielding promising experimental results.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"105 ","pages":"Article 104328"},"PeriodicalIF":2.6,"publicationDate":"2024-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142659379","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Lightweight macro-pixel quality enhancement network for light field images compressed by versatile video coding 通过多功能视频编码压缩光场图像的轻量级宏像素质量增强网络

IF 2.6 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2024-10-30 DOI: 10.1016/j.jvcir.2024.104329

Hongyue Huang , Chen Cui , Chuanmin Jia , Xinfeng Zhang , Siwei Ma

Previous research demonstrated that filtering Macro-Pixels (MPs) in a decoded Light Field Image (LFI) sequence can effectively enhances the quality of the corresponding Sub-Aperture Images (SAIs). In this paper, we propose a deep-learning-based quality enhancement model following the MP-wise processing approach tailored to LFIs encoded by the Versatile Video Coding (VVC) standard. The proposed novel Res2Net Quality Enhancement Convolutional Neural Network (R2NQE-CNN) architecture is both lightweight and powerful, in which the Res2Net modules are employed to perform LFI filtering for the first time, and are implemented with a novel improved 3D-feature-processing structure. The proposed method incorporates only 205K model parameters and achieves significant Y-BD-rate reductions over VVC of up to 32%, representing a relative improvement of up to 33% compared to the state-of-the-art method, which has more than three times the number of parameters of our proposed model.

以往的研究表明，对解码光场图像（LFI）序列中的宏像素（MP）进行过滤，可有效提高相应子孔径图像（SAI）的质量。在本文中，我们针对多功能视频编码（VVC）标准编码的光场图像，提出了一种基于深度学习的质量增强模型，该模型采用了MP-wise处理方法。所提出的新型 Res2Net 质量增强卷积神经网络（R2NQE-CNN）架构既轻便又强大，其中首次采用了 Res2Net 模块来执行 LFI 过滤，并通过新型改进 3D 特征处理结构来实现。所提出的方法仅包含 205K 个模型参数，与 VVC 相比，Y-BD 速率显著降低了 32%，与最先进方法相比，相对改进高达 33%，而最先进方法的参数数量是我们所提出模型的三倍多。

引用次数: 0

TrMLGAN: Transmission MultiLoss Generative Adversarial Network framework for image dehazing TrMLGAN：用于图像去毛刺的传输多损失生成对抗网络框架

IF 2.6 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2024-10-28 DOI: 10.1016/j.jvcir.2024.104324

Pulkit Dwivedi, Soumendu Chakraborty

Hazy environments significantly degrade image quality, leading to poor contrast and reduced visibility. Existing dehazing methods often struggle to predict the transmission map, which is crucial for accurate dehazing. This study introduces the Transmission MultiLoss Generative Adversarial Network (TrMLGAN), a novel framework designed to enhance transmission map estimation for improved dehazing. The transmission map is initially computed using a dark channel prior-based approach and refined using the TrMLGAN framework, which leverages Generative Adversarial Networks (GANs). By integrating multiple loss functions, such as adversarial, pixel-wise similarity, perceptual similarity, and SSIM losses, our method focuses on various aspects of image quality. This enables robust dehazing performance without direct dependence on ground-truth images. Evaluations using PSNR, SSIM, FADE, NIQE, BRISQUE, and SSEQ metrics show that TrMLGAN significantly outperforms state-of-the-art methods across datasets including D-HAZY, HSTS, SOTS Outdoor, NH-HAZE, and D-Hazy, validating its potential for real-world applications.

雾霾环境会大大降低图像质量，导致对比度差和能见度降低。现有的去噪方法往往难以预测透射图，而透射图对于准确去噪至关重要。本研究介绍了传输多损失生成对抗网络（TrMLGAN），这是一个新颖的框架，旨在增强传输图估算以改善去噪效果。传输图最初采用基于暗信道先验的方法计算，然后利用生成式对抗网络（GAN）的 TrMLGAN 框架进行改进。通过整合多种损失函数，如对抗损失、像素相似性损失、感知相似性损失和 SSIM 损失，我们的方法侧重于图像质量的各个方面。这样就能在不直接依赖地面实况图像的情况下实现稳健的去毛刺性能。使用 PSNR、SSIM、FADE、NIQE、BRISQUE 和 SSEQ 等指标进行的评估表明，在包括 D-HAZY、HSTS、SOTS Outdoor、NH-HAZE 和 D-Hazy 等数据集上，TrMLGAN 明显优于最先进的方法，验证了其在实际应用中的潜力。

{"title":"TrMLGAN: Transmission MultiLoss Generative Adversarial Network framework for image dehazing","authors":"Pulkit Dwivedi, Soumendu Chakraborty","doi":"10.1016/j.jvcir.2024.104324","DOIUrl":"10.1016/j.jvcir.2024.104324","url":null,"abstract":"<div><div>Hazy environments significantly degrade image quality, leading to poor contrast and reduced visibility. Existing dehazing methods often struggle to predict the transmission map, which is crucial for accurate dehazing. This study introduces the Transmission MultiLoss Generative Adversarial Network (TrMLGAN), a novel framework designed to enhance transmission map estimation for improved dehazing. The transmission map is initially computed using a dark channel prior-based approach and refined using the TrMLGAN framework, which leverages Generative Adversarial Networks (GANs). By integrating multiple loss functions, such as adversarial, pixel-wise similarity, perceptual similarity, and SSIM losses, our method focuses on various aspects of image quality. This enables robust dehazing performance without direct dependence on ground-truth images. Evaluations using PSNR, SSIM, FADE, NIQE, BRISQUE, and SSEQ metrics show that TrMLGAN significantly outperforms state-of-the-art methods across datasets including D-HAZY, HSTS, SOTS Outdoor, NH-HAZE, and D-Hazy, validating its potential for real-world applications.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"105 ","pages":"Article 104324"},"PeriodicalIF":2.6,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142552678","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Video Question Answering: A survey of the state-of-the-art 视频问题解答：最新技术调查

IF 2.6 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2024-10-28 DOI: 10.1016/j.jvcir.2024.104320

Jeshmol P.J., Binsu C. Kovoor

Video Question Answering (VideoQA) emerges as a prominent trend in the domain of Artificial Intelligence, Computer Vision, and Natural Language Processing. It involves developing systems capable of understanding, analyzing, and responding to questions about the content of videos. The Proposed survey presents an in-depth overview of the current landscape of Question Answering, shedding light on the challenges, methodologies, datasets, and innovative approaches in the domain. The key components of the Video Question Answering (VideoQA) framework include video feature extraction, question processing, reasoning, and response generation. It underscores the importance of datasets in shaping VideoQA research and the diversity of question types, from factual inquiries to spatial and temporal reasoning. The survey highlights the ongoing research directions and future prospects for VideoQA. Finally, the proposed survey gives a road map for future explorations at the intersection of multiple disciplines, emphasizing the ultimate objective of pushing the boundaries of knowledge and innovation.

视频问题解答（VideoQA）是人工智能、计算机视觉和自然语言处理领域的一个突出趋势。它涉及开发能够理解、分析和回答有关视频内容问题的系统。本调查报告深入概述了问题解答的现状，揭示了该领域的挑战、方法、数据集和创新方法。视频问题解答（VideoQA）框架的关键组成部分包括视频特征提取、问题处理、推理和响应生成。它强调了数据集在视频问题解答研究中的重要性，以及问题类型的多样性，从事实查询到空间和时间推理。调查报告强调了视频质量保证正在进行的研究方向和未来前景。最后，建议的调查为未来在多学科交叉领域的探索提供了路线图，强调了推动知识和创新边界的最终目标。

引用次数: 0

Consistent prototype contrastive learning for weakly supervised person search 针对弱监督人员搜索的一致原型对比学习

IF 2.6 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2024-10-28 DOI: 10.1016/j.jvcir.2024.104321

Huadong Lin , Xiaohan Yu , Pengcheng Zhang , Xiao Bai , Jin Zheng

Weakly supervised person search simultaneously addresses detection and re-identification tasks without relying on person identity labels. Prototype-based contrastive learning is commonly used to address unsupervised person re-identification. We argue that prototypes suffer from spatial, temporal, and label inconsistencies, which result in their inaccurate representation. In this paper, we propose a novel Consistent Prototype Contrastive Learning (CPCL) framework to address prototype inconsistency. For spatial inconsistency, a greedy update strategy is developed to introduce ground truth proposals in the training process and update the memory bank only with the ground truth features. To improve temporal consistency, CPCL employs a local window strategy to calculate the prototype within a specific temporal domain window. To tackle label inconsistency, CPCL adopts a prototype nearest neighbor consistency method that leverages the intrinsic information of the prototypes to rectify the pseudo-labels. Experimentally, the proposed method exhibits remarkable performance improvements on both the CUHK-SYSU and PRW datasets, achieving an mAP of 90.2% and 29.3% respectively. Moreover, it achieves state-of-the-art performance on the CUHK-SYSU dataset. The code will be available on the project website: https://github.com/JackFlying/cpcl.

弱监督式人物搜索可同时处理检测和重新识别任务，而无需依赖人物身份标签。基于原型的对比学习通常用于解决无监督的人物再识别问题。我们认为，原型存在空间、时间和标签不一致的问题，这导致了原型表征的不准确。在本文中，我们提出了一个新颖的一致原型对比学习（CPCL）框架来解决原型不一致的问题。针对空间不一致性，我们开发了一种贪婪更新策略，在训练过程中引入地面实况建议，并仅使用地面实况特征更新记忆库。为提高时间一致性，CPCL 采用了局部窗口策略，在特定时域窗口内计算原型。为解决标签不一致问题，CPCL 采用了原型近邻一致性方法，利用原型的内在信息来纠正伪标签。实验表明，所提出的方法在 CUHK-SYSU 和 PRW 数据集上都有显著的性能改进，mAP 分别达到 90.2% 和 29.3%。此外，该方法在 CUHK-SYSU 数据集上也达到了最先进的性能。代码可在项目网站 https://github.com/JackFlying/cpcl 上获取。

{"title":"Consistent prototype contrastive learning for weakly supervised person search","authors":"Huadong Lin , Xiaohan Yu , Pengcheng Zhang , Xiao Bai , Jin Zheng","doi":"10.1016/j.jvcir.2024.104321","DOIUrl":"10.1016/j.jvcir.2024.104321","url":null,"abstract":"<div><div>Weakly supervised person search simultaneously addresses detection and re-identification tasks without relying on person identity labels. Prototype-based contrastive learning is commonly used to address unsupervised person re-identification. We argue that prototypes suffer from spatial, temporal, and label inconsistencies, which result in their inaccurate representation. In this paper, we propose a novel Consistent Prototype Contrastive Learning (CPCL) framework to address prototype inconsistency. For spatial inconsistency, a greedy update strategy is developed to introduce ground truth proposals in the training process and update the memory bank only with the ground truth features. To improve temporal consistency, CPCL employs a local window strategy to calculate the prototype within a specific temporal domain window. To tackle label inconsistency, CPCL adopts a prototype nearest neighbor consistency method that leverages the intrinsic information of the prototypes to rectify the pseudo-labels. Experimentally, the proposed method exhibits remarkable performance improvements on both the CUHK-SYSU and PRW datasets, achieving an mAP of 90.2% and 29.3% respectively. Moreover, it achieves state-of-the-art performance on the CUHK-SYSU dataset. The code will be available on the project website: <span><span>https://github.com/JackFlying/cpcl</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"105 ","pages":"Article 104321"},"PeriodicalIF":2.6,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142586124","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

MT-Net: Single image dehazing based on meta learning, knowledge transfer and contrastive learning MT-Net：基于元学习、知识迁移和对比学习的单幅图像去毛刺技术

IF 2.6 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2024-10-28 DOI: 10.1016/j.jvcir.2024.104325

Jianlei Liu, Bingqing Yang, Shilong Wang, Maoli Wang

Single image dehazing is becoming increasingly important as its results impact the efficiency of subsequent computer vision tasks. While many methods have been proposed to address this challenge, existing dehazing approaches often exhibit limited adaptability to different types of images and lack future learnability. In light of this, we propose a dehazing network based on meta-learning, knowledge transfer, and contrastive learning, abbreviated as MT-Net. In our approach, we combine knowledge transfer with meta-learning to tackle these challenges, thus enhancing the network’s generalization performance. We refine the structure of knowledge transfer by introducing a two-phases approach to facilitate learning under the guidance of teacher networks and learning committee networks. We also optimize the negative examples of contrastive learning to reduce the contrast space. Extensive experiments conducted on synthetic and real datasets demonstrate the remarkable performance of our method in both quantitative and qualitative comparisons. The code has been released on https://github.com/71717171fan/MT-Net.

单幅图像去毛刺变得越来越重要，因为其结果会影响后续计算机视觉任务的效率。虽然已经提出了许多方法来应对这一挑战，但现有的去毛刺方法往往对不同类型图像的适应性有限，而且缺乏未来可学习性。有鉴于此，我们提出了一种基于元学习、知识转移和对比学习的去毛刺网络，简称 MT-Net。在我们的方法中，我们将知识转移与元学习相结合来应对这些挑战，从而提高网络的泛化性能。我们完善了知识转移的结构，引入了两阶段方法，以促进在教师网络和学习委员会网络指导下的学习。我们还优化了对比学习的负面示例，以缩小对比空间。在合成数据集和真实数据集上进行的大量实验表明，我们的方法在定量和定性比较方面都表现出色。代码已发布在 https://github.com/71717171fan/MT-Net 上。

{"title":"MT-Net: Single image dehazing based on meta learning, knowledge transfer and contrastive learning","authors":"Jianlei Liu, Bingqing Yang, Shilong Wang, Maoli Wang","doi":"10.1016/j.jvcir.2024.104325","DOIUrl":"10.1016/j.jvcir.2024.104325","url":null,"abstract":"<div><div>Single image dehazing is becoming increasingly important as its results impact the efficiency of subsequent computer vision tasks. While many methods have been proposed to address this challenge, existing dehazing approaches often exhibit limited adaptability to different types of images and lack future learnability. In light of this, we propose a dehazing network based on meta-learning, knowledge transfer, and contrastive learning, abbreviated as MT-Net. In our approach, we combine knowledge transfer with meta-learning to tackle these challenges, thus enhancing the network’s generalization performance. We refine the structure of knowledge transfer by introducing a two-phases approach to facilitate learning under the guidance of teacher networks and learning committee networks. We also optimize the negative examples of contrastive learning to reduce the contrast space. Extensive experiments conducted on synthetic and real datasets demonstrate the remarkable performance of our method in both quantitative and qualitative comparisons. The code has been released on <span><span>https://github.com/71717171fan/MT-Net</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"105 ","pages":"Article 104325"},"PeriodicalIF":2.6,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142572543","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Human gait recognition using joint spatiotemporal modulation in deep convolutional neural networks 利用深度卷积神经网络的时空联合调制识别人类步态

IF 2.6 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2024-10-28 DOI: 10.1016/j.jvcir.2024.104322

Mohammad Iman Junaid , Allam Jaya Prakash , Samit Ari

Gait, a person’s distinctive walking pattern, offers a promising biometric modality for surveillance applications. Unlike fingerprints or iris scans, gait can be captured from a distance without the subject’s direct cooperation or awareness. This makes it ideal for surveillance and security applications. Traditional convolutional neural networks (CNNs) often struggle with the inherent variations within video data, limiting their effectiveness in gait recognition. The proposed technique in this work introduces a unique joint spatial–temporal modulation network designed to overcome this limitation. By extracting discriminative feature representations across varying frame levels, the network effectively leverages both spatial and temporal variations within video sequences. The proposed architecture integrates attention-based CNNs for spatial feature extraction and a Bidirectional Long Short-Term Memory (Bi-LSTM) network with a temporal attention module to analyse temporal dynamics. The use of attention in spatial and temporal blocks enhances the network’s capability of focusing on the most relevant segments of the video data. This can improve efficiency since the combined approach enhances learning capabilities when processing complex gait videos. We evaluated the effectiveness of the proposed network using two major datasets, namely CASIA-B and OUMVLP. Experimental analysis on CASIA B demonstrates that the proposed network achieves an average rank-1 accuracy of 98.20% for normal walking, 94.50% for walking with a bag and 80.40% for clothing scenarios. The proposed network also achieved an accuracy of 89.10% for OU-MVLP. These results show the proposed method‘s ability to generalize to large-scale data and consistently outperform current state-of-the-art gait recognition techniques.

步态是一个人独特的行走模式，它为监控应用提供了一种前景广阔的生物识别模式。与指纹或虹膜扫描不同，步态可以在远距离捕捉，而无需当事人直接配合或意识到。这使其成为监控和安全应用的理想选择。传统的卷积神经网络（CNN）往往难以应对视频数据中固有的变化，从而限制了其在步态识别中的有效性。这项工作中提出的技术引入了独特的时空联合调制网络，旨在克服这一限制。通过提取不同帧级的判别特征表征，该网络可有效利用视频序列中的空间和时间变化。所提出的架构整合了基于注意力的 CNN（用于空间特征提取）和双向长短时记忆（Bi-LSTM）网络，后者带有一个时间注意力模块，用于分析时间动态。在空间和时间块中使用注意力可增强网络关注视频数据中最相关片段的能力。这可以提高效率，因为在处理复杂步态视频时，这种组合方法增强了学习能力。我们使用 CASIA-B 和 OUMVLP 这两个主要数据集评估了拟议网络的有效性。CASIA B 数据集的实验分析表明，所提出的网络在正常行走、背包行走和穿衣场景中的平均秩-1 准确率分别为 98.20%、94.50% 和 80.40%。在 OU-MVLP 中，所提出的网络也达到了 89.10% 的准确率。这些结果表明，所提出的方法能够通用于大规模数据，并持续优于当前最先进的步态识别技术。

{"title":"Human gait recognition using joint spatiotemporal modulation in deep convolutional neural networks","authors":"Mohammad Iman Junaid , Allam Jaya Prakash , Samit Ari","doi":"10.1016/j.jvcir.2024.104322","DOIUrl":"10.1016/j.jvcir.2024.104322","url":null,"abstract":"<div><div>Gait, a person’s distinctive walking pattern, offers a promising biometric modality for surveillance applications. Unlike fingerprints or iris scans, gait can be captured from a distance without the subject’s direct cooperation or awareness. This makes it ideal for surveillance and security applications. Traditional convolutional neural networks (CNNs) often struggle with the inherent variations within video data, limiting their effectiveness in gait recognition. The proposed technique in this work introduces a unique joint spatial–temporal modulation network designed to overcome this limitation. By extracting discriminative feature representations across varying frame levels, the network effectively leverages both spatial and temporal variations within video sequences. The proposed architecture integrates attention-based CNNs for spatial feature extraction and a Bidirectional Long Short-Term Memory (Bi-LSTM) network with a temporal attention module to analyse temporal dynamics. The use of attention in spatial and temporal blocks enhances the network’s capability of focusing on the most relevant segments of the video data. This can improve efficiency since the combined approach enhances learning capabilities when processing complex gait videos. We evaluated the effectiveness of the proposed network using two major datasets, namely CASIA-B and OUMVLP. Experimental analysis on CASIA B demonstrates that the proposed network achieves an average rank-1 accuracy of 98.20% for normal walking, 94.50% for walking with a bag and 80.40% for clothing scenarios. The proposed network also achieved an accuracy of 89.10% for OU-MVLP. These results show the proposed method‘s ability to generalize to large-scale data and consistently outperform current state-of-the-art gait recognition techniques.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"105 ","pages":"Article 104322"},"PeriodicalIF":2.6,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142572372","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0