Journal of Visual Communication and Image Representation最新文献_第10页

VLMAR: Maritime scene anomaly detection via retrieval-augmented vision-language models VLMAR：基于检索增强视觉语言模型的海事场景异常检测

IF 3.1 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2026-01-01 Epub Date: 2025-12-04 DOI: 10.1016/j.jvcir.2025.104669

Shen Wang , Chunsheng Yang , Chengtao Cai

Maritime anomaly detection is crucial for ensuring navigational safety and marine security. However, the global navigation safety is significantly challenged by the lack of comprehensive understanding of abnormal maritime ship behaviors. Drawing inspiration from the advanced reasoning capabilities of large language models, we introduce VLMAR, a novel vision-language framework that synergizes retrieval-augmented knowledge grounding and chain-of-thought reasoning to address these challenges. Our approach consists of two key innovations: (1) The VLMAR dataset is a large-scale multimodal repository containing 80,000 automatic identification system records, 11,500 synthetic aperture radar images, 5750 AIS text reports, and 27,000 behavioral narratives; (2) The VLMAR model architecture links real-time sensor data with maritime knowledge through dynamic retrieval and uses chain-of-thought fusion to interpret complex behaviors. Experimental results show that VLMAR achieves 94.77% Rank-1 accuracy in AIS retrieval and 89.10% accuracy in anomaly detection, significantly outperforming existing VLMs. Beyond performance, VLMAR reveals that aligning spatiotemporal AIS data with SAR imagery enables interpretable detection of hidden anomalies such as AIS spoofing and unauthorized route deviations, offering reliable explanations for safety-critical maritime decisions. This research establishes a new benchmark for maritime artificial intelligence systems, demonstrating how hybrid retrieval-generation paradigms can enhance situational awareness and support human-aligned decision-making.

海上异常检测是保障航行安全和海洋治安的重要手段。然而，由于缺乏对海上船舶异常行为的全面认识，全球航行安全面临重大挑战。从大型语言模型的高级推理能力中获得灵感，我们引入了VLMAR，这是一种新的视觉语言框架，它将检索增强知识基础和思维链推理协同起来，以解决这些挑战。我们的方法包括两个关键创新：(1)VLMAR数据集是一个大型多模式存储库，包含80,000条自动识别系统记录，11,500张合成孔径雷达图像，5750份AIS文本报告和27,000个行为叙述；(2) VLMAR模型架构通过动态检索将实时传感器数据与海事知识联系起来，并利用思维链融合来解释复杂行为。实验结果表明，VLMAR在AIS检索中的Rank-1准确率为94.77%，在异常检测中的准确率为89.10%，显著优于现有的VLMAR。除了性能之外，VLMAR还表明，将AIS时空数据与SAR图像相匹配，可以对AIS欺骗和未经授权的路线偏差等隐藏异常进行可解释的检测，为安全关键的海上决策提供可靠的解释。本研究为海事人工智能系统建立了一个新的基准，展示了混合检索生成范式如何增强态势感知并支持与人类一致的决策。

{"title":"VLMAR: Maritime scene anomaly detection via retrieval-augmented vision-language models","authors":"Shen Wang , Chunsheng Yang , Chengtao Cai","doi":"10.1016/j.jvcir.2025.104669","DOIUrl":"10.1016/j.jvcir.2025.104669","url":null,"abstract":"<div><div>Maritime anomaly detection is crucial for ensuring navigational safety and marine security. However, the global navigation safety is significantly challenged by the lack of comprehensive understanding of abnormal maritime ship behaviors. Drawing inspiration from the advanced reasoning capabilities of large language models, we introduce VLMAR, a novel vision-language framework that synergizes retrieval-augmented knowledge grounding and chain-of-thought reasoning to address these challenges. Our approach consists of two key innovations: (1) The VLMAR dataset is a large-scale multimodal repository containing 80,000 automatic identification system records, 11,500 synthetic aperture radar images, 5750 AIS text reports, and 27,000 behavioral narratives; (2) The VLMAR model architecture links real-time sensor data with maritime knowledge through dynamic retrieval and uses chain-of-thought fusion to interpret complex behaviors. Experimental results show that VLMAR achieves 94.77% Rank-1 accuracy in AIS retrieval and 89.10% accuracy in anomaly detection, significantly outperforming existing VLMs. Beyond performance, VLMAR reveals that aligning spatiotemporal AIS data with SAR imagery enables interpretable detection of hidden anomalies such as AIS spoofing and unauthorized route deviations, offering reliable explanations for safety-critical maritime decisions. This research establishes a new benchmark for maritime artificial intelligence systems, demonstrating how hybrid retrieval-generation paradigms can enhance situational awareness and support human-aligned decision-making.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"115 ","pages":"Article 104669"},"PeriodicalIF":3.1,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145694373","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A multi-modal 3D object detection framework based on enhanced Convolution, mixed Sampling, and Image-Point cloud bidirectional fusion 一种基于增强卷积、混合采样和图像点云双向融合的多模态三维目标检测框架

IF 3.1 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2025-12-01 Epub Date: 2025-10-28 DOI: 10.1016/j.jvcir.2025.104624

Xin Zhou , Xiaolong Xu

As deep learning and computer vision technologies advance, the demand for 3D object detection in autonomous driving is increasing. In this paper, we address the limitations of single-modal approaches by proposing a method that fuses LiDAR point clouds and RGB images. First, we provide an overview and discussion of existing multi-modal fusion methods in the literature. Second, we designed the enhanced convolutional module (SEC-Block) based on the channel attention mechanism to effectively capture and represent the key features in images. Then, we employed the Mixed Sampling strategy (M-FPS) to address the challenges in point cloud sampling. We also design the Att-Fusion module to handle the fusion of point clouds and images. The Att-Fusion module adaptively estimates important features of images and point clouds, fully exploiting their complementarity for efficient fusion. We integrate SEC-Block, M−FPS, and Att-Fusion into a multi-modal 3D object detection model named PAINet. Experimental results demonstrate that PAINet achieves a 3D detection accuracy of 82.59% for moderate-level cars on the KITTI test set, outperforming other state-of-the-art models and providing an effective solution for environmental perception in autonomous driving systems.

随着深度学习和计算机视觉技术的发展，自动驾驶对三维目标检测的需求日益增加。在本文中，我们通过提出一种融合激光雷达点云和RGB图像的方法来解决单模态方法的局限性。首先，我们对文献中现有的多模态融合方法进行了概述和讨论。其次，我们设计了基于通道注意机制的增强卷积模块（SEC-Block），以有效地捕获和表示图像中的关键特征。然后，我们采用混合采样策略（M-FPS）来解决点云采样中的挑战。我们还设计了at - fusion模块来处理点云和图像的融合。Att-Fusion模块自适应估计图像和点云的重要特征，充分利用它们的互补性进行高效融合。我们将SEC-Block、M - FPS和at - fusion集成到一个名为PAINet的多模态3D物体检测模型中。实验结果表明，在KITTI测试集上，PAINet对中等级别汽车的3D检测准确率达到82.59%，优于其他最先进的模型，为自动驾驶系统的环境感知提供了有效的解决方案。

{"title":"A multi-modal 3D object detection framework based on enhanced Convolution, mixed Sampling, and Image-Point cloud bidirectional fusion","authors":"Xin Zhou , Xiaolong Xu","doi":"10.1016/j.jvcir.2025.104624","DOIUrl":"10.1016/j.jvcir.2025.104624","url":null,"abstract":"<div><div>As deep learning and computer vision technologies advance, the demand for 3D object detection in autonomous driving is increasing. In this paper, we address the limitations of single-modal approaches by proposing a method that fuses LiDAR point clouds and RGB images. First, we provide an overview and discussion of existing multi-modal fusion methods in the literature. Second, we designed the enhanced convolutional module (SEC-Block) based on the channel attention mechanism to effectively capture and represent the key features in images. Then, we employed the Mixed Sampling strategy (M-FPS) to address the challenges in point cloud sampling. We also design the Att-Fusion module to handle the fusion of point clouds and images. The Att-Fusion module adaptively estimates important features of images and point clouds, fully exploiting their complementarity for efficient fusion. We integrate SEC-Block, M−FPS, and Att-Fusion into a multi-modal 3D object detection model named PAINet. Experimental results demonstrate that PAINet achieves a 3D detection accuracy of 82.59% for moderate-level cars on the KITTI test set, outperforming other state-of-the-art models and providing an effective solution for environmental perception in autonomous driving systems.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"113 ","pages":"Article 104624"},"PeriodicalIF":3.1,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145418409","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

SemMatcher: Semantic-aware feature matching with neighborhood consensus SemMatcher：基于邻域一致性的语义感知特征匹配

IF 3.1 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2025-12-01 Epub Date: 2025-10-17 DOI: 10.1016/j.jvcir.2025.104611

Qimin Jiang , Xiaoyong Lu , Dong Liang , Songlin Du

Local feature matching is the core of many computer vision tasks. Current methods only consider the extracted points individually, disregarding the connections between keypoints and the scene information, making feature matching challenging in scenarios with rich changes in viewpoint and illumination. To address this problem, this paper proposes SemMatcher, a novel semantic-aware feature matching framework which combines scene information overlooked by keypoints. Specifically, co-visual regions are filtered out through semantic segmentation for more focused learning of subsequent attention mechanisms, which refer to obtaining regions of the same category in two images. In SemMatcher, we design a semantic-aware attention mechanism, which pays more attention to co-visual regions unlike conventional global learning, achieving a win–win situation in terms of efficiency and performance. Besides, to build connections between keypoints, we introduce a semantic-aware neighborhood consensus which incorporates neighborhood consensus into attentional aggregation and constructs contextualized neighborhood information. Extensive experiments on homography estimation, pose estimation and image matching demonstrate that the model is superior to other methods and yields outstanding performance improvements.

局部特征匹配是许多计算机视觉任务的核心。目前的方法只是单独考虑提取的点，忽略了关键点与场景信息之间的联系，使得在视点和光照变化丰富的场景下，特征匹配变得困难。为了解决这一问题，本文提出了一种新的语义感知特征匹配框架SemMatcher，该框架结合了被关键点忽略的场景信息。具体而言，通过语义分割过滤掉共视觉区域，以便对后续注意机制进行更集中的学习，即在两幅图像中获得相同类别的区域。在SemMatcher中，我们设计了一种语义感知的注意机制，与传统的全局学习不同，该机制更多地关注共同视觉区域，实现了效率和性能的双赢。此外，为了建立关键点之间的联系，我们引入了一种语义感知的邻域共识，该共识将邻域共识融入到注意聚合中，并构建了语境化的邻域信息。在单应性估计、姿态估计和图像匹配方面的大量实验表明，该模型优于其他方法，并取得了显著的性能改进。

{"title":"SemMatcher: Semantic-aware feature matching with neighborhood consensus","authors":"Qimin Jiang , Xiaoyong Lu , Dong Liang , Songlin Du","doi":"10.1016/j.jvcir.2025.104611","DOIUrl":"10.1016/j.jvcir.2025.104611","url":null,"abstract":"<div><div>Local feature matching is the core of many computer vision tasks. Current methods only consider the extracted points individually, disregarding the connections between keypoints and the scene information, making feature matching challenging in scenarios with rich changes in viewpoint and illumination. To address this problem, this paper proposes SemMatcher, a novel semantic-aware feature matching framework which combines scene information overlooked by keypoints. Specifically, co-visual regions are filtered out through semantic segmentation for more focused learning of subsequent attention mechanisms, which refer to obtaining regions of the same category in two images. In SemMatcher, we design a semantic-aware attention mechanism, which pays more attention to co-visual regions unlike conventional global learning, achieving a win–win situation in terms of efficiency and performance. Besides, to build connections between keypoints, we introduce a semantic-aware neighborhood consensus which incorporates neighborhood consensus into attentional aggregation and constructs contextualized neighborhood information. Extensive experiments on homography estimation, pose estimation and image matching demonstrate that the model is superior to other methods and yields outstanding performance improvements.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"113 ","pages":"Article 104611"},"PeriodicalIF":3.1,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145365337","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Dehaze-cGAN: Image dehazing using a multi-head attention-based conditional GAN for traffic video monitoring Dehaze-cGAN：用于交通视频监控的基于多头注意力的条件GAN图像去雾

IF 3.1 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2025-12-01 Epub Date: 2025-10-27 DOI: 10.1016/j.jvcir.2025.104619

Maheshkumar H. Kolekar, Hemang Dipakbhai Chhatbar, Samprit Bose

Image dehazing is a process of eliminating haze to improve image clarity, essential for activities using computer vision. Traditional dehazing methods depend on manually constructed priors and assumptions, which frequently do not generalize in complex real-world scenes with dense or uneven haze, leading to inconsistent and suboptimal results. In this paper, we introduce, Dehaze-cGAN, a novel conditional Generative Adversarial Network designed for effective single-image dehazing. The proposed model features a Multi-Head Self Attention UNet generator that captures both local textures and long-range spatial dependencies. Complementing this, a channel attention-guided discriminator selectively emphasizes important feature channels, enhancing its ability to distinguish real haze-free images from generated outputs. Comprehensive experiments on synthetic datasets, real-world datasets, and natural hazy datasets depict that Dehaze-cGAN consistently surpasses state-of-the-art methods. The practical effectiveness of the model is further validated through significant improvements in license plate detection accuracy on dehazed traffic images.

图像去雾是一种消除雾霾以提高图像清晰度的过程，对于使用计算机视觉的活动至关重要。传统的除雾方法依赖于人工构建的先验和假设，这些先验和假设在雾霾密集或不均匀的复杂现实场景中往往不能泛化，导致结果不一致和次优。在本文中，我们介绍了Dehaze-cGAN，一种新的条件生成对抗网络，用于有效的单幅图像去雾。该模型具有多头自注意UNet生成器，可以捕获局部纹理和远程空间依赖关系。与此相补充的是，通道注意引导鉴别器选择性地强调重要的特征通道，增强其从生成的输出中区分真实无雾图像的能力。在合成数据集、真实世界数据集和自然朦胧数据集上进行的综合实验表明，Dehaze-cGAN始终优于最先进的方法。通过对去雾交通图像车牌检测精度的显著提高，进一步验证了该模型的实际有效性。

{"title":"Dehaze-cGAN: Image dehazing using a multi-head attention-based conditional GAN for traffic video monitoring","authors":"Maheshkumar H. Kolekar, Hemang Dipakbhai Chhatbar, Samprit Bose","doi":"10.1016/j.jvcir.2025.104619","DOIUrl":"10.1016/j.jvcir.2025.104619","url":null,"abstract":"<div><div>Image dehazing is a process of eliminating haze to improve image clarity, essential for activities using computer vision. Traditional dehazing methods depend on manually constructed priors and assumptions, which frequently do not generalize in complex real-world scenes with dense or uneven haze, leading to inconsistent and suboptimal results. In this paper, we introduce, Dehaze-cGAN, a novel conditional Generative Adversarial Network designed for effective single-image dehazing. The proposed model features a Multi-Head Self Attention UNet generator that captures both local textures and long-range spatial dependencies. Complementing this, a channel attention-guided discriminator selectively emphasizes important feature channels, enhancing its ability to distinguish real haze-free images from generated outputs. Comprehensive experiments on synthetic datasets, real-world datasets, and natural hazy datasets depict that Dehaze-cGAN consistently surpasses state-of-the-art methods. The practical effectiveness of the model is further validated through significant improvements in license plate detection accuracy on dehazed traffic images.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"113 ","pages":"Article 104619"},"PeriodicalIF":3.1,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145418408","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Small object detection in aerial traffic imagery: A benchmark for motorbike-dominated road scenes 航空交通图像中的小目标检测：摩托车主导道路场景的基准

IF 3.1 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2025-12-01 Epub Date: 2025-10-10 DOI: 10.1016/j.jvcir.2025.104603

Dung Truong , Quang Nguyen , Khanh-Duy Nguyen , Tam V. Nguyen , Khang Nguyen

Unmanned Aerial Vehicles (UAVs) have become indispensable for traffic monitoring, urban planning, and disaster management, particularly in high-density traffic environments like those in Southeast Asia. Vietnamese traffic, characterized by its high density of compact vehicles and unconventional patterns, poses unique challenges for object detection systems. Moreover, UAV imagery introduces additional complexities, such as variable object orientations and high-density scenes, which existing algorithms struggle to handle effectively. In this paper, we present two novel UAV datasets, UIT-Drone4 and UIT-Drone7 with 4 and 7 classes, respectively. These datasets encompass diverse environments, from urban traffic to rural roads and market areas, and provide detailed annotations for object orientation. We benchmark ten state-of-the-art object detection methods, including YOLOv8-v11 and orientation-specific approaches such as Oriented RepPoints, SASM, RTMDet, and Rotated Faster R-CNN, to evaluate their performance under real-world conditions. Our results reveal critical limitations in current methods when applied to motorbike-dominated traffic, highlighting challenges such as high object density, complex orientations, and varying environmental conditions. The UIT-Drone4 and UIT-Drone7 datasets are publicly available at UIT-Drone4-Link and UIT-Drone7-Link, respectively.

无人驾驶飞行器（uav）在交通监控、城市规划和灾害管理方面已成为不可或缺的工具，特别是在东南亚等高密度交通环境中。越南交通的特点是紧凑车辆的高密度和非常规模式，对目标检测系统提出了独特的挑战。此外，无人机图像引入了额外的复杂性，例如可变物体方向和高密度场景，现有算法难以有效处理。本文提出了两个新的无人机数据集，分别为4类和7类，分别为unit - drone4和unit - drone7。这些数据集涵盖了不同的环境，从城市交通到农村道路和市场区域，并提供了面向对象的详细注释。我们对十种最先进的目标检测方法进行了基准测试，包括YOLOv8-v11和定向特定方法，如定向RepPoints、SASM、RTMDet和旋转更快的R-CNN，以评估它们在现实世界条件下的性能。我们的研究结果揭示了当前方法在应用于摩托车主导的交通时的关键局限性，突出了诸如高物体密度、复杂方向和变化的环境条件等挑战。unit - drone4和unit - drone7数据集可分别在unit - drone4 - link和unit - drone7 - link上公开获取。

{"title":"Small object detection in aerial traffic imagery: A benchmark for motorbike-dominated road scenes","authors":"Dung Truong , Quang Nguyen , Khanh-Duy Nguyen , Tam V. Nguyen , Khang Nguyen","doi":"10.1016/j.jvcir.2025.104603","DOIUrl":"10.1016/j.jvcir.2025.104603","url":null,"abstract":"<div><div>Unmanned Aerial Vehicles (UAVs) have become indispensable for traffic monitoring, urban planning, and disaster management, particularly in high-density traffic environments like those in Southeast Asia. Vietnamese traffic, characterized by its high density of compact vehicles and unconventional patterns, poses unique challenges for object detection systems. Moreover, UAV imagery introduces additional complexities, such as variable object orientations and high-density scenes, which existing algorithms struggle to handle effectively. In this paper, we present two novel UAV datasets, UIT-Drone4 and UIT-Drone7 with 4 and 7 classes, respectively. These datasets encompass diverse environments, from urban traffic to rural roads and market areas, and provide detailed annotations for object orientation. We benchmark ten state-of-the-art object detection methods, including YOLOv8-v11 and orientation-specific approaches such as Oriented RepPoints, SASM, RTMDet, and Rotated Faster R-CNN, to evaluate their performance under real-world conditions. Our results reveal critical limitations in current methods when applied to motorbike-dominated traffic, highlighting challenges such as high object density, complex orientations, and varying environmental conditions. The UIT-Drone4 and UIT-Drone7 datasets are publicly available at <span><span>UIT-Drone4-Link</span><svg><path></path></svg></span> and <span><span>UIT-Drone7-Link</span><svg><path></path></svg></span>, respectively.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"113 ","pages":"Article 104603"},"PeriodicalIF":3.1,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145278066","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Retrieval augmented generation for smart calorie estimation in complex food scenarios 复杂食物场景中智能卡路里估算的检索增强生成

IF 3.1 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2025-12-01 Epub Date: 2025-11-03 DOI: 10.1016/j.jvcir.2025.104632

Mayank Sah, Saurya Suman, Jimson Mathew

Accurate food recognition and calorie estimation are critical for managing diet-related health issues such as obesity and diabetes. Traditional food logging methods rely on manual input, leading to inaccurate nutritional records. Although recent advances in computer vision and deep learning offer automated solutions, existing models struggle with generalizability due to homogeneous datasets and limited representation of complex cuisines like Indian food. This paper introduces a dataset containing over 15,000 images of 56 popular Indian food items. Curated from diverse sources, including social media and real-world photography, the dataset aims to capture the complexity of Indian meals, where multiple food items often appear together in a single image. This ensures greater lighting, presentation, and image quality variability compared to existing data sets. We evaluated the data set with various YOLO-based models, including YOLOv5 through YOLOv12, and enhanced the backbone with omniscale feature learning from OSNet, improving detection accuracy. In addition, we integrate a Retrieval-Augmented-Generation (RAG) module with YOLO, which refines food identification by associating fine-grained food categories with nutritional information, ingredients, and recipes. Our approach demonstrates improved performance in recognizing complex meals. It addresses key challenges in food recognition, offering a scalable solution for accurate calorie estimation, especially for culturally diverse cuisines like Indian food.

准确的食物识别和卡路里估算对于管理与饮食相关的健康问题（如肥胖和糖尿病）至关重要。传统的食物记录方法依赖于人工输入，导致营养记录不准确。尽管计算机视觉和深度学习的最新进展提供了自动化解决方案，但由于数据集同质，并且对印度菜等复杂美食的代表性有限，现有模型难以泛化。本文介绍了一个数据集，其中包含56种受欢迎的印度食品的15,000多张图像。该数据集来自各种来源，包括社交媒体和现实世界的照片，旨在捕捉印度食物的复杂性，其中多种食物经常一起出现在一张图片中。与现有数据集相比，这确保了更大的照明、呈现和图像质量可变性。我们使用多种基于YOLOv5到YOLOv12的模型对数据集进行评估，并通过从OSNet学习全尺度特征来增强主干，提高检测精度。此外，我们还将检索增强生成（RAG）模块与YOLO集成在一起，该模块通过将细粒度食品类别与营养信息、成分和食谱相关联来改进食品识别。我们的方法在识别复杂膳食方面表现出更高的性能。它解决了食物识别中的关键挑战，为准确估计卡路里提供了可扩展的解决方案，特别是对于印度菜等文化多样化的美食。

{"title":"Retrieval augmented generation for smart calorie estimation in complex food scenarios","authors":"Mayank Sah, Saurya Suman, Jimson Mathew","doi":"10.1016/j.jvcir.2025.104632","DOIUrl":"10.1016/j.jvcir.2025.104632","url":null,"abstract":"<div><div>Accurate food recognition and calorie estimation are critical for managing diet-related health issues such as obesity and diabetes. Traditional food logging methods rely on manual input, leading to inaccurate nutritional records. Although recent advances in computer vision and deep learning offer automated solutions, existing models struggle with generalizability due to homogeneous datasets and limited representation of complex cuisines like Indian food. This paper introduces a dataset containing over 15,000 images of 56 popular Indian food items. Curated from diverse sources, including social media and real-world photography, the dataset aims to capture the complexity of Indian meals, where multiple food items often appear together in a single image. This ensures greater lighting, presentation, and image quality variability compared to existing data sets. We evaluated the data set with various YOLO-based models, including YOLOv5 through YOLOv12, and enhanced the backbone with omniscale feature learning from OSNet, improving detection accuracy. In addition, we integrate a Retrieval-Augmented-Generation (RAG) module with YOLO, which refines food identification by associating fine-grained food categories with nutritional information, ingredients, and recipes. Our approach demonstrates improved performance in recognizing complex meals. It addresses key challenges in food recognition, offering a scalable solution for accurate calorie estimation, especially for culturally diverse cuisines like Indian food.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"113 ","pages":"Article 104632"},"PeriodicalIF":3.1,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145466971","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Multiple cross-modal complementation network for lightweight RGB-D salient object detection 轻量级RGB-D显著目标检测的多跨模态互补网络

IF 3.1 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2025-12-01 Epub Date: 2025-11-01 DOI: 10.1016/j.jvcir.2025.104622

Changhe Zhang, Fen Chen, Lian Huang, Zongju Peng, Xin Hu

The large model sizes and high computational costs of traditional convolutional neural networks hinder the deployment of RGB-D salient object detection (SOD) models on mobile devices. To effectively balance and improve the efficiency and accuracy of RGB-D SOD, we propose a multiple cross-modal complementation network (MCCNet) which fully utilizes complementary information in multiple dimensions. First, according to the information complementarity between depth features and RGB features , we propose a multiple cross-modal complementation (MCC) module to strengthen the feature representation and fusion ability of lightweight networks. Secondly, based on the MCC module, we propose a global and local features cooperative depth enhancement module to improve the quality of depth maps. Finally, we propose an RGB-assisted extraction and fusion backbone. RGB features are fed into this backbone and assist in the extraction of depth features, so as to be efficiently fused with extracted depth features. The experimental results on five challenging datasets show that the MCCNet achieves 1955fps on a single RTX 4090 GPU with few parameters (5.5M), and performs favorably against 12 state-of-the-art RGB-D SOD methods in term of accuracy.

传统卷积神经网络模型尺寸大、计算成本高，阻碍了RGB-D显著目标检测（SOD）模型在移动设备上的应用。为了有效平衡和提高RGB-D SOD的效率和准确性，我们提出了一个多维度充分利用互补信息的多跨模态互补网络（mcnet）。首先，根据深度特征与RGB特征之间的信息互补性，提出了多模态互补（multi - cross-modal complementary， MCC）模块，增强了轻量化网络的特征表示和融合能力；其次，在MCC模块的基础上，提出了全局与局部特征协同深度增强模块，提高深度图质量；最后，我们提出了一个rgb辅助提取和融合骨干。将RGB特征输入到该主干中，辅助深度特征的提取，从而与提取的深度特征有效融合。在5个具有挑战性的数据集上的实验结果表明，MCCNet在单个RTX 4090 GPU上以较少的参数（5.5M）达到了1955fps，并且在精度方面优于12种最先进的RGB-D SOD方法。

{"title":"Multiple cross-modal complementation network for lightweight RGB-D salient object detection","authors":"Changhe Zhang, Fen Chen, Lian Huang, Zongju Peng, Xin Hu","doi":"10.1016/j.jvcir.2025.104622","DOIUrl":"10.1016/j.jvcir.2025.104622","url":null,"abstract":"<div><div>The large model sizes and high computational costs of traditional convolutional neural networks hinder the deployment of RGB-D salient object detection (SOD) models on mobile devices. To effectively balance and improve the efficiency and accuracy of RGB-D SOD, we propose a multiple cross-modal complementation network (MCCNet) which fully utilizes complementary information in multiple dimensions. First, according to the information complementarity between depth features and RGB features , we propose a multiple cross-modal complementation (MCC) module to strengthen the feature representation and fusion ability of lightweight networks. Secondly, based on the MCC module, we propose a global and local features cooperative depth enhancement module to improve the quality of depth maps. Finally, we propose an RGB-assisted extraction and fusion backbone. RGB features are fed into this backbone and assist in the extraction of depth features, so as to be efficiently fused with extracted depth features. The experimental results on five challenging datasets show that the MCCNet achieves 1955fps on a single RTX 4090 GPU with few parameters (5.5M), and performs favorably against 12 state-of-the-art RGB-D SOD methods in term of accuracy.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"113 ","pages":"Article 104622"},"PeriodicalIF":3.1,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145466972","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

LoVCS: A local voxel center based descriptor for 3D object recognition LoVCS：基于局部体素中心的三维物体识别描述符

IF 3.1 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2025-12-01 Epub Date: 2025-10-23 DOI: 10.1016/j.jvcir.2025.104621

Wuyong Tao , Xianghong Hua , Bufan Zhao , Dong Chen , Chong Wu , Danhua Min

3D object recognition remains an active research area in computer vision and graphics. Recognizing objects in cluttered scenes is challenging due to clutter and occlusion. Local feature descriptors (LFDs), known for their robustness to clutter and occlusion, are widely used for 3D object recognition. However, existing LFDs are often affected by noise and varying point density, leading to poor descriptor matching performance. To address this, we propose a new LFD in this paper. First, a novel weighting strategy is introduced, utilizing projection distances to calculate weights for neighboring points, thereby constructing a robust local reference frame (LRF). Next, a new feature attribution (i.e., local voxel center) is proposed to compute voxel values. These voxel values are concatenated to form the final feature descriptor. This feature attribution is resistant to noise and varying point density, enhancing the overall robustness of the LFD. Additionally, we design a 3D transformation estimation method to generate transformation hypotheses. This method ranks correspondences by distance ratio and traverses the top-ranked ones to compute transformations, reducing iterations and eliminating randomness while allowing predetermined iteration counts. Experiments demonstrate that the proposed LRF achieves high repeatability and the LFD exhibits excellent matching performance. The transformation estimation method is more accurate and computationally efficient. Overall, our 3D object recognition method achieves a high recognition rate. On three experimental datasets, it gets the recognition rates of 99.07%, 98.31% and 81.13%, respectively, surpassing the comparative methods. The code is available at: https://github.com/taowuyong?tab=repositories.

三维物体识别是计算机视觉和图形学领域的一个活跃研究领域。由于杂乱和遮挡，在杂乱的场景中识别物体是具有挑战性的。局部特征描述子（lfd）以其对杂波和遮挡的鲁棒性被广泛应用于三维目标识别。然而，现有的lfd经常受到噪声和点密度变化的影响，导致描述符匹配性能差。为了解决这个问题，本文提出了一种新的LFD。首先，引入一种新的加权策略，利用投影距离计算相邻点的权重，从而构建鲁棒局部参考帧（LRF）；其次，提出了一种新的特征归属（即局部体素中心）来计算体素值。这些体素值被连接起来形成最终的特征描述符。这种特征属性可以抵抗噪声和变化的点密度，增强LFD的整体鲁棒性。此外，我们设计了一种三维变换估计方法来生成变换假设。该方法根据距离比对对应进行排序，遍历排名靠前的对应进行变换计算，减少了迭代，消除了随机性，同时允许预先确定迭代次数。实验表明，所提出的LRF具有较高的重复性，LFD具有良好的匹配性能。变换估计方法更精确，计算效率更高。总体而言，我们的三维物体识别方法实现了较高的识别率。在三个实验数据集上，该方法的识别率分别达到99.07%、98.31%和81.13%，均优于对比方法。代码可从https://github.com/taowuyong?tab=repositories获得。

{"title":"LoVCS: A local voxel center based descriptor for 3D object recognition","authors":"Wuyong Tao , Xianghong Hua , Bufan Zhao , Dong Chen , Chong Wu , Danhua Min","doi":"10.1016/j.jvcir.2025.104621","DOIUrl":"10.1016/j.jvcir.2025.104621","url":null,"abstract":"<div><div>3D object recognition remains an active research area in computer vision and graphics. Recognizing objects in cluttered scenes is challenging due to clutter and occlusion. Local feature descriptors (LFDs), known for their robustness to clutter and occlusion, are widely used for 3D object recognition. However, existing LFDs are often affected by noise and varying point density, leading to poor descriptor matching performance. To address this, we propose a new LFD in this paper. First, a novel weighting strategy is introduced, utilizing projection distances to calculate weights for neighboring points, thereby constructing a robust local reference frame (LRF). Next, a new feature attribution (i.e., local voxel center) is proposed to compute voxel values. These voxel values are concatenated to form the final feature descriptor. This feature attribution is resistant to noise and varying point density, enhancing the overall robustness of the LFD. Additionally, we design a 3D transformation estimation method to generate transformation hypotheses. This method ranks correspondences by distance ratio and traverses the top-ranked ones to compute transformations, reducing iterations and eliminating randomness while allowing predetermined iteration counts. Experiments demonstrate that the proposed LRF achieves high repeatability and the LFD exhibits excellent matching performance. The transformation estimation method is more accurate and computationally efficient. Overall, our 3D object recognition method achieves a high recognition rate. On three experimental datasets, it gets the recognition rates of 99.07%, 98.31% and 81.13%, respectively, surpassing the comparative methods. The code is available at: <span><span>https://github.com/taowuyong?tab=repositories</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"113 ","pages":"Article 104621"},"PeriodicalIF":3.1,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145365339","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Real-time facial expression recognition via quaternion Gabor convolutional neural network 基于四元数Gabor卷积神经网络的实时面部表情识别

IF 3.1 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2025-12-01 Epub Date: 2025-10-27 DOI: 10.1016/j.jvcir.2025.104625

Yu Zhou , Liyuan Guo , Beibei Jiang , Bo Wang , Kunlei Jing

Real-time facial expression recognition (FER) has become a hot spot in computer vision. In recent years, convolutional neural networks (CNNs) have been widely employed for FER. However, traditional CNN architecture usually struggles to balance performance and computation, which is crucial for real-time applications. Another challenge is the inadequate consideration of color information. CNNs always treat the color image as three independent channels or convert it to grayscale, losing important information related to facial expressions. To address these problems, we propose a lightweight quaternion Gabor CNN (LQG-CNN) for real-time FER. LQG-CNN encodes color images into quaternions, allowing it to naturally handle the interrelationships between color channels within a quaternion framework. Additionally, quaternion Gabor convolutional layers are introduced to capture spatial transformations, which require fewer parameters and offer faster inference speeds, making real-time FER feasible. Experiments on three datasets demonstrate that LQG-CNN achieves cost-efficient performance, outperforming other methods. Code will be available at https://github.com/jiangbeibe/LQG-CNN.

实时面部表情识别已成为计算机视觉领域的研究热点。近年来，卷积神经网络（cnn）被广泛应用于人工神经网络。然而，传统的CNN架构通常难以平衡性能和计算，这对实时应用至关重要。另一个挑战是对颜色信息的考虑不足。cnn总是将彩色图像作为三个独立的通道处理或将其转换为灰度，从而丢失与面部表情相关的重要信息。为了解决这些问题，我们提出了一种用于实时FER的轻量级四元数Gabor CNN （LQG-CNN）。LQG-CNN将彩色图像编码为四元数，允许它在四元数框架内自然地处理颜色通道之间的相互关系。此外，引入四元数Gabor卷积层来捕获空间变换，需要更少的参数并提供更快的推理速度，使实时FER成为可能。在三个数据集上的实验表明，LQG-CNN达到了性价比，优于其他方法。代码将在https://github.com/jiangbeibe/LQG-CNN上提供。

{"title":"Real-time facial expression recognition via quaternion Gabor convolutional neural network","authors":"Yu Zhou , Liyuan Guo , Beibei Jiang , Bo Wang , Kunlei Jing","doi":"10.1016/j.jvcir.2025.104625","DOIUrl":"10.1016/j.jvcir.2025.104625","url":null,"abstract":"<div><div>Real-time facial expression recognition (FER) has become a hot spot in computer vision. In recent years, convolutional neural networks (CNNs) have been widely employed for FER. However, traditional CNN architecture usually struggles to balance performance and computation, which is crucial for real-time applications. Another challenge is the inadequate consideration of color information. CNNs always treat the color image as three independent channels or convert it to grayscale, losing important information related to facial expressions. To address these problems, we propose a lightweight quaternion Gabor CNN (LQG-CNN) for real-time FER. LQG-CNN encodes color images into quaternions, allowing it to naturally handle the interrelationships between color channels within a quaternion framework. Additionally, quaternion Gabor convolutional layers are introduced to capture spatial transformations, which require fewer parameters and offer faster inference speeds, making real-time FER feasible. Experiments on three datasets demonstrate that LQG-CNN achieves cost-efficient performance, outperforming other methods. Code will be available at <span><span>https://github.com/jiangbeibe/LQG-CNN</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":54755,"journal":{"name":"Journal of Visual Communication and Image Representation","volume":"113 ","pages":"Article 104625"},"PeriodicalIF":3.1,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145418439","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

MIEI:A KID-based quality assessment metric for grayscale industrial equipment images MIEI：基于kid的灰度工业设备图像质量评估度量

IF 3.1 4区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Journal of Visual Communication and Image Representation

Pub Date : 2025-12-01 Epub Date: 2025-11-01 DOI: 10.1016/j.jvcir.2025.104626

TianXu Han, Jiani Sun, Haihong Li, Yanjun Zhang

Existing Image Quality Assessment metrics exhibit limited adaptability to complex industrial scenarios, constraining evaluation accuracy for Industrial Equipment Images (IEIs). We investigate with grayscale IEIs and propose a new metric named MIEI. First, we design the parametric Sobel (Para-Sobel) that dynamically adjusts central-row weight coefficients to compensate for edge detection errors induced by fixed weights in traditional operators. Second, we introduce a geometric constraint module, GOML, that couples pixel-level gradient magnitude with directional features to simultaneously capture edge length and scale variation in key regions. Experiments demonstrate that the Para-Sobel improves edge continuity detection accuracy by 24 % (FOM) and 9.6 % (SSIM) over traditional Sobel. The GOML-integrated model achieves a 14.38 % higher Edge Preservation Ratio than baseline models. Collectively, MIEI outperforms KID by 12.24 % in KROCC and 21.17 % in PLCC across critical metrics, while maintaining real-time inference at 49.5 ms.

现有的图像质量评估指标对复杂工业场景的适应性有限，限制了工业设备图像（iei）的评估准确性。我们对灰度iei进行了研究，并提出了一种新的度量，称为MIEI。首先，我们设计了参数化Sobel （Para-Sobel）算法，动态调整中心行权系数，以补偿传统算子中固定权值引起的边缘检测误差。其次，引入几何约束模块GOML，将像素级梯度大小与方向特征相结合，同时捕获关键区域的边缘长度和尺度变化。实验表明，与传统的索贝尔算法相比，Para-Sobel算法的边缘连续性检测精度分别提高了24% （FOM）和9.6% （SSIM）。与基线模型相比，gml集成模型的边缘保留率提高了14.38%。总的来说，MIEI在KROCC和PLCC的关键指标上比KID高出12.24%和21.17%，同时保持了49.5 ms的实时推断。

引用次数: 0