Pub Date : 2024-06-05DOI: 10.1109/tmm.2024.3410133
Ke Liu, Jiwei Wei, Jie Zou, Peng Wang, Yang Yang, Heng Tao Shen
{"title":"Improving Pre-trained Model-based Speech Emotion Recognition from a Low-level Speech Feature Perspective","authors":"Ke Liu, Jiwei Wei, Jie Zou, Peng Wang, Yang Yang, Heng Tao Shen","doi":"10.1109/tmm.2024.3410133","DOIUrl":"https://doi.org/10.1109/tmm.2024.3410133","url":null,"abstract":"","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"12 1","pages":""},"PeriodicalIF":7.3,"publicationDate":"2024-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141968856","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-05DOI: 10.1109/tmm.2024.3410129
Jiayi Li, Min Jiang, Jun Kong, Xuefeng Tao, Xi Luo
{"title":"Learning Semantic Polymorphic Mapping for Text-Based Person Retrieval","authors":"Jiayi Li, Min Jiang, Jun Kong, Xuefeng Tao, Xi Luo","doi":"10.1109/tmm.2024.3410129","DOIUrl":"https://doi.org/10.1109/tmm.2024.3410129","url":null,"abstract":"","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"29 1","pages":""},"PeriodicalIF":7.3,"publicationDate":"2024-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141968875","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-05DOI: 10.1109/tmm.2024.3410139
Zhenglong Cui, Da Yang, Hao Sheng, Sizhe Wang, Rongshan Chen, Ruixuan Cong, Wei Ke
{"title":"Triple Consistency for Transparent Cheating Problem in Light Field Depth Estimation","authors":"Zhenglong Cui, Da Yang, Hao Sheng, Sizhe Wang, Rongshan Chen, Ruixuan Cong, Wei Ke","doi":"10.1109/tmm.2024.3410139","DOIUrl":"https://doi.org/10.1109/tmm.2024.3410139","url":null,"abstract":"","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"13 1","pages":""},"PeriodicalIF":7.3,"publicationDate":"2024-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141968852","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-05DOI: 10.1109/TMM.2024.3410116
Chao Jiao;Huanqiang Zeng;Jing Chen;Chih-Hsien Hsia;Tianlei Wang;Kai-Kuang Ma
Screen content coding (SCC) in Versatile Video Coding (VVC) improves the coding efficiency of screen content videos (SCVs) significantly but results in high computational complexity due to the quad-tree plus multi-type tree (QTMT) structure of the coding unit (CU) partitioning. Therefore, we make the first attempt to reduce the encoding complexity from the perspective of CU partitioning for SCC in VVC. To this end, a fast CU partition prediction method is technically developed for VVC-SCC. First, to solve the problem of lacking sufficient SCC training data, SCVs are collected to establish a database containing CUs of various sizes and corresponding partition labels. Second, to determine the partition decision in advance, a novel WA-CNN model is proposed, which is capable of predicting two large CUs for VVC-SCC by adjusting the feature channels based on the size of input CU blocks. Finally, considering the imbalanced proportion of diverse partition decisions, a loss function with the weight that equalizes the contribution of imbalanced data is formulated to train the proposed WA-CNN model. Experimental results show that the proposed model reduces the SCC intra-encoding time by 35.65% ${sim }$ 38.31% with an average of 1.84% ${sim }$ 2.42% BDBR increase.
多功能视频编码（VVC）中的屏幕内容编码（SCC）大大提高了屏幕内容视频（SCV）的编码效率，但由于编码单元（CU）分区的四叉树加多类型树（QTMT）结构，导致了较高的计算复杂度。因此，我们首次尝试从 CU 分区的角度降低 VVC 中 SCC 的编码复杂度。为此，我们从技术上为 VVC-SCC 开发了一种快速 CU 分区预测方法。首先，为了解决缺乏足够 SCC 训练数据的问题，收集了 SCV，建立了一个包含不同大小 CU 和相应分区标签的数据库。其次，为了提前确定分区决策，提出了一种新颖的 WA-CNN 模型，该模型能够根据输入 CU 块的大小调整特征通道，从而预测 VVC-SCC 的两个大 CU。最后，考虑到不同分区决策的不平衡比例，制定了一个具有均衡不平衡数据贡献权重的损失函数来训练所提出的 WA-CNN 模型。实验结果表明，所提出的模型减少了 35.65%${sim }$38.31% 的 SCC 内部编码时间，平均增加了 1.84%${sim }$2.42% 的 BDBR。
{"title":"Width-Adaptive CNN: Fast CU Partition Prediction for VVC Screen Content Coding","authors":"Chao Jiao;Huanqiang Zeng;Jing Chen;Chih-Hsien Hsia;Tianlei Wang;Kai-Kuang Ma","doi":"10.1109/TMM.2024.3410116","DOIUrl":"10.1109/TMM.2024.3410116","url":null,"abstract":"Screen content coding (SCC) in Versatile Video Coding (VVC) improves the coding efficiency of screen content videos (SCVs) significantly but results in high computational complexity due to the quad-tree plus multi-type tree (QTMT) structure of the coding unit (CU) partitioning. Therefore, we make the first attempt to reduce the encoding complexity from the perspective of CU partitioning for SCC in VVC. To this end, a fast CU partition prediction method is technically developed for VVC-SCC. First, to solve the problem of lacking sufficient SCC training data, SCVs are collected to establish a database containing CUs of various sizes and corresponding partition labels. Second, to determine the partition decision in advance, a novel WA-CNN model is proposed, which is capable of predicting two large CUs for VVC-SCC by adjusting the feature channels based on the size of input CU blocks. Finally, considering the imbalanced proportion of diverse partition decisions, a loss function with the weight that equalizes the contribution of imbalanced data is formulated to train the proposed WA-CNN model. Experimental results show that the proposed model reduces the SCC intra-encoding time by 35.65%\u0000<inline-formula><tex-math>${sim }$</tex-math></inline-formula>\u000038.31% with an average of 1.84%\u0000<inline-formula><tex-math>${sim }$</tex-math></inline-formula>\u00002.42% BDBR increase.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"26 ","pages":"9372-9382"},"PeriodicalIF":8.4,"publicationDate":"2024-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141968853","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-04DOI: 10.1109/TMM.2024.3407664
Zheng Wang;Zhenwei Gao;Mengqun Han;Yang Yang;Heng Tao Shen
Based on deterministic single-point embedding, most extant image-text retrieval methods only focus on the match of ground truth while suffering from one-to-many correspondence, where besides annotated positives, many similar instances of another modality should be retrieved by a given query. Recent solutions of probabilistic embedding and rectangle mapping still encounter some drawbacks, albeit their promising effectiveness at multiple matches. Meanwhile, the exploration of one-to-many correspondence is still insufficient. Therefore, this paper proposes a novel geometric representation to E stimate the S emantics of heterogeneous data via S ector E mbedding (dubbed ESSE ). Specifically, a given image/text can be projected as a sector, where its symmetric axis represents mean semantics and the aperture estimates uncertainty. Further, a sector matching loss is introduced to better handle the multiplicity by considering the sine of included angles as distance calculation, which encourages candidates to be contained by the apertures of a query sector. The experimental results on three widely used benchmarks CUB, Flickr30 K and MS-COCO reveal that sector embedding can achieve competitive performance on multiple matches and also improve the traditional ground-truth matching of the baselines. Additionally, we also verify the generalization to video-text retrieval on two extensively used datasets of MSRVTT and MSVD, and to text-based person retrieval on CUHK-PEDES. This superiority and effectiveness can also demonstrate that the bounded property of the aperture can better estimate semantic uncertainty when compared to prior remedies.
{"title":"Estimating the Semantics via Sector Embedding for Image-Text Retrieval","authors":"Zheng Wang;Zhenwei Gao;Mengqun Han;Yang Yang;Heng Tao Shen","doi":"10.1109/TMM.2024.3407664","DOIUrl":"10.1109/TMM.2024.3407664","url":null,"abstract":"Based on deterministic single-point embedding, most extant image-text retrieval methods only focus on the match of ground truth while suffering from one-to-many correspondence, where besides annotated positives, many similar instances of another modality should be retrieved by a given query. Recent solutions of probabilistic embedding and rectangle mapping still encounter some drawbacks, albeit their promising effectiveness at multiple matches. Meanwhile, the exploration of one-to-many correspondence is still insufficient. Therefore, this paper proposes a novel geometric representation to \u0000<underline>E</u>\u0000stimate the \u0000<underline>S</u>\u0000emantics of heterogeneous data via \u0000<underline>S</u>\u0000ector \u0000<underline>E</u>\u0000mbedding (dubbed \u0000<bold>ESSE</b>\u0000). Specifically, a given image/text can be projected as a sector, where its symmetric axis represents mean semantics and the aperture estimates uncertainty. Further, a sector matching loss is introduced to better handle the multiplicity by considering the sine of included angles as distance calculation, which encourages candidates to be contained by the apertures of a query sector. The experimental results on three widely used benchmarks CUB, Flickr30 K and MS-COCO reveal that sector embedding can achieve competitive performance on multiple matches and also improve the traditional ground-truth matching of the baselines. Additionally, we also verify the generalization to video-text retrieval on two extensively used datasets of MSRVTT and MSVD, and to text-based person retrieval on CUHK-PEDES. This superiority and effectiveness can also demonstrate that the bounded property of the aperture can better estimate semantic uncertainty when compared to prior remedies.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"26 ","pages":"10342-10353"},"PeriodicalIF":8.4,"publicationDate":"2024-06-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141968859","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-03DOI: 10.1109/TMM.2024.3407661
Lei Ma;Xin Luo;Hanyu Hong;Fanman Meng;Qingbo Wu
Image retrieval with fine-grained categories is an extremely challenging task due to the high intraclass variance and low interclass variance. Most previous works have focused on localizing discriminative image regions in isolation, but have rarely exploited correlations across the different discriminative regions to alleviate intraclass differences. In addition, the intraclass compactness of embedding features is ensured by extra regularization terms that only exist during the training phase, which appear to generalize less well in the inference phase. Finally, the information granularity of the distance measure should distinguish subtle visual differences and the correlation between the embedding features and the quantized features should be maximized sufficiently. To address the above issues, we propose a logit variated product quantization method based on part interaction and metric learning with knowledge distillation for fine-grained image retrieval. Specifically, we introduce a causal context module into the deep navigator to generate discriminative regions and utilize a channelwise cross-part fusion transformer to model the part correlations while alleviating intraclass differences. Subsequently, we design a logit variation module based on a weighted sum scheme to further reduce the intraclass variance of the embedding features directly and enhance the learning power of the quantization model. Finally, we propose a novel product quantization loss based on metric learning and knowledge distillation to enhance the correlation between the embedding features and the quantized features and allow the quantization features to learn more knowledge from the embedding features. The experimental results on several fine-grained datasets demonstrate that the proposed method is superior to state-of-the-art fine-grained image retrieval methods.
{"title":"Logit Variated Product Quantization Based on Parts Interaction and Metric Learning With Knowledge Distillation for Fine-Grained Image Retrieval","authors":"Lei Ma;Xin Luo;Hanyu Hong;Fanman Meng;Qingbo Wu","doi":"10.1109/TMM.2024.3407661","DOIUrl":"10.1109/TMM.2024.3407661","url":null,"abstract":"Image retrieval with fine-grained categories is an extremely challenging task due to the high intraclass variance and low interclass variance. Most previous works have focused on localizing discriminative image regions in isolation, but have rarely exploited correlations across the different discriminative regions to alleviate intraclass differences. In addition, the intraclass compactness of embedding features is ensured by extra regularization terms that only exist during the training phase, which appear to generalize less well in the inference phase. Finally, the information granularity of the distance measure should distinguish subtle visual differences and the correlation between the embedding features and the quantized features should be maximized sufficiently. To address the above issues, we propose a logit variated product quantization method based on part interaction and metric learning with knowledge distillation for fine-grained image retrieval. Specifically, we introduce a causal context module into the deep navigator to generate discriminative regions and utilize a channelwise cross-part fusion transformer to model the part correlations while alleviating intraclass differences. Subsequently, we design a logit variation module based on a weighted sum scheme to further reduce the intraclass variance of the embedding features directly and enhance the learning power of the quantization model. Finally, we propose a novel product quantization loss based on metric learning and knowledge distillation to enhance the correlation between the embedding features and the quantized features and allow the quantization features to learn more knowledge from the embedding features. The experimental results on several fine-grained datasets demonstrate that the proposed method is superior to state-of-the-art fine-grained image retrieval methods.","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"26 ","pages":"10406-10419"},"PeriodicalIF":8.4,"publicationDate":"2024-06-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141968857","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-03DOI: 10.1109/tmm.2024.3408683
Marco Cotogni, Marco Arazzi, Claudio Cusano
{"title":"PhotoStyle60: A Photographic Style Dataset for Photo Authorship Attribution and Photographic Style Transfer","authors":"Marco Cotogni, Marco Arazzi, Claudio Cusano","doi":"10.1109/tmm.2024.3408683","DOIUrl":"https://doi.org/10.1109/tmm.2024.3408683","url":null,"abstract":"","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"91 1","pages":""},"PeriodicalIF":7.3,"publicationDate":"2024-06-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141968865","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-05-31DOI: 10.1109/tmm.2024.3408060
Xichu Ma, Yuchen Wang, Ye Wang
{"title":"Symbolic Music Generation from Graph-Learning-based Preference Modeling and Textual Queries","authors":"Xichu Ma, Yuchen Wang, Ye Wang","doi":"10.1109/tmm.2024.3408060","DOIUrl":"https://doi.org/10.1109/tmm.2024.3408060","url":null,"abstract":"","PeriodicalId":13273,"journal":{"name":"IEEE Transactions on Multimedia","volume":"116 1","pages":""},"PeriodicalIF":7.3,"publicationDate":"2024-05-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141193073","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：