Pub Date : 2024-11-01DOI: 10.1109/LSP.2024.3490399
Shiyuan Tang;Jiangqun Ni;Wenkang Su;Yulin Zhang
This letter concentrates on the challenges of deep learning-based robust image watermarking against print-scanning, print-camera, and screen-shooting attacks for “physical channel transmission”. Given the excellent performance demonstrated by wavelet domain watermarking, in this paper, we incorporate the wavelet integrated convolutional neural networks (CNNs) and propose a Deep Wavelet-domain Watermarking (DWW) model, which is dedicated to embedding watermarks in the wavelet domain rather than the spatial domain of the previous arts. In addition, a frequency-domain enhanced mask loss is developed to increase the loss weight in the high-frequency regions of the image during back-propagation, thereby encouraging the model to embed the message in low-frequency components with priority so as to improve the robustness performance. Experiment results show that the proposed DWW consistently outperforms other state-of-the-art (SOTA) schemes by a clear margin in terms of embedding capacity, imperceptibility, and robustness.
{"title":"DWW: Robust Deep Wavelet-Domain Watermarking With Enhanced Frequency Mask","authors":"Shiyuan Tang;Jiangqun Ni;Wenkang Su;Yulin Zhang","doi":"10.1109/LSP.2024.3490399","DOIUrl":"https://doi.org/10.1109/LSP.2024.3490399","url":null,"abstract":"This letter concentrates on the challenges of deep learning-based robust image watermarking against print-scanning, print-camera, and screen-shooting attacks for “physical channel transmission”. Given the excellent performance demonstrated by wavelet domain watermarking, in this paper, we incorporate the wavelet integrated convolutional neural networks (CNNs) and propose a Deep Wavelet-domain Watermarking (DWW) model, which is dedicated to embedding watermarks in the wavelet domain rather than the spatial domain of the previous arts. In addition, a frequency-domain enhanced mask loss is developed to increase the loss weight in the high-frequency regions of the image during back-propagation, thereby encouraging the model to embed the message in low-frequency components with priority so as to improve the robustness performance. Experiment results show that the proposed DWW consistently outperforms other state-of-the-art (SOTA) schemes by a clear margin in terms of embedding capacity, imperceptibility, and robustness.","PeriodicalId":13154,"journal":{"name":"IEEE Signal Processing Letters","volume":"31 ","pages":"3074-3078"},"PeriodicalIF":3.2,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142600133","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-31DOI: 10.1109/LSP.2024.3490403
Xiuping Peng;Yu Wang;Zilong Liu
This letter is concerned with efficient design of two-dimensional (2-D) Golay complementary array sets (GCASs) with ideal aperiodic sums for two correlation directions. Two new direct constructions of 2-D GCASs with highly flexible array sizes are proposed. The core idea is to truncate certain columns from large arrays generated by 2-D extended generalized Boolean functions (EGBFs). We show that these 2-D GCASs lead to highly flexible uniform rectangular array (URA) configurations for precoding matrices in omni-directional massive multi-input multi-output (MIMO) transmission.
{"title":"New Constructions of 2-D Golay Complementary Array Sets With Highly Flexible Array Sizes for Massive MIMO Omni-Directional Transmission","authors":"Xiuping Peng;Yu Wang;Zilong Liu","doi":"10.1109/LSP.2024.3490403","DOIUrl":"https://doi.org/10.1109/LSP.2024.3490403","url":null,"abstract":"This letter is concerned with efficient design of two-dimensional (2-D) Golay complementary array sets (GCASs) with ideal aperiodic sums for two correlation directions. Two new direct constructions of 2-D GCASs with highly flexible array sizes are proposed. The core idea is to truncate certain columns from large arrays generated by 2-D extended generalized Boolean functions (EGBFs). We show that these 2-D GCASs lead to highly flexible uniform rectangular array (URA) configurations for precoding matrices in omni-directional massive multi-input multi-output (MIMO) transmission.","PeriodicalId":13154,"journal":{"name":"IEEE Signal Processing Letters","volume":"31 ","pages":"3129-3133"},"PeriodicalIF":3.2,"publicationDate":"2024-10-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142672093","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-31DOI: 10.1109/LSP.2024.3488521
Mengxiao Tian;Shuo Yang;Xinxiao Wu;Yunde Jia
When applying a trained image-text matching model to a new scenario, the performance may largely degrade due to domain shift, which makes it impractical in real-world applications. In this paper, we make the first attempt on adapting the image-text matching model well-trained on a labeled source domain to an unlabeled target domain in the absence of source data, namely, source-free image-text matching. This task is challenging since it has no direct access to the source data when learning to reduce the doma in shift. To address this challenge, we propose a simple yet effective method that introduces uncertainty-aware learning to generate high-quality pseudo-pairs of image and text for target adaptation. Specifically, starting with using the pre-trained source model to retrieve several top-ranked image-text pairs from the target domain as pseudo-pairs, we then model uncertainty of each pseudo-pair by calculating the variance of retrieved texts (resp. images) given the paired image (resp. text) as query, and finally incorporate the uncertainty into an objective function to down-weight noisy pseudo-pairs for better training, thereby enhancing adaptation. This uncertainty-aware training approach can be generally applied on all existing models. Extensive experiments on the COCO and Flickr30K datasets demonstrate the effectiveness of the proposed method.
将训练有素的图像-文本匹配模型应用到新的场景时，其性能可能会因领域转移而大幅下降，这使其在实际应用中变得不切实际。在本文中，我们首次尝试在没有源数据的情况下，将在标注源领域训练有素的图像文本匹配模型应用到无标注的目标领域，即无源图像文本匹配。这项任务极具挑战性，因为它在学习减少转移中的 doma 时无法直接访问源数据。为了应对这一挑战，我们提出了一种简单而有效的方法，即引入不确定性感知学习，生成高质量的图像和文本伪对，用于目标适配。具体来说，我们首先使用预先训练好的源模型从目标领域中检索出几个排名靠前的图像-文本配对作为伪配对，然后以配对图像（或文本）为查询条件，通过计算检索到的文本（或图像）的方差来建立每个伪配对的不确定性模型，最后将不确定性纳入目标函数，以降低噪声伪配对的权重，从而提高训练效果，增强适应性。这种不确定性感知训练方法可普遍应用于所有现有模型。在 COCO 和 Flickr30K 数据集上进行的大量实验证明了所提方法的有效性。
{"title":"Source-Free Image-Text Matching via Uncertainty-Aware Learning","authors":"Mengxiao Tian;Shuo Yang;Xinxiao Wu;Yunde Jia","doi":"10.1109/LSP.2024.3488521","DOIUrl":"https://doi.org/10.1109/LSP.2024.3488521","url":null,"abstract":"When applying a trained image-text matching model to a new scenario, the performance may largely degrade due to domain shift, which makes it impractical in real-world applications. In this paper, we make the first attempt on adapting the image-text matching model well-trained on a labeled source domain to an unlabeled target domain in the absence of source data, namely, source-free image-text matching. This task is challenging since it has no direct access to the source data when learning to reduce the doma in shift. To address this challenge, we propose a simple yet effective method that introduces uncertainty-aware learning to generate high-quality pseudo-pairs of image and text for target adaptation. Specifically, starting with using the pre-trained source model to retrieve several top-ranked image-text pairs from the target domain as pseudo-pairs, we then model uncertainty of each pseudo-pair by calculating the variance of retrieved texts (resp. images) given the paired image (resp. text) as query, and finally incorporate the uncertainty into an objective function to down-weight noisy pseudo-pairs for better training, thereby enhancing adaptation. This uncertainty-aware training approach can be generally applied on all existing models. Extensive experiments on the COCO and Flickr30K datasets demonstrate the effectiveness of the proposed method.","PeriodicalId":13154,"journal":{"name":"IEEE Signal Processing Letters","volume":"31 ","pages":"3059-3063"},"PeriodicalIF":3.2,"publicationDate":"2024-10-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142595033","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Accurate depth estimation is crucial for various applications that require precise 3D information about the surrounding environment. In this paper, we propose MonoBooster, a feature aggregation architecture to enhance the performance of self-supervised monocular depth estimation. Specifically, we introduce a semi-dense skip connection scheme to aggregate multi-level features extracted from the backbone network. Additionally, we present a novel Cross-Level Attention (CLA) module to fuse the connected features. The CLA module captures spatial correlation using pyramid depth-wise convolution and adaptively extracts channel information from both low-level and high-level features, facilitating the translation from input RGB image to estimated depth map. Experimental results on the KITTI and Make3D datasets validate the effectiveness of the proposed MonoBooster. Notably, the MonoBooster architecture is flexible and can be seamlessly integrated into popular backbones, resulting in enhanced depth estimation performance.
{"title":"MonoBooster: Semi-Dense Skip Connection With Cross-Level Attention for Boosting Self-Supervised Monocular Depth Estimation","authors":"Changhao Wang;Guanwen Zhang;Zhengyun Cheng;Wei Zhou","doi":"10.1109/LSP.2024.3488499","DOIUrl":"https://doi.org/10.1109/LSP.2024.3488499","url":null,"abstract":"Accurate depth estimation is crucial for various applications that require precise 3D information about the surrounding environment. In this paper, we propose MonoBooster, a feature aggregation architecture to enhance the performance of self-supervised monocular depth estimation. Specifically, we introduce a semi-dense skip connection scheme to aggregate multi-level features extracted from the backbone network. Additionally, we present a novel Cross-Level Attention (CLA) module to fuse the connected features. The CLA module captures spatial correlation using pyramid depth-wise convolution and adaptively extracts channel information from both low-level and high-level features, facilitating the translation from input RGB image to estimated depth map. Experimental results on the KITTI and Make3D datasets validate the effectiveness of the proposed MonoBooster. Notably, the MonoBooster architecture is flexible and can be seamlessly integrated into popular backbones, resulting in enhanced depth estimation performance.","PeriodicalId":13154,"journal":{"name":"IEEE Signal Processing Letters","volume":"31 ","pages":"3069-3073"},"PeriodicalIF":3.2,"publicationDate":"2024-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142600132","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-29DOI: 10.1109/LSP.2024.3487795
Lin Zheng;Han Zhu;Sanli Tian;Qingwei Zhao;Ta Li
Serialized Output Training (SOT) has emerged as the mainstream approach for addressing the multi-talker overlapped speech recognition challenge due to its simplicity. However, SOT encounters cross-domain performance degradation which hinders its application. Meanwhile, traditional domain adaption methods may harm the accuracy of speaker change point prediction evaluated by UD-CER, which is an important metric in SOT. To solve these issues, we propose Pseudo-Labeling based SOT (PL-SOT) for domain adaptation by treating speaker change token ( $< $ sc $>$ ) specially during training to increase the accuracy of speaker change point prediction. Firstly, we improve CTC loss by proposing Weakening and Enhancing CTC (WE-CTC) loss to weaken the learning of error-prone labels surrounding $<$ sc $>$ while enhance the emission probability of $< $ sc $>$ through modifying posteriors of the pseudo-labels. Secondly, we introduce Weighted Confidence Filter (WCF) that assigns higher scores of $<$ sc $>$ to exclude low-quality pseudo-labels without hurting the $< $ sc $>$ prediction. Experimental results show that PL-SOT achieves 17.7%/12.8% average relative reduction of CER/UD-CER, with AliMeeting as source domain and AISHELL-4 along with MagicData-RAMC as target domain.
{"title":"Unsupervised Domain Adaptation on End-to-End Multi-Talker Overlapped Speech Recognition","authors":"Lin Zheng;Han Zhu;Sanli Tian;Qingwei Zhao;Ta Li","doi":"10.1109/LSP.2024.3487795","DOIUrl":"https://doi.org/10.1109/LSP.2024.3487795","url":null,"abstract":"Serialized Output Training (SOT) has emerged as the mainstream approach for addressing the multi-talker overlapped speech recognition challenge due to its simplicity. However, SOT encounters cross-domain performance degradation which hinders its application. Meanwhile, traditional domain adaption methods may harm the accuracy of speaker change point prediction evaluated by UD-CER, which is an important metric in SOT. To solve these issues, we propose Pseudo-Labeling based SOT (PL-SOT) for domain adaptation by treating speaker change token (\u0000<inline-formula><tex-math>$< $</tex-math></inline-formula>\u0000sc\u0000<inline-formula><tex-math>$>$</tex-math></inline-formula>\u0000) specially during training to increase the accuracy of speaker change point prediction. Firstly, we improve CTC loss by proposing \u0000<italic>Weakening and Enhancing CTC</i>\u0000 (WE-CTC) loss to weaken the learning of error-prone labels surrounding \u0000<inline-formula><tex-math>$<$</tex-math></inline-formula>\u0000sc\u0000<inline-formula><tex-math>$>$</tex-math></inline-formula>\u0000 while enhance the emission probability of \u0000<inline-formula><tex-math>$< $</tex-math></inline-formula>\u0000sc\u0000<inline-formula><tex-math>$>$</tex-math></inline-formula>\u0000 through modifying posteriors of the pseudo-labels. Secondly, we introduce \u0000<italic>Weighted Confidence Filter</i>\u0000 (WCF) that assigns higher scores of \u0000<inline-formula><tex-math>$<$</tex-math></inline-formula>\u0000sc\u0000<inline-formula><tex-math>$>$</tex-math></inline-formula>\u0000 to exclude low-quality pseudo-labels without hurting the \u0000<inline-formula><tex-math>$< $</tex-math></inline-formula>\u0000sc\u0000<inline-formula><tex-math>$>$</tex-math></inline-formula>\u0000 prediction. Experimental results show that PL-SOT achieves 17.7%/12.8% average relative reduction of CER/UD-CER, with AliMeeting as source domain and AISHELL-4 along with MagicData-RAMC as target domain.","PeriodicalId":13154,"journal":{"name":"IEEE Signal Processing Letters","volume":"31 ","pages":"3119-3123"},"PeriodicalIF":3.2,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142671995","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The increasing adoption of Artificial Intelligence (AI) in engineering problems calls for the development of calibration methods capable of offering robust statistical reliability guarantees. The calibration of black box AI models is carried out via the optimization of hyperparameters dictating architecture, optimization, and/or inference configuration. Prior work has introduced learn-then-test (LTT), a calibration procedure for hyperparameter optimization (HPO) that provides statistical guarantees on average performance measures. Recognizing the importance of controlling risk-aware objectives in engineering contexts, this work introduces a variant of LTT that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem.
{"title":"Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization","authors":"Amirmohammad Farzaneh;Sangwoo Park;Osvaldo Simeone","doi":"10.1109/LSP.2024.3486238","DOIUrl":"https://doi.org/10.1109/LSP.2024.3486238","url":null,"abstract":"The increasing adoption of Artificial Intelligence (AI) in engineering problems calls for the development of calibration methods capable of offering robust statistical reliability guarantees. The calibration of black box AI models is carried out via the optimization of hyperparameters dictating architecture, optimization, and/or inference configuration. Prior work has introduced learn-then-test (LTT), a calibration procedure for hyperparameter optimization (HPO) that provides statistical guarantees on average performance measures. Recognizing the importance of controlling risk-aware objectives in engineering contexts, this work introduces a variant of LTT that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem.","PeriodicalId":13154,"journal":{"name":"IEEE Signal Processing Letters","volume":"31 ","pages":"3044-3048"},"PeriodicalIF":3.2,"publicationDate":"2024-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142595068","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-24DOI: 10.1109/LSP.2024.3486113
Ting Lv;Chuanming Ji;Hong Jiang;Yu Liu
In the field of infrared and visible image fusion, current algorithms often focus on complex feature extraction and sophisticated fusion mechanisms, ignoring the issues of information redundancy and feature imbalance. These limit effective information aggregation. To address these issues, this paper proposes a hierarchical fusion strategy with a two-stage training network, abbreviated as HF2TNet, which achieves effective information aggregation in a staged manner. In the initial training stage, a three-stream encoder-decoder architecture is proposed, seamlessly integrating CNN and transformer modules. This architecture extracts both global and local features from visible and infrared images, capturing their shared attributes before the fusion process. Moreover, a multi-shared attention module (MSAM) is proposed to profoundly reconstruct and augment the visible and infrared features, ensuring the preservation and enhancement of details across modalities. In the subsequent stage, HF2TNet utilizes the pre-integrated features as query inputs for the dual MSAMs. These modules interact with the previously reconstructed infrared and visible features to enhance complementary information and ensure a balanced feature fusion. Experimental results indicate HF2TNet's superior performance on standard datasets like MSRS and TNO, especially in complex scenes, demonstrating its potential in multimodal image fusion.
{"title":"HF2TNet: A Hierarchical Fusion Two-Stage Training Network for Infrared and Visible Image Fusion","authors":"Ting Lv;Chuanming Ji;Hong Jiang;Yu Liu","doi":"10.1109/LSP.2024.3486113","DOIUrl":"https://doi.org/10.1109/LSP.2024.3486113","url":null,"abstract":"In the field of infrared and visible image fusion, current algorithms often focus on complex feature extraction and sophisticated fusion mechanisms, ignoring the issues of information redundancy and feature imbalance. These limit effective information aggregation. To address these issues, this paper proposes a hierarchical fusion strategy with a two-stage training network, abbreviated as HF2TNet, which achieves effective information aggregation in a staged manner. In the initial training stage, a three-stream encoder-decoder architecture is proposed, seamlessly integrating CNN and transformer modules. This architecture extracts both global and local features from visible and infrared images, capturing their shared attributes before the fusion process. Moreover, a multi-shared attention module (MSAM) is proposed to profoundly reconstruct and augment the visible and infrared features, ensuring the preservation and enhancement of details across modalities. In the subsequent stage, HF2TNet utilizes the pre-integrated features as query inputs for the dual MSAMs. These modules interact with the previously reconstructed infrared and visible features to enhance complementary information and ensure a balanced feature fusion. Experimental results indicate HF2TNet's superior performance on standard datasets like MSRS and TNO, especially in complex scenes, demonstrating its potential in multimodal image fusion.","PeriodicalId":13154,"journal":{"name":"IEEE Signal Processing Letters","volume":"31 ","pages":"3164-3168"},"PeriodicalIF":3.2,"publicationDate":"2024-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142672063","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-22DOI: 10.1109/LSP.2024.3484932
Kyungdeuk Ko;Bokyeung Lee;Jonghwan Hong;Hanseok Ko
In recent years, with the advancement of deep learning technology and the emergence of smart devices, there has been a growing interest in keyword spotting (KWS), which is used to activate AI systems with automatic speech recognition and text-to-speech. However, smart devices with KWS often encounter false alarm errors when inputting unexpected words. To address this issue, existing KWS methods typically train non-target words as an unknown class. Despite these efforts, there is still a possibility that unseen words not trained as part of the unknown class could be misclassified as one of the target words. To overcome this limitation, we propose a new method named Keyword Feature Augmentation (KFA) for open-set KWS. KFA performs feature augmentation through adversarial learning to increase the loss. The augmented features are constrained within a limited space using label smoothing. Unlike other generative model-based open set recognition (OSR) methods, KFA does not require any additional training parameters or repeated operation for inference. As a result, KFA has achieved a 0.955 AUROC score and 97.34% target class accuracy for Google Speech Commands V1, and a 0.959 AUROC score and 98.17% target class accuracy for Google Speech Commands V2, which is the highest performance when compared to various OSR methods.
{"title":"KFA: Keyword Feature Augmentation for Open Set Keyword Spotting","authors":"Kyungdeuk Ko;Bokyeung Lee;Jonghwan Hong;Hanseok Ko","doi":"10.1109/LSP.2024.3484932","DOIUrl":"https://doi.org/10.1109/LSP.2024.3484932","url":null,"abstract":"In recent years, with the advancement of deep learning technology and the emergence of smart devices, there has been a growing interest in keyword spotting (KWS), which is used to activate AI systems with automatic speech recognition and text-to-speech. However, smart devices with KWS often encounter false alarm errors when inputting unexpected words. To address this issue, existing KWS methods typically train non-target words as an \u0000<italic>unknown</i>\u0000 class. Despite these efforts, there is still a possibility that unseen words not trained as part of the \u0000<italic>unknown</i>\u0000 class could be misclassified as one of the target words. To overcome this limitation, we propose a new method named Keyword Feature Augmentation (KFA) for open-set KWS. KFA performs feature augmentation through adversarial learning to increase the loss. The augmented features are constrained within a limited space using label smoothing. Unlike other generative model-based open set recognition (OSR) methods, KFA does not require any additional training parameters or repeated operation for inference. As a result, KFA has achieved a 0.955 AUROC score and 97.34% target class accuracy for Google Speech Commands V1, and a 0.959 AUROC score and 98.17% target class accuracy for Google Speech Commands V2, which is the highest performance when compared to various OSR methods.","PeriodicalId":13154,"journal":{"name":"IEEE Signal Processing Letters","volume":"31 ","pages":"2985-2989"},"PeriodicalIF":3.2,"publicationDate":"2024-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142524238","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-21DOI: 10.1109/LSP.2024.3484289
Feicheng Huang;Wenbo Zhou;Yue Huang;Xinghao Ding
Data pruning is observed to substantially reduce the computation and memory costs of model training. Previous studies have primarily focused on constructing a series of coresets with representative samples by leveraging predefined rules for evaluating sample importance. Learning dynamics and selection bias, however, are rarely being considered. In this letter, a novel Sample-wise Dynamic Probabilistic Pruning (SwDPP) method is proposed for efficient training. Specifically, instead of hard-pruning the samples that are considered easy or well-learned, we formulate the pruning process as a probabilistic sampling problem. This is achieved by a carefully-designed soft-selection mechanism, which constantly expresses learning dynamics and relaxes selection bias. Moreover, to alleviate the accuracy drop under high pruning rates, we introduce a probabilistic Mixup strategy for information diversity maintenance. Extensive experiments conducted on CIFAR-10, CIFAR-100 and Tiny-ImageNet show that, the proposed SwDPP outperforms current state-of-the-art methods across various pruning settings. Notably, on CIFAR-10 and CIFAR-100, SwDPP achieves lossless training acceleration using only 70% of the data per epoch.
{"title":"Efficient Training Acceleration via Sample-Wise Dynamic Probabilistic Pruning","authors":"Feicheng Huang;Wenbo Zhou;Yue Huang;Xinghao Ding","doi":"10.1109/LSP.2024.3484289","DOIUrl":"https://doi.org/10.1109/LSP.2024.3484289","url":null,"abstract":"Data pruning is observed to substantially reduce the computation and memory costs of model training. Previous studies have primarily focused on constructing a series of coresets with representative samples by leveraging predefined rules for evaluating sample importance. Learning dynamics and selection bias, however, are rarely being considered. In this letter, a novel Sample-wise Dynamic Probabilistic Pruning (SwDPP) method is proposed for efficient training. Specifically, instead of hard-pruning the samples that are considered easy or well-learned, we formulate the pruning process as a probabilistic sampling problem. This is achieved by a carefully-designed soft-selection mechanism, which constantly expresses learning dynamics and relaxes selection bias. Moreover, to alleviate the accuracy drop under high pruning rates, we introduce a probabilistic Mixup strategy for information diversity maintenance. Extensive experiments conducted on CIFAR-10, CIFAR-100 and Tiny-ImageNet show that, the proposed SwDPP outperforms current state-of-the-art methods across various pruning settings. Notably, on CIFAR-10 and CIFAR-100, SwDPP achieves lossless training acceleration using only 70% of the data per epoch.","PeriodicalId":13154,"journal":{"name":"IEEE Signal Processing Letters","volume":"31 ","pages":"3034-3038"},"PeriodicalIF":3.2,"publicationDate":"2024-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142595116","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-21DOI: 10.1109/LSP.2024.3484328
Yaqi Chen;Tong Niu;Hao Zhang;Wenlin Zhang;Dan Qu
Recent advancements in large-scale pre-trained automatic speech recognition (ASR) foundation models (e.g., Whisper) have exhibited remarkable performance in speech processing tasks. A recently emerging paradigm, prompt tuning, offers a parameter-efficient approach for fine-tuning, which has proven to be effective in enhancing the adaptation of pre-trained models to downstream tasks. In this paper, we first explore the prompting method for low-resource speech recognition based on Whisper. Although effective, it poses a challenge in the few-shot scenario due to its high sensitivity to initialization. To address this problem, we propose a novel meta-prompt for low-resource speech recognition that leverages the benefits of meta-learning for fast learning. Moreover, we further present a lightweight version of meta-prompt that omits the learning of encoder-prompt, reducing computational and storage costs. Extensive experiments on FLEURS datasets demonstrate consistent improvements across eleven target languages, showing better generalizability. Notably, meta-prompt achieves similar performance with a 20%-shot compared to prompt tuning with a 50%-shot setting, suggesting excellent few-shot learning ability.
{"title":"Meta-Prompt: Boosting Whisper's Performance in Low-Resource Speech Recognition","authors":"Yaqi Chen;Tong Niu;Hao Zhang;Wenlin Zhang;Dan Qu","doi":"10.1109/LSP.2024.3484328","DOIUrl":"https://doi.org/10.1109/LSP.2024.3484328","url":null,"abstract":"Recent advancements in large-scale pre-trained automatic speech recognition (ASR) foundation models (e.g., Whisper) have exhibited remarkable performance in speech processing tasks. A recently emerging paradigm, prompt tuning, offers a parameter-efficient approach for fine-tuning, which has proven to be effective in enhancing the adaptation of pre-trained models to downstream tasks. In this paper, we first explore the prompting method for low-resource speech recognition based on Whisper. Although effective, it poses a challenge in the few-shot scenario due to its high sensitivity to initialization. To address this problem, we propose a novel meta-prompt for low-resource speech recognition that leverages the benefits of meta-learning for fast learning. Moreover, we further present a lightweight version of meta-prompt that omits the learning of encoder-prompt, reducing computational and storage costs. Extensive experiments on FLEURS datasets demonstrate consistent improvements across eleven target languages, showing better generalizability. Notably, meta-prompt achieves similar performance with a 20%-shot compared to prompt tuning with a 50%-shot setting, suggesting excellent few-shot learning ability.","PeriodicalId":13154,"journal":{"name":"IEEE Signal Processing Letters","volume":"31 ","pages":"3039-3043"},"PeriodicalIF":3.2,"publicationDate":"2024-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142595034","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：