Data augmentation methods have been employed to address the deficiencies in dysarthric speech datasets, achieving state-of-the-art (SOTA) results in the Dysarthric Speech Recognition (DSR) task. Current research on Dysarthric Speech Synthesis (DSS), however, fails to focus on the encoding of pathological features in dysarthric speech. The dysarthric speech is characterized by its discontinuous pronunciation, uncontrolled volume, slow speech, and excessive nasal sounds. Moreover, compared with typical speech, the dysarthric speech contains more non-stationary components generated by the explosive pronunciation, hoarseness, and air-flow noise during the pronunciation. We propose a DSS model named the Long-range and Non-stationary Variational Autoencoder (LNVAE). The LNVAE estimates the acoustic parameters of dysarthric speech by encoding the long-range dependency duration of phonemes in frame-level representations of dysarthric speech. Moreover, the LNVAE employs the Gaussian noise perturbation within the latent variables to capture the non-stationary fluctuations in dysarthric speech. The experiments were conducted on the speech synthesis and recognition tasks using the CDSD Chinese and UASpeech English corpora. The dysarthric speech synthesized by the LNVAE achieved the best performance across 29 and 28 objective metrics in the Chinese and English datasets, respectively. The synthesized speech also received the highest score from speech rehabilitation experts in the MOS experiments. The Whisper model fine-tuned on the synthesized data, achieved the lowest CER on the Chinese CDSD dataset. Moreover, for the UASpeech dataset, we increased the data by 0.5 times to fine-tune the DSR model, yet surpassed the current SOTA method, which uses four times more augmentation data, by 4.52$%$.
{"title":"Long-Range and Non-Stationary Encoding for Dysarthric Speech Data Augmentation","authors":"Daipeng Zhang;Hongcheng Zhang;Wenhuan Lu;Wei Li;Jinghong Wang;Jianguo Wei","doi":"10.1109/JSTSP.2025.3562417","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3562417","url":null,"abstract":"Data augmentation methods have been employed to address the deficiencies in dysarthric speech datasets, achieving state-of-the-art (SOTA) results in the Dysarthric Speech Recognition (DSR) task. Current research on Dysarthric Speech Synthesis (DSS), however, fails to focus on the encoding of pathological features in dysarthric speech. The dysarthric speech is characterized by its discontinuous pronunciation, uncontrolled volume, slow speech, and excessive nasal sounds. Moreover, compared with typical speech, the dysarthric speech contains more non-stationary components generated by the explosive pronunciation, hoarseness, and air-flow noise during the pronunciation. We propose a DSS model named the Long-range and Non-stationary Variational Autoencoder (LNVAE). The LNVAE estimates the acoustic parameters of dysarthric speech by encoding the long-range dependency duration of phonemes in frame-level representations of dysarthric speech. Moreover, the LNVAE employs the Gaussian noise perturbation within the latent variables to capture the non-stationary fluctuations in dysarthric speech. The experiments were conducted on the speech synthesis and recognition tasks using the CDSD Chinese and UASpeech English corpora. The dysarthric speech synthesized by the LNVAE achieved the best performance across 29 and 28 objective metrics in the Chinese and English datasets, respectively. The synthesized speech also received the highest score from speech rehabilitation experts in the MOS experiments. The Whisper model fine-tuned on the synthesized data, achieved the lowest CER on the Chinese CDSD dataset. Moreover, for the UASpeech dataset, we increased the data by 0.5 times to fine-tune the DSR model, yet surpassed the current SOTA method, which uses four times more augmentation data, by 4.52<inline-formula> <tex-math>$%$</tex-math></inline-formula>.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 5","pages":"767-782"},"PeriodicalIF":13.7,"publicationDate":"2025-04-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145659215","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-04-15DOI: 10.1109/JSTSP.2025.3560513
Wenxuan Wu;Xueyuan Chen;Shuai Wang;Jiadong Wang;Lingwei Meng;Xixin Wu;Helen Meng;Haizhou Li
Audio-Visual Target Speaker Extraction (AV-TSE) aims to mimic the human ability to enhance auditory perception using visual cues. Although numerous models have been proposed recently, most of them estimate target signals by primarily relying on local dependencies within acoustic features, underutilizing the human-like capacity to infer unclear parts of speech through contextual information. This limitation results in not only suboptimal performance but also inconsistent extraction quality across the utterance, with some segments exhibiting poor quality or inadequate suppression of interfering speakers. To close this gap, we propose a model-agnostic strategy called the Mask-And-Recover (MAR). It integrates both inter- and intra-modality contextual correlations to enable global inference within extraction modules. Additionally, to better target challenging parts within each sample, we introduce a Fine-grained Confidence Score (FCS) model to assess extraction quality and guide extraction modules to emphasize improvement on low-quality segments. To validate the effectiveness of our proposed model-agnostic training paradigm, six popular AV-TSE backbones were adopted for evaluation on the VoxCeleb2 dataset, demonstrating consistent performance improvements across various metrics.
{"title":"$C^{2}$AV-TSE: Context and Confidence-Aware Audio Visual Target Speaker Extraction","authors":"Wenxuan Wu;Xueyuan Chen;Shuai Wang;Jiadong Wang;Lingwei Meng;Xixin Wu;Helen Meng;Haizhou Li","doi":"10.1109/JSTSP.2025.3560513","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3560513","url":null,"abstract":"Audio-Visual Target Speaker Extraction (AV-TSE) aims to mimic the human ability to enhance auditory perception using visual cues. Although numerous models have been proposed recently, most of them estimate target signals by primarily relying on local dependencies within acoustic features, underutilizing the human-like capacity to infer unclear parts of speech through contextual information. This limitation results in not only suboptimal performance but also inconsistent extraction quality across the utterance, with some segments exhibiting poor quality or inadequate suppression of interfering speakers. To close this gap, we propose a model-agnostic strategy called the Mask-And-Recover (MAR). It integrates both inter- and intra-modality contextual correlations to enable global inference within extraction modules. Additionally, to better target challenging parts within each sample, we introduce a Fine-grained Confidence Score (FCS) model to assess extraction quality and guide extraction modules to emphasize improvement on low-quality segments. To validate the effectiveness of our proposed model-agnostic training paradigm, six popular AV-TSE backbones were adopted for evaluation on the VoxCeleb2 dataset, demonstrating consistent performance improvements across various metrics.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 4","pages":"646-657"},"PeriodicalIF":8.7,"publicationDate":"2025-04-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144502887","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-04-10DOI: 10.1109/JSTSP.2025.3559763
Hang Chen;Chen-Yue Zhang;Qing Wang;Jun Du;Sabato Marco Siniscalchi;Shi-Fu Xiong;Gen-Shun Wan
To advance audio-visual speech enhancement (AVSE) research in low-quality video settings, we introduce the multimodal information-based speech processing-low quality video (MISP-LQV) benchmark, which includes a 120-hour real-world Mandarin audio-visual dataset, two video degradation simulation methods, and benchmark results from several well-known AVSE models. We also propose a novel hybrid pixel and contour network (HPCNet), incorporating a lip reconstruction and distillation (LRD) module and a contour graph convolution (CGConv) layer. Specifically, the LRD module reconstructs high-quality lip frames from low-quality audio-visual data, utilizing knowledge distillation from a teacher model trained on high-quality data. The CGConv layer employs spatio-temporal and semantic-contextual graphs to capture complex relationships among lip landmark points. Extensive experiments on the MISP-LQV benchmark reveal the performance degradation caused by low-quality video across various AVSE models. Notably, including real/simulated low-quality videos in AVSE training enhances its robustness to low-quality videos but degrades the performance of high-quality videos.The proposed HPCNet demonstrates strong robustness against video quality degradation, which can be attributed to (1) the reconstructed lip frames closely aligning with high-quality frames and (2) the contour features exhibiting consistency across different video quality levels. The generalizability of HPCNet also has been validated through experiments on the 2nd COG-MHEAR AVSE Challenge dataset.
{"title":"HPCNet: Hybrid Pixel and Contour Network for Audio-Visual Speech Enhancement With Low-Quality Video","authors":"Hang Chen;Chen-Yue Zhang;Qing Wang;Jun Du;Sabato Marco Siniscalchi;Shi-Fu Xiong;Gen-Shun Wan","doi":"10.1109/JSTSP.2025.3559763","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3559763","url":null,"abstract":"To advance audio-visual speech enhancement (AVSE) research in low-quality video settings, we introduce the multimodal information-based speech processing-low quality video (MISP-LQV) benchmark, which includes a 120-hour real-world Mandarin audio-visual dataset, two video degradation simulation methods, and benchmark results from several well-known AVSE models. We also propose a novel hybrid pixel and contour network (HPCNet), incorporating a lip reconstruction and distillation (LRD) module and a contour graph convolution (CGConv) layer. Specifically, the LRD module reconstructs high-quality lip frames from low-quality audio-visual data, utilizing knowledge distillation from a teacher model trained on high-quality data. The CGConv layer employs spatio-temporal and semantic-contextual graphs to capture complex relationships among lip landmark points. Extensive experiments on the MISP-LQV benchmark reveal the performance degradation caused by low-quality video across various AVSE models. Notably, including real/simulated low-quality videos in AVSE training enhances its robustness to low-quality videos but degrades the performance of high-quality videos.The proposed HPCNet demonstrates strong robustness against video quality degradation, which can be attributed to (1) the reconstructed lip frames closely aligning with high-quality frames and (2) the contour features exhibiting consistency across different video quality levels. The generalizability of HPCNet also has been validated through experiments on the 2nd COG-MHEAR AVSE Challenge dataset.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 4","pages":"671-684"},"PeriodicalIF":8.7,"publicationDate":"2025-04-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144502911","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-04-07DOI: 10.1109/JSTSP.2025.3558653
Qingtian Xu;Jie Zhang;Zhenhua Ling
Brain-assisted speech enhancement (BASE) that utilizes electroencephalogram (EEG) signals as an assistive modality has shown a great potential for extracting the target speaker in multi-talker conditions. This is feasible as the EEG measurements contain the auditory attention of hearing-impaired listeners that can be leveraged to classify the target identity. Considering that an EEG cap with sparse channels exhibits multiple benefits and in practice many electrodes might contribute marginally, the EEG channel selection for BASE is desired. This problem was tackled in a subject-invariant manner in literature, the resulting BASE performance varies significantly across subjects. In this work, we therefore propose an input-independent subject-adaptive channel selection method for BASE, called subject-adaptive convolutional regularization selection (SA-ConvRS), which enables a personalized informative channel distribution. We observe the abnormal over memory phenomenon that facilitates the model to perform BASE without any brain signals, which often occurs in related fields due to the data recording and validation conditions. To remove this effect, we further design a task-based multi-process adversarial training (TMAT) approach by exploiting pseudo-EEG inputs. Experimental results on a public dataset show that the proposed SA-ConvRS can achieve subject-adaptive channel selections and keep the BASE performance close to the full-channel upper bound; the TMAT can avoid the over memory problem without sacrificing the performance of SA-ConvRS.
{"title":"Input-Independent Subject-Adaptive Channel Selection for Brain-Assisted Speech Enhancement","authors":"Qingtian Xu;Jie Zhang;Zhenhua Ling","doi":"10.1109/JSTSP.2025.3558653","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3558653","url":null,"abstract":"Brain-assisted speech enhancement (BASE) that utilizes electroencephalogram (EEG) signals as an assistive modality has shown a great potential for extracting the target speaker in multi-talker conditions. This is feasible as the EEG measurements contain the auditory attention of hearing-impaired listeners that can be leveraged to classify the target identity. Considering that an EEG cap with sparse channels exhibits multiple benefits and in practice many electrodes might contribute marginally, the EEG channel selection for BASE is desired. This problem was tackled in a subject-invariant manner in literature, the resulting BASE performance varies significantly across subjects. In this work, we therefore propose an input-independent subject-adaptive channel selection method for BASE, called subject-adaptive convolutional regularization selection (SA-ConvRS), which enables a personalized informative channel distribution. We observe the abnormal <italic>over memory</i> phenomenon that facilitates the model to perform BASE without any brain signals, which often occurs in related fields due to the data recording and validation conditions. To remove this effect, we further design a task-based multi-process adversarial training (TMAT) approach by exploiting pseudo-EEG inputs. Experimental results on a public dataset show that the proposed SA-ConvRS can achieve subject-adaptive channel selections and keep the BASE performance close to the full-channel upper bound; the TMAT can avoid the over memory problem without sacrificing the performance of SA-ConvRS.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 4","pages":"658-670"},"PeriodicalIF":8.7,"publicationDate":"2025-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144502947","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Recent advances in Synthetic Aperture Radar (SAR) sensors and innovative advanced imagery techniques have enabled SAR systems to acquire very high-resolution images with wide swaths, large bandwidth and in multiple polarization channels. The improvements of the SAR system capabilities also imply a significant increase in SAR data acquisition rates, such that efficient and effective compression methods become necessary. The compression of SAR raw data plays a crucial role in addressing the challenges posed by downlink and memory limitations onboard the SAR satellites and directly affects the quality of the generated SAR image. Neural data compression techniques using deep models have attracted many interests for natural image compression tasks and demonstrated promising results. In this study, neural data compression is extended into the complex domain to develop a Complex-Valued (CV) autoencoder-based data compression for SAR raw data. To this end, the basic fundamentals of data compression and Rate-Distortion (RD) theory are reviewed, well known data compression methods, Block Adaptive Quantization (BAQ) and JPEG2000 methods, are implemented and tested for SAR raw data compression, and a neural data compression based on CV autoencoders is developed for SAR raw data. Furthermore, since the available Sentinel-1 SAR raw products are already compressed with Flexible Dynamic BAQ (FDBAQ), an adaptation procedure applied to the decoded SAR raw data to generate SAR raw data with quasi-uniform quantization that resemble the statistics of the uncompressed SAR raw data onboard the satellites.
{"title":"Complex-Valued Autoencoder-Based Neural Data Compression for SAR Raw Data","authors":"Reza Mohammadi Asiyabi;Mihai Datcu;Andrei Anghel;Adrian Focsa;Michele Martone;Paola Rizzoli;Ernesto Imbembo","doi":"10.1109/JSTSP.2025.3558651","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3558651","url":null,"abstract":"Recent advances in Synthetic Aperture Radar (SAR) sensors and innovative advanced imagery techniques have enabled SAR systems to acquire very high-resolution images with wide swaths, large bandwidth and in multiple polarization channels. The improvements of the SAR system capabilities also imply a significant increase in SAR data acquisition rates, such that efficient and effective compression methods become necessary. The compression of SAR raw data plays a crucial role in addressing the challenges posed by downlink and memory limitations onboard the SAR satellites and directly affects the quality of the generated SAR image. Neural data compression techniques using deep models have attracted many interests for natural image compression tasks and demonstrated promising results. In this study, neural data compression is extended into the complex domain to develop a Complex-Valued (CV) autoencoder-based data compression for SAR raw data. To this end, the basic fundamentals of data compression and Rate-Distortion (RD) theory are reviewed, well known data compression methods, Block Adaptive Quantization (BAQ) and JPEG2000 methods, are implemented and tested for SAR raw data compression, and a neural data compression based on CV autoencoders is developed for SAR raw data. Furthermore, since the available Sentinel-1 SAR raw products are already compressed with Flexible Dynamic BAQ (FDBAQ), an adaptation procedure applied to the decoded SAR raw data to generate SAR raw data with quasi-uniform quantization that resemble the statistics of the uncompressed SAR raw data onboard the satellites.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 3","pages":"572-582"},"PeriodicalIF":8.7,"publicationDate":"2025-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10955162","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144073335","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-04-07DOI: 10.1109/JSTSP.2025.3558654
Xinyuan Qian;Jiaran Gao;Yaodan Zhang;Qiquan Zhang;Hexin Liu;Leibny Paola Garcia Perera;Haizhou Li
Speech enhancement plays an essential role in various applications, and the integration of visual information has been demonstrated to bring substantial advantages. However, the majority of current research concentrates on the examination of facial and lip movements, which can be compromised or entirely inaccessible in scenarios where occlusions occur or when the camera view is distant. Whereas contextual visual cues from the surrounding environment have been overlooked: for example, when we see a dog bark, our brain has the innate ability to discern and filter out the barking noise. To this end, in this paper, we introduce a novel task, i.e. Scene-aware Audio-Visual Speech Enhancement (SAV-SE). To our best knowledge, this is the first proposal to use rich contextual information from synchronized video as auxiliary cues to indicate the type of noise, which eventually improves the speech enhancement performance. Specifically, we propose the VC-S $^{2}$ E method, which incorporates the Conformer and Mamba modules for their complementary strengths. Extensive experiments are conducted on public MUSIC, AVSpeech and AudioSet datasets, where the results demonstrate the superiority of VC-S $^{2}$ E over other competitive methods.
{"title":"SAV-SE: Scene-Aware Audio-Visual Speech Enhancement With Selective State Space Model","authors":"Xinyuan Qian;Jiaran Gao;Yaodan Zhang;Qiquan Zhang;Hexin Liu;Leibny Paola Garcia Perera;Haizhou Li","doi":"10.1109/JSTSP.2025.3558654","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3558654","url":null,"abstract":"Speech enhancement plays an essential role in various applications, and the integration of visual information has been demonstrated to bring substantial advantages. However, the majority of current research concentrates on the examination of facial and lip movements, which can be compromised or entirely inaccessible in scenarios where occlusions occur or when the camera view is distant. Whereas contextual visual cues from the surrounding environment have been overlooked: for example, when we see a dog bark, our brain has the innate ability to discern and filter out the barking noise. To this end, in this paper, we introduce a novel task, i.e. Scene-aware Audio-Visual Speech Enhancement (SAV-SE). To our best knowledge, this is the first proposal to use rich contextual information from synchronized video as auxiliary cues to indicate the type of noise, which eventually improves the speech enhancement performance. Specifically, we propose the VC-S <inline-formula><tex-math>$^{2}$</tex-math></inline-formula> E method, which incorporates the Conformer and Mamba modules for their complementary strengths. Extensive experiments are conducted on public MUSIC, AVSpeech and AudioSet datasets, where the results demonstrate the superiority of VC-S <inline-formula><tex-math>$^{2}$</tex-math></inline-formula> E over other competitive methods.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 4","pages":"623-634"},"PeriodicalIF":8.7,"publicationDate":"2025-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144502891","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-04-01DOI: 10.1109/JSTSP.2025.3554136
Kevin Arias;Pablo Gomez;Carlos Hinojosa;Juan Carlos Niebles;Henry Arguello
Due to the advancements in deep image generation models, ensuring digital image authenticity, integrity, and confidentiality becomes challenging. While many active image manipulation detection methods embed digital signatures post-image acquisition, the vulnerabilities persist if unauthorized access occurs before this embedding or the embedding software is compromised. This work introduces an optics-based active image manipulation detection approach that learns the structure of a color-coded aperture (CCA), which encodes the light within the camera and embeds a highly reliable and imperceptible optical signature before image acquisition. We optimize our camera model with our proposed image manipulation detection network via end-to-end training. We validate our approach with extensive simulations and a proof-of-concept optical system. The results show that our method outperforms the state-of-the-art active image manipulation detection techniques.
{"title":"Protecting Images From Manipulations With Deep Optical Signatures","authors":"Kevin Arias;Pablo Gomez;Carlos Hinojosa;Juan Carlos Niebles;Henry Arguello","doi":"10.1109/JSTSP.2025.3554136","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3554136","url":null,"abstract":"Due to the advancements in deep image generation models, ensuring digital image authenticity, integrity, and confidentiality becomes challenging. While many active image manipulation detection methods embed digital signatures post-image acquisition, the vulnerabilities persist if unauthorized access occurs before this embedding or the embedding software is compromised. This work introduces an optics-based active image manipulation detection approach that learns the structure of a color-coded aperture (CCA), which encodes the light within the camera and embeds a highly reliable and imperceptible optical signature before image acquisition. We optimize our camera model with our proposed image manipulation detection network via end-to-end training. We validate our approach with extensive simulations and a proof-of-concept optical system. The results show that our method outperforms the state-of-the-art active image manipulation detection techniques.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 3","pages":"549-558"},"PeriodicalIF":8.7,"publicationDate":"2025-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144073333","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-03-31DOI: 10.1109/JSTSP.2025.3555067
Manuel Castillo-Cara;Jesus Martínez-Gómez;Javier Ballesteros-Jerez;Ismael García-Varea;Raúl García-Castro;Luis Orozco-Barbosa
Indoor localization determines an object's position within enclosed spaces, with applications in navigation, asset tracking, robotics, and context-aware computing. Technologies range from WiFi and Bluetooth to advanced systems like Massive Multiple Input-Multiple Output (MIMO). MIMO, initially designed to enhance wireless communication, is now key in indoor positioning due to its spatial diversity and multipath propagation. This study integrates MIMO-based indoor localization with Hybrid Neural Networks (HyNN), converting structured datasets into synthetic images using TINTO. This research marks the first application of HyNNs using synthetic images for MIMO-based indoor localization. Our key contributions include: (i) adapting TINTO for regression problems; (ii) using synthetic images as input data for our model; (iii) designing a novel HyNN with a Convolutional Neural Network branch for synthetic images and an MultiLayer Percetron branch for tidy data; and (iv) demonstrating improved results and metrics compared to prior literature. These advancements highlight the potential of HyNNs in enhancing the accuracy and efficiency of indoor localization systems.
{"title":"MIMO-Based Indoor Localisation With Hybrid Neural Networks: Leveraging Synthetic Images From Tidy Data for Enhanced Deep Learning","authors":"Manuel Castillo-Cara;Jesus Martínez-Gómez;Javier Ballesteros-Jerez;Ismael García-Varea;Raúl García-Castro;Luis Orozco-Barbosa","doi":"10.1109/JSTSP.2025.3555067","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3555067","url":null,"abstract":"Indoor localization determines an object's position within enclosed spaces, with applications in navigation, asset tracking, robotics, and context-aware computing. Technologies range from WiFi and Bluetooth to advanced systems like Massive Multiple Input-Multiple Output (MIMO). MIMO, initially designed to enhance wireless communication, is now key in indoor positioning due to its spatial diversity and multipath propagation. This study integrates MIMO-based indoor localization with Hybrid Neural Networks (HyNN), converting structured datasets into synthetic images using TINTO. This research marks the first application of HyNNs using synthetic images for MIMO-based indoor localization. Our key contributions include: (i) adapting TINTO for regression problems; (ii) using synthetic images as input data for our model; (iii) designing a novel HyNN with a Convolutional Neural Network branch for synthetic images and an MultiLayer Percetron branch for tidy data; and (iv) demonstrating improved results and metrics compared to prior literature. These advancements highlight the potential of HyNNs in enhancing the accuracy and efficiency of indoor localization systems.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 3","pages":"559-571"},"PeriodicalIF":8.7,"publicationDate":"2025-03-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10946146","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144073334","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In daily life, we encounter a variety of sounds, both desirable and undesirable, with limited control over their presence and volume. Our work introduces “Listen, Chat, and Remix” (LCR), a novel multimodal sound remixer that controls each sound source in a mixture based on user-provided text instructions. LCR distinguishes itself with a user-friendly text interface and its unique ability to remix multiple sound sources simultaneously within a mixture, without needing to separate them. Users input open-vocabulary text prompts, which are interpreted by a large language model to create a semantic filter for remixing the sound mixture. The system then decomposes the mixture into its components, applies the semantic filter, and reassembles filtered components back to the desired output. We developed a 160-hour dataset with over 100 k mixtures, including speech and various audio sources, along with text prompts for diverse remixing tasks including extraction, removal, and volume control of single or multiple sources. Our experiments demonstrate significant improvements in signal quality across all remixing tasks and robust performance in zero-shot scenarios with varying numbers and types of sound sources.
{"title":"Listen, Chat, and Remix: Text-Guided Soundscape Remixing for Enhanced Auditory Experience","authors":"Xilin Jiang;Cong Han;Yinghao Aaron Li;Nima Mesgarani","doi":"10.1109/JSTSP.2025.3570103","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3570103","url":null,"abstract":"In daily life, we encounter a variety of sounds, both desirable and undesirable, with limited control over their presence and volume. Our work introduces “Listen, Chat, and Remix” (LCR), a novel multimodal sound remixer that controls each sound source in a mixture based on user-provided text instructions. LCR distinguishes itself with a user-friendly text interface and its unique ability to remix multiple sound sources simultaneously within a mixture, without needing to separate them. Users input open-vocabulary text prompts, which are interpreted by a large language model to create a semantic filter for remixing the sound mixture. The system then decomposes the mixture into its components, applies the semantic filter, and reassembles filtered components back to the desired output. We developed a 160-hour dataset with over 100 k mixtures, including speech and various audio sources, along with text prompts for diverse remixing tasks including extraction, removal, and volume control of single or multiple sources. Our experiments demonstrate significant improvements in signal quality across all remixing tasks and robust performance in zero-shot scenarios with varying numbers and types of sound sources.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 4","pages":"635-645"},"PeriodicalIF":8.7,"publicationDate":"2025-03-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144502888","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-03-19DOI: 10.1109/JSTSP.2025.3552918
Dahu Wang;Chang Liu
Independent component analysis (ICA) is widely applied in remote sensing signal processing. Among various ICA algorithms, the modified semidefinite programming (MSDP) algorithm stands out. However, the efficacy and safety of MSDP depend on the distribution of data. Our research found that MSDP is better suited for handling data with a super-Gaussian distribution. As real-world data usually exhibit a combination of sub-Gaussian and super-Gaussian distributions, MSDP faces challenges in accurately extracting all independent components (ICs). To solve this problem, we conducted a comprehensive analysis of the MSDP algorithm and introduced an enhanced version, the sign-enhanced MSDP (SMSDP) algorithm. By incorporating the sign function into the projected Hessian matrix, SMSDP enables the algorithm to effectively extract ICs from data characterized by a mixture of sub-Gaussian and super-Gaussian distributions. Furthermore, we provided a detailed comparison with MSDP to illustrate why SMSDP can achieve more accurate eigenpairs. Some experiments have demonstrated the effectiveness of SMSDP. The experiments in blind separation of image/sound, radar clutter removal, and real hyperspectral feature extraction also show the superiority of SMSDP in improving the accuracy of IC extraction.
{"title":"Sign-Enhanced Semidefinite Programming Algorithm and its Application to Independent Component Analysis","authors":"Dahu Wang;Chang Liu","doi":"10.1109/JSTSP.2025.3552918","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3552918","url":null,"abstract":"Independent component analysis (ICA) is widely applied in remote sensing signal processing. Among various ICA algorithms, the modified semidefinite programming (MSDP) algorithm stands out. However, the efficacy and safety of MSDP depend on the distribution of data. Our research found that MSDP is better suited for handling data with a super-Gaussian distribution. As real-world data usually exhibit a combination of sub-Gaussian and super-Gaussian distributions, MSDP faces challenges in accurately extracting all independent components (ICs). To solve this problem, we conducted a comprehensive analysis of the MSDP algorithm and introduced an enhanced version, the sign-enhanced MSDP (SMSDP) algorithm. By incorporating the sign function into the projected Hessian matrix, SMSDP enables the algorithm to effectively extract ICs from data characterized by a mixture of sub-Gaussian and super-Gaussian distributions. Furthermore, we provided a detailed comparison with MSDP to illustrate why SMSDP can achieve more accurate eigenpairs. Some experiments have demonstrated the effectiveness of SMSDP. The experiments in blind separation of image/sound, radar clutter removal, and real hyperspectral feature extraction also show the superiority of SMSDP in improving the accuracy of IC extraction.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 3","pages":"536-548"},"PeriodicalIF":8.7,"publicationDate":"2025-03-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144073332","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：