Pub Date : 2025-04-21DOI: 10.1109/OJSP.2025.3562797
Nicholas Mehlman;Shri Narayanan
As deep learning models have proliferated, concerns about their reliability and security have also increased. One significant challenge is understanding adversarial perturbations, which can alter a model's predictions despite being very small in magnitude. Prior work has proposed that this phenomenon results from a fundamental deficit in supervised learning, by which classifiers exploit whatever input features are more predictive, regardless of whether or not these features are robust to adversarial attacks. In this paper, we consider feature robustness in the context of contrastive self-supervised learning methods that have become especially common in recent years. Our findings suggest that the features learned during self-supervision are, in fact, more resistant to adversarial perturbations than those generated from supervised learning. However, we also find that these self-supervised features exhibit poorer inter-class disentanglement, limiting their contribution to overall classifier robustness.
{"title":"Adversarial Robustness of Self-Supervised Learning Features","authors":"Nicholas Mehlman;Shri Narayanan","doi":"10.1109/OJSP.2025.3562797","DOIUrl":"https://doi.org/10.1109/OJSP.2025.3562797","url":null,"abstract":"As deep learning models have proliferated, concerns about their reliability and security have also increased. One significant challenge is understanding adversarial perturbations, which can alter a model's predictions despite being very small in magnitude. Prior work has proposed that this phenomenon results from a fundamental deficit in supervised learning, by which classifiers exploit whatever input features are more predictive, regardless of whether or not these features are robust to adversarial attacks. In this paper, we consider feature robustness in the context of contrastive self-supervised learning methods that have become especially common in recent years. Our findings suggest that the features learned during self-supervision are, in fact, more resistant to adversarial perturbations than those generated from supervised learning. However, we also find that these self-supervised features exhibit poorer inter-class disentanglement, limiting their contribution to overall classifier robustness.","PeriodicalId":73300,"journal":{"name":"IEEE open journal of signal processing","volume":"6 ","pages":"468-477"},"PeriodicalIF":2.9,"publicationDate":"2025-04-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10971198","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143908351","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-04-07DOI: 10.1109/OJSP.2025.3558686
Costas A. Kokke;Mario Coutino;Richard Heusdens;Geert Leus
Sparse array design is used to help reduce computational, hardware, and power requirements compared to uniform arrays while maintaining acceptable performance. Although minimizing the Cramér-Rao bound has been adopted previously for sparse sensing, it did not consider multiple targets and unknown target directions. To handle the unknown target directions when optimizing the Cramér-Rao bound, we propose to use the worst-case Cramér-Rao bound of two uncorrelated equal power sources with arbitrary angles. This new worst-case two-target Cramér-Rao bound metric has some resemblance to the peak sidelobe level metric which is commonly used in unknown multi-target scenarios. We cast the sensor selection problem for 3-D arrays using the worst-case two-target Cramér-Rao bound as a convex semi-definite program and obtain the binary selection by randomized rounding. We illustrate the proposed method through numerical examples, comparing it to solutions obtained by minimizing the single-target Cramér-Rao bound, minimizing the Cramér-Rao bound for known target angles, the concentric rectangular array and the boundary array. We show that our method selects a combination of edge and center elements, which contrasts with solutions obtained by minimizing the single-target Cramér-Rao bound. The proposed selections also exhibit lower peak sidelobe levels without the need for sidelobe level constraints.
{"title":"Array Design for Angle of Arrival Estimation Using the Worst-Case Two-Target Cramér-Rao Bound","authors":"Costas A. Kokke;Mario Coutino;Richard Heusdens;Geert Leus","doi":"10.1109/OJSP.2025.3558686","DOIUrl":"https://doi.org/10.1109/OJSP.2025.3558686","url":null,"abstract":"Sparse array design is used to help reduce computational, hardware, and power requirements compared to uniform arrays while maintaining acceptable performance. Although minimizing the Cramér-Rao bound has been adopted previously for sparse sensing, it did not consider multiple targets and unknown target directions. To handle the unknown target directions when optimizing the Cramér-Rao bound, we propose to use the worst-case Cramér-Rao bound of two uncorrelated equal power sources with arbitrary angles. This new worst-case two-target Cramér-Rao bound metric has some resemblance to the peak sidelobe level metric which is commonly used in unknown multi-target scenarios. We cast the sensor selection problem for 3-D arrays using the worst-case two-target Cramér-Rao bound as a convex semi-definite program and obtain the binary selection by randomized rounding. We illustrate the proposed method through numerical examples, comparing it to solutions obtained by minimizing the single-target Cramér-Rao bound, minimizing the Cramér-Rao bound for known target angles, the concentric rectangular array and the boundary array. We show that our method selects a combination of edge and center elements, which contrasts with solutions obtained by minimizing the single-target Cramér-Rao bound. The proposed selections also exhibit lower peak sidelobe levels without the need for sidelobe level constraints.","PeriodicalId":73300,"journal":{"name":"IEEE open journal of signal processing","volume":"6 ","pages":"453-467"},"PeriodicalIF":2.9,"publicationDate":"2025-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10955272","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143896495","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-04-02DOI: 10.1109/OJSP.2025.3557332
Erik G. Larsson;Nicolò Michelusi
The decentralized gradient descent (DGD) algorithm, and its sibling, diffusion, are workhorses in decentralized machine learning, distributed inference and estimation, and multi-agent coordination. We propose a novel, principled framework for the analysis of DGD and diffusion for strongly convex, smooth objectives, and arbitrary undirected topologies, using contraction mappings coupled with a result called the mean Hessian theorem (MHT). The use of these tools yields tight convergence bounds, both in the noise-free and noisy regimes. While these bounds are qualitatively similar to results found in the literature, our approach using contractions together with the MHT decouples the algorithm dynamics (how quickly the algorithm converges to its fixed point) from its asymptotic convergence properties (how far the fixed point is from the global optimum). This yields a simple, intuitive analysis that is accessible to a broader audience. Extensions are provided to multiple local gradient updates, time-varying step sizes, noisy gradients (stochastic DGD and diffusion), communication noise, and random topologies.
{"title":"Unified Analysis of Decentralized Gradient Descent: A Contraction Mapping Framework","authors":"Erik G. Larsson;Nicolò Michelusi","doi":"10.1109/OJSP.2025.3557332","DOIUrl":"https://doi.org/10.1109/OJSP.2025.3557332","url":null,"abstract":"The decentralized gradient descent (DGD) algorithm, and its sibling, diffusion, are workhorses in decentralized machine learning, distributed inference and estimation, and multi-agent coordination. We propose a novel, principled framework for the analysis of DGD and diffusion for strongly convex, smooth objectives, and arbitrary undirected topologies, using contraction mappings coupled with a result called the mean Hessian theorem (MHT). The use of these tools yields tight convergence bounds, both in the noise-free and noisy regimes. While these bounds are qualitatively similar to results found in the literature, our approach using contractions together with the MHT decouples the algorithm dynamics (how quickly the algorithm converges to its fixed point) from its asymptotic convergence properties (how far the fixed point is from the global optimum). This yields a simple, intuitive analysis that is accessible to a broader audience. Extensions are provided to multiple local gradient updates, time-varying step sizes, noisy gradients (stochastic DGD and diffusion), communication noise, and random topologies.","PeriodicalId":73300,"journal":{"name":"IEEE open journal of signal processing","volume":"6 ","pages":"507-529"},"PeriodicalIF":2.9,"publicationDate":"2025-04-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10947567","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144117149","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Estimating time-varying signals becomes particularly challenging in the face of non-Gaussian (e.g., sparse) and/or rapidly time-varying process noise. By building upon the recent progress in the approximate message passing (AMP) paradigm, this paper unifies the vector variant of AMP (i.e., VAMP) with the Kalman filter (KF) into a unified message passing framework. The new algorithm (coined VAMP-KF) does not restrict the process noise to a specific structure (e.g., same support over time), thereby accounting for non-Gaussian process noise sources that are uncorrelated both component-wise and over time. For the sake of theoretical performance prediction, we conduct a state evolution (SE) analysis of the proposed algorithm and show its consistency with the asymptotic empirical mean-squared error (MSE). Numerical results using sparse noise dynamics with different sparsity ratios demonstrate unambiguously the effectiveness of the proposed VAMP-KF algorithm and its superiority over state-of-the-art algorithms both in terms of reconstruction accuracy and computational complexity.
{"title":"VAMP-Based Kalman Filtering Under Non-Gaussian Process Noise","authors":"Tiancheng Gao;Mohamed Akrout;Faouzi Bellili;Amine Mezghani","doi":"10.1109/OJSP.2025.3557271","DOIUrl":"https://doi.org/10.1109/OJSP.2025.3557271","url":null,"abstract":"Estimating time-varying signals becomes particularly challenging in the face of non-Gaussian (e.g., sparse) and/or rapidly time-varying process noise. By building upon the recent progress in the approximate message passing (AMP) paradigm, this paper unifies the vector variant of AMP (i.e., VAMP) with the Kalman filter (KF) into a unified message passing framework. The new algorithm (coined VAMP-KF) does not restrict the process noise to a specific structure (e.g., same support over time), thereby accounting for non-Gaussian process noise sources that are uncorrelated both component-wise and over time. For the sake of theoretical performance prediction, we conduct a state evolution (SE) analysis of the proposed algorithm and show its consistency with the asymptotic empirical mean-squared error (MSE). Numerical results using sparse noise dynamics with different sparsity ratios demonstrate unambiguously the effectiveness of the proposed VAMP-KF algorithm and its superiority over state-of-the-art algorithms both in terms of reconstruction accuracy and computational complexity.","PeriodicalId":73300,"journal":{"name":"IEEE open journal of signal processing","volume":"6 ","pages":"434-452"},"PeriodicalIF":2.9,"publicationDate":"2025-04-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10947573","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143908383","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Semantic segmentation of remote sensing images is extensively used in crop cover and type analysis, and environmental monitoring. In the semantic segmentation of remote sensing images, owning to the specificity of remote sensing images, not only the local context is required, but also the global context information makes an important role in it. Inspired by the powerful global modelling capability of Swin Transformer, we propose the LSENet network, which follows the encoder-decoder architecture of the UNet network. In encoding phase, we propose spatial enhancement module (SEM), which helps Swin Transformer further enhance feature extraction by encoding spatial information. In decoding stage, we propose local enhancement module (LEM), which is embedded in the Swin Transformer to improve the Swin Transformer to assist the network to obtain more local semantic information so as to classify pixels more accurately, especially in the edge region, the adding of LEM enables to obtain smoother edges. The experimental results on the Vaihingen and Potsdam datasets demonstrate the effectiveness of our proposed method. Specifically, the mIoU metric is 78.58% on the Potsdam dataset, 72.59% on the Vaihingen dataset and 64.49% on the OpenEarthMap dataset.
{"title":"Swin Transformer With Spatial and Local Context Augmentation for Enhanced Semantic Segmentation of Remote Sensing Images","authors":"Rong-Xing Ding;Yi-Han Xu;Gang Yu;Wen Zhou;Ding Zhou","doi":"10.1109/OJSP.2025.3573202","DOIUrl":"https://doi.org/10.1109/OJSP.2025.3573202","url":null,"abstract":"Semantic segmentation of remote sensing images is extensively used in crop cover and type analysis, and environmental monitoring. In the semantic segmentation of remote sensing images, owning to the specificity of remote sensing images, not only the local context is required, but also the global context information makes an important role in it. Inspired by the powerful global modelling capability of Swin Transformer, we propose the LSENet network, which follows the encoder-decoder architecture of the UNet network. In encoding phase, we propose spatial enhancement module (SEM), which helps Swin Transformer further enhance feature extraction by encoding spatial information. In decoding stage, we propose local enhancement module (LEM), which is embedded in the Swin Transformer to improve the Swin Transformer to assist the network to obtain more local semantic information so as to classify pixels more accurately, especially in the edge region, the adding of LEM enables to obtain smoother edges. The experimental results on the Vaihingen and Potsdam datasets demonstrate the effectiveness of our proposed method. Specifically, the mIoU metric is 78.58% on the Potsdam dataset, 72.59% on the Vaihingen dataset and 64.49% on the OpenEarthMap dataset.","PeriodicalId":73300,"journal":{"name":"IEEE open journal of signal processing","volume":"6 ","pages":"608-620"},"PeriodicalIF":2.9,"publicationDate":"2025-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=11011931","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144299229","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-03-23DOI: 10.1109/OJSP.2025.3572759
Mazen Abdelfattah;Z. Jane Wang;Rabab Ward
Safe navigation of autonomous vehicles requires accurate and rapid understanding of their dynamic 3D environment. Scene flow estimation models this dynamic environment by predicting point motion between sequential point cloud scans, and is crucial for safe navigation. Existing state-of-the-art scene flow estimation methods, based on test-time optimization, achieve high accuracy but suffer from significant latency, limiting their applicability in real-time onboard systems. This latency stems from both the iterative test-time optimization process and the inherent delay of waiting for the LiDAR to acquire a complete $360^circ$ scan. To overcome this bottleneck, we introduce a novel streaming scene flow framework leveraging the sequential nature of LiDAR slice acquisition, demonstrating a dramatic reduction in end-to-end latency. Instead of waiting for the full $360^circ$ scan, our method immediately estimates scene flow using each LiDAR slice once it is captured. To mitigate the reduced context of individual slices, we propose a novel contextual augmentation technique that expands the target slice by a small angular margin, incorporating crucial slice boundary information. Furthermore, to enhance test-time optimization within our streaming framework, our novel initialization scheme ’warm-starts' the current optimization using optimized parameters from the preceding slice. This achieves substantial speedups while maintaining, and in some cases surpassing, full-scan accuracy. We rigorously evaluate our approach on the challenging Waymo and Argoverse datasets, demonstrating significant latency reduction without compromising scene flow quality. This work paves the way for deploying high-accuracy, real-time scene flow algorithms in autonomous driving, advancing the field towards more responsive and safer autonomous systems.
{"title":"Streaming LiDAR Scene Flow Estimation","authors":"Mazen Abdelfattah;Z. Jane Wang;Rabab Ward","doi":"10.1109/OJSP.2025.3572759","DOIUrl":"https://doi.org/10.1109/OJSP.2025.3572759","url":null,"abstract":"Safe navigation of autonomous vehicles requires accurate and rapid understanding of their dynamic 3D environment. Scene flow estimation models this dynamic environment by predicting point motion between sequential point cloud scans, and is crucial for safe navigation. Existing state-of-the-art scene flow estimation methods, based on test-time optimization, achieve high accuracy but suffer from significant latency, limiting their applicability in real-time onboard systems. This latency stems from both the iterative test-time optimization process and the inherent delay of waiting for the LiDAR to acquire a complete <inline-formula><tex-math>$360^circ$</tex-math></inline-formula> scan. To overcome this bottleneck, we introduce a novel <italic>streaming</i> scene flow framework leveraging the sequential nature of LiDAR slice acquisition, demonstrating a dramatic reduction in end-to-end latency. Instead of waiting for the full <inline-formula><tex-math>$360^circ$</tex-math></inline-formula> scan, our method immediately estimates scene flow using each LiDAR slice once it is captured. To mitigate the reduced context of individual slices, we propose a novel contextual augmentation technique that expands the target slice by a small angular margin, incorporating crucial slice boundary information. Furthermore, to enhance test-time optimization within our streaming framework, our novel initialization scheme ’warm-starts' the current optimization using optimized parameters from the preceding slice. This achieves substantial speedups while maintaining, and in some cases surpassing, full-scan accuracy. We rigorously evaluate our approach on the challenging Waymo and Argoverse datasets, demonstrating significant latency reduction without compromising scene flow quality. This work paves the way for deploying high-accuracy, real-time scene flow algorithms in autonomous driving, advancing the field towards more responsive and safer autonomous systems.","PeriodicalId":73300,"journal":{"name":"IEEE open journal of signal processing","volume":"6 ","pages":"590-598"},"PeriodicalIF":2.9,"publicationDate":"2025-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=11012710","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144243669","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-03-23DOI: 10.1109/OJSP.2025.3572820
Jeova Farias Sales Rocha Neto
Image Segmentation is one of the core tasks in Computer Vision, and solving it often depends on modeling the image appearance data via the color distributions of each of its constituent regions. Whereas many segmentation algorithms handle the appearance model dependence using alternation or implicit methods, we propose here a new approach to directly estimate them from the image without prior information on the underlying segmentation. Our method uses local high-order color statistics from the image as an input to a tensor factorization-based estimator for latent variable models. This approach is able to estimate models in multi-region images and automatically output the regions' proportions without prior user interaction, overcoming the drawbacks of a prior attempt to this problem. We also demonstrate the performance of our proposed method in many challenging synthetic and real imaging scenarios and show that it leads to an efficient segmentation algorithm.
{"title":"Appearance Estimation and Image Segmentation via Tensor Factorization","authors":"Jeova Farias Sales Rocha Neto","doi":"10.1109/OJSP.2025.3572820","DOIUrl":"https://doi.org/10.1109/OJSP.2025.3572820","url":null,"abstract":"Image Segmentation is one of the core tasks in Computer Vision, and solving it often depends on modeling the image appearance data via the color distributions of each of its constituent regions. Whereas many segmentation algorithms handle the appearance model dependence using alternation or implicit methods, we propose here a new approach to directly estimate them from the image without prior information on the underlying segmentation. Our method uses local high-order color statistics from the image as an input to a tensor factorization-based estimator for latent variable models. This approach is able to estimate models in multi-region images and automatically output the regions' proportions without prior user interaction, overcoming the drawbacks of a prior attempt to this problem. We also demonstrate the performance of our proposed method in many challenging synthetic and real imaging scenarios and show that it leads to an efficient segmentation algorithm.","PeriodicalId":73300,"journal":{"name":"IEEE open journal of signal processing","volume":"6 ","pages":"581-589"},"PeriodicalIF":2.9,"publicationDate":"2025-03-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=11011680","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144243738","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-03-22DOI: 10.1109/OJSP.2025.3572840
Thuc Nguyen Huu;Vinh Van Duong;Jonghoon Yim;Byeungwoo Jeon
This paper addresses the unique challenges of compressing raw plenoptic video, in which the inherent hexagonal micro-image layout and sparse distribution of motion vectors (MVs) often diminish the coding efficiency of conventional block-based motion compensation. To mitigate excessive overhead from motion vector difference (MVD) signaling, we use three specialized MV resolutions: a hexagonal-lattice (HL) alignment that matches the micro-image structure, an integer-pel resolution, and a quarter-pel resolution. We then develop a rate-distortion (RD)-optimized scheme that adaptively selects the most suitable MV resolution at the coding unit level. By integrating our approach into the Versatile Video Coding (VVC) framework, the proposed method reduces MVD bits significantly while preserving high prediction accuracy. Experiments using two comprehensive plenoptic camera datasets — lenslet 1.0 and lenslet 2.0 — demonstrate substantial gains over the VVC anchor, achieving average Bjontegaard–Delta rate savings of 5.90% and 1.80%, respectively. These results confirm that combining HL and conventional resolutions in an RD-optimized manner substantially enhances motion prediction efficiency for raw plenoptic video.
{"title":"Adaptive Motion Vector Resolutions in Raw Plenoptic Video Coding","authors":"Thuc Nguyen Huu;Vinh Van Duong;Jonghoon Yim;Byeungwoo Jeon","doi":"10.1109/OJSP.2025.3572840","DOIUrl":"https://doi.org/10.1109/OJSP.2025.3572840","url":null,"abstract":"This paper addresses the unique challenges of compressing raw plenoptic video, in which the inherent hexagonal micro-image layout and sparse distribution of motion vectors (MVs) often diminish the coding efficiency of conventional block-based motion compensation. To mitigate excessive overhead from motion vector difference (MVD) signaling, we use three specialized MV resolutions: a hexagonal-lattice (HL) alignment that matches the micro-image structure, an integer-pel resolution, and a quarter-pel resolution. We then develop a rate-distortion (RD)-optimized scheme that adaptively selects the most suitable MV resolution at the coding unit level. By integrating our approach into the Versatile Video Coding (VVC) framework, the proposed method reduces MVD bits significantly while preserving high prediction accuracy. Experiments using two comprehensive plenoptic camera datasets — lenslet 1.0 and lenslet 2.0 — demonstrate substantial gains over the VVC anchor, achieving average Bjontegaard–Delta rate savings of 5.90% and 1.80%, respectively. These results confirm that combining HL and conventional resolutions in an RD-optimized manner substantially enhances motion prediction efficiency for raw plenoptic video.","PeriodicalId":73300,"journal":{"name":"IEEE open journal of signal processing","volume":"6 ","pages":"917-925"},"PeriodicalIF":2.7,"publicationDate":"2025-03-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=11010129","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144758367","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We propose a novel snapshot hyperspectral imaging method that incorporates co-designed optics, a color filter array (CFA), and an unrolled post-processing network through end-to-end learning. The camera optics consists of a fixed refractive lens and a diffractive optical element (DOE). The learned DOE and CFA efficiently encode the hyperspectral data cube on the sensor via phase and amplitude modulation at the camera aperture and sensor planes, respectively. Subsequently, the unrolled network reconstructs the hyperspectral images from the sensor signal with high accuracy. We conduct extensive simulations to analyze and validate the performance of the proposed method for several CFA models and in non-ideal imaging conditions. We demonstrate that the Gaussian model is effective for parameterizing the spectral transmission functions of CFA pixels, providing high reconstruction accuracy and being relatively easy to implement. Furthermore, we show that learned CFA patterns are effective when optimally coupled with co-designed diffractive-refractive optics. We evaluate the robustness of our method against sensor noise and potential inaccuracies in the fabrication of the DOE and CFA. Our results show that our method achieves superior reconstruction quality compared to state-of-the-art methods, excelling in both spatial and spectral detail recovery and maintaining robustness against realistic noise levels.
{"title":"Snapshot Hyperspectral Imaging With Co-Designed Optics, Color Filter Array, and Unrolled Network","authors":"Ayoung Kim;Ugur Akpinar;Erdem Sahin;Atanas Gotchev","doi":"10.1109/OJSP.2025.3571675","DOIUrl":"https://doi.org/10.1109/OJSP.2025.3571675","url":null,"abstract":"We propose a novel snapshot hyperspectral imaging method that incorporates co-designed optics, a color filter array (CFA), and an unrolled post-processing network through end-to-end learning. The camera optics consists of a fixed refractive lens and a diffractive optical element (DOE). The learned DOE and CFA efficiently encode the hyperspectral data cube on the sensor via phase and amplitude modulation at the camera aperture and sensor planes, respectively. Subsequently, the unrolled network reconstructs the hyperspectral images from the sensor signal with high accuracy. We conduct extensive simulations to analyze and validate the performance of the proposed method for several CFA models and in non-ideal imaging conditions. We demonstrate that the Gaussian model is effective for parameterizing the spectral transmission functions of CFA pixels, providing high reconstruction accuracy and being relatively easy to implement. Furthermore, we show that learned CFA patterns are effective when optimally coupled with co-designed diffractive-refractive optics. We evaluate the robustness of our method against sensor noise and potential inaccuracies in the fabrication of the DOE and CFA. Our results show that our method achieves superior reconstruction quality compared to state-of-the-art methods, excelling in both spatial and spectral detail recovery and maintaining robustness against realistic noise levels.","PeriodicalId":73300,"journal":{"name":"IEEE open journal of signal processing","volume":"6 ","pages":"599-607"},"PeriodicalIF":2.9,"publicationDate":"2025-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=11008739","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144243739","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The increasing popularity of generative AI has led to a significant rise in deepfake content, creating an urgent need for generalized and reliable deepfake detection methods. Since existing approaches rely on either spatial-domain features or frequency-domain features, they struggle to generalize across unseen datasets, especially those with subtle manipulations. To address these challenges, a novel end-to-end Wavelet Central Difference Convolutional Vision Transformer framework is designed to enhance spatial-frequency deepfake detection. Unlike previous methods, this approach applies the Discrete Wavelet Transform for multi-level frequency decomposition and Central Difference Convolution to capture local fine-grained discrepancies and focus on texture variances, while also incorporating Vision Transformers for global contextual understanding. The Frequency-Spatial Feature Fusion Attention module integrates these features, enabling the effective detection of fake artifacts. Moreover, in contrast to earlier work, subtle perturbations to both spatial and frequency domains are introduced to further improve generalization. Generalization cross-dataset evaluations demonstrate that WaViT-CDC outperforms state-of-the-art methods, when trained on both low-quality and high-quality face images, achieving an average performance increase of 2.5% and 4.5% on challenging high-resolution, real-world datasets such as Celeb-DF and WildDeepfake.
{"title":"WaViT-CDC: Wavelet Vision Transformer With Central Difference Convolutions for Spatial-Frequency Deepfake Detection","authors":"Nour Eldin Alaa Badr;Jean-Christophe Nebel;Darrel Greenhill;Xing Liang","doi":"10.1109/OJSP.2025.3571679","DOIUrl":"https://doi.org/10.1109/OJSP.2025.3571679","url":null,"abstract":"The increasing popularity of generative AI has led to a significant rise in deepfake content, creating an urgent need for generalized and reliable deepfake detection methods. Since existing approaches rely on either spatial-domain features or frequency-domain features, they struggle to generalize across unseen datasets, especially those with subtle manipulations. To address these challenges, a novel end-to-end Wavelet Central Difference Convolutional Vision Transformer framework is designed to enhance spatial-frequency deepfake detection. Unlike previous methods, this approach applies the Discrete Wavelet Transform for multi-level frequency decomposition and Central Difference Convolution to capture local fine-grained discrepancies and focus on texture variances, while also incorporating Vision Transformers for global contextual understanding. The Frequency-Spatial Feature Fusion Attention module integrates these features, enabling the effective detection of fake artifacts. Moreover, in contrast to earlier work, subtle perturbations to both spatial and frequency domains are introduced to further improve generalization. Generalization cross-dataset evaluations demonstrate that WaViT-CDC outperforms state-of-the-art methods, when trained on both low-quality and high-quality face images, achieving an average performance increase of 2.5% and 4.5% on challenging high-resolution, real-world datasets such as Celeb-DF and WildDeepfake.","PeriodicalId":73300,"journal":{"name":"IEEE open journal of signal processing","volume":"6 ","pages":"621-630"},"PeriodicalIF":2.9,"publicationDate":"2025-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=11007485","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144299230","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：