We consider the problem of estimating the covariance matrix of a random signal observed through unknown translations (modeled by cyclic shifts) and corrupted by noise. Solving this problem allows to discover low-rank structures masked by the existence of translations (which act as nuisance parameters), with direct application to principal components analysis. We assume that the underlying signal is of length $L$ and follows a standard factor model with mean zero and $r$ normally distributed factors. To recover the covariance matrix in this case, we propose to employ the second- and fourth-order shift-invariant moments of the signal known as the power spectrum and the trispectrum. We prove that they are sufficient for recovering the covariance matrix (under a certain technical condition) when $r<sqrt{L}$ . Correspondingly, we provide a polynomial-time procedure for estimating the covariance matrix from many (translated and noisy) observations, where no explicit knowledge of $r$ is required, and prove the procedure's statistical consistency. While our results establish that covariance estimation is possible from the power spectrum and the trispectrum for low-rank covariance matrices, we prove that this is not the case for full-rank covariance matrices. We conduct numerical experiments that corroborate our theoretical findings and demonstrate the favourable performance of our algorithms in various settings, including in high levels of noise.
{"title":"Multi-reference factor analysis: low-rank covariance estimation under unknown translations","authors":"Boris Landa;Yoel Shkolnisky","doi":"10.1093/imaiai/iaaa019","DOIUrl":"https://doi.org/10.1093/imaiai/iaaa019","url":null,"abstract":"We consider the problem of estimating the covariance matrix of a random signal observed through unknown translations (modeled by cyclic shifts) and corrupted by noise. Solving this problem allows to discover low-rank structures masked by the existence of translations (which act as nuisance parameters), with direct application to principal components analysis. We assume that the underlying signal is of length \u0000<tex>$L$</tex>\u0000 and follows a standard factor model with mean zero and \u0000<tex>$r$</tex>\u0000 normally distributed factors. To recover the covariance matrix in this case, we propose to employ the second- and fourth-order shift-invariant moments of the signal known as the power spectrum and the trispectrum. We prove that they are sufficient for recovering the covariance matrix (under a certain technical condition) when \u0000<tex>$r<sqrt{L}$</tex>\u0000. Correspondingly, we provide a polynomial-time procedure for estimating the covariance matrix from many (translated and noisy) observations, where no explicit knowledge of \u0000<tex>$r$</tex>\u0000 is required, and prove the procedure's statistical consistency. While our results establish that covariance estimation is possible from the power spectrum and the trispectrum for low-rank covariance matrices, we prove that this is not the case for full-rank covariance matrices. We conduct numerical experiments that corroborate our theoretical findings and demonstrate the favourable performance of our algorithms in various settings, including in high levels of noise.","PeriodicalId":45437,"journal":{"name":"Information and Inference-A Journal of the Ima","volume":"10 3","pages":"773-812"},"PeriodicalIF":1.6,"publicationDate":"2021-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1093/imaiai/iaaa019","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50224071","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We consider the problem of estimating a cloud of points from numerous noisy observations of that cloud after unknown rotations and possibly reflections. This is an instance of the general problem of estimation under group action, originally inspired by applications in three-dimensional imaging and computer vision. We focus on a regime where the noise level is larger than the magnitude of the signal, so much so that the rotations cannot be estimated reliably. We propose a simple and efficient procedure based on invariant polynomials (effectively: the Gram matrices) to recover the signal, and we assess it against fundamental limits of the problem that we derive. We show our approach adapts to the noise level and is statistically optimal (up to constants) for both the low and high noise regimes. In studying the variance of our estimator, we encounter the question of the sensivity of a type of thin Cholesky factorization, for which we provide an improved bound which may be of independent interest.
{"title":"The generalized orthogonal Procrustes problem in the high noise regime","authors":"Thomas Pumir;Amit Singer;Nicolas Boumal","doi":"10.1093/imaiai/iaaa035","DOIUrl":"https://doi.org/10.1093/imaiai/iaaa035","url":null,"abstract":"We consider the problem of estimating a cloud of points from numerous noisy observations of that cloud after unknown rotations and possibly reflections. This is an instance of the general problem of estimation under group action, originally inspired by applications in three-dimensional imaging and computer vision. We focus on a regime where the noise level is larger than the magnitude of the signal, so much so that the rotations cannot be estimated reliably. We propose a simple and efficient procedure based on invariant polynomials (effectively: the Gram matrices) to recover the signal, and we assess it against fundamental limits of the problem that we derive. We show our approach adapts to the noise level and is statistically optimal (up to constants) for both the low and high noise regimes. In studying the variance of our estimator, we encounter the question of the sensivity of a type of thin Cholesky factorization, for which we provide an improved bound which may be of independent interest.","PeriodicalId":45437,"journal":{"name":"Information and Inference-A Journal of the Ima","volume":"10 3","pages":"921-954"},"PeriodicalIF":1.6,"publicationDate":"2021-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1093/imaiai/iaaa035","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50224074","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We introduce and analyze a general framework for empirical risk minimization in which the observations and models of interest may be stationary systems or processes. Within the framework, which is presented in terms of dynamical systems, empirical risk minimization can be studied as a two-step procedure in which (i) the trajectory of an observed (but unknown) system is fit by a trajectory of a known reference system via minimization of cumulative per-state loss, and (ii) an invariant parameter estimate is obtained from the initial state of the best fit trajectory. We show that the weak limits of the empirical measures of best-matched trajectories are dynamically invariant couplings (joinings) of the observed and reference systems with minimal risk. Moreover, we establish that the family of risk-minimizing joinings is convex and compact and that it fully characterizes the asymptotic behavior of the estimated parameters, directly addressing identifiability. Our analysis of empirical risk minimization applies to well-studied problems such as maximum likelihood estimation and non-linear regression, as well as more complex problems in which the models of interest are stationary processes. To illustrate the latter, we undertake an extended analysis of system identification from quantized trajectories subject to noise, a problem at the intersection of dynamics and statistics.
{"title":"Empirical risk minimization for dynamical systems and stationary processes","authors":"Kevin McGoff;Andrew B Nobel","doi":"10.1093/imaiai/iaaa043","DOIUrl":"https://doi.org/10.1093/imaiai/iaaa043","url":null,"abstract":"We introduce and analyze a general framework for empirical risk minimization in which the observations and models of interest may be stationary systems or processes. Within the framework, which is presented in terms of dynamical systems, empirical risk minimization can be studied as a two-step procedure in which (i) the trajectory of an observed (but unknown) system is fit by a trajectory of a known reference system via minimization of cumulative per-state loss, and (ii) an invariant parameter estimate is obtained from the initial state of the best fit trajectory. We show that the weak limits of the empirical measures of best-matched trajectories are dynamically invariant couplings (joinings) of the observed and reference systems with minimal risk. Moreover, we establish that the family of risk-minimizing joinings is convex and compact and that it fully characterizes the asymptotic behavior of the estimated parameters, directly addressing identifiability. Our analysis of empirical risk minimization applies to well-studied problems such as maximum likelihood estimation and non-linear regression, as well as more complex problems in which the models of interest are stationary processes. To illustrate the latter, we undertake an extended analysis of system identification from quantized trajectories subject to noise, a problem at the intersection of dynamics and statistics.","PeriodicalId":45437,"journal":{"name":"Information and Inference-A Journal of the Ima","volume":"10 3","pages":"1073-1104"},"PeriodicalIF":1.6,"publicationDate":"2021-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1093/imaiai/iaaa043","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50347112","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We develop in this paper a novel intrinsic classification algorithm—multi-frequency class averaging (MFCA)—for classifying noisy projection images obtained from three-dimensional cryo-electron microscopy by the similarity among their viewing directions. This new algorithm leverages multiple irreducible representations of the unitary group to introduce additional redundancy into the representation of the optimal in-plane rotational alignment, extending and outperforming the existing class averaging algorithm that uses only a single representation. The formal algebraic model and representation theoretic patterns of the proposed MFCA algorithm extend the framework of Hadani and Singer to arbitrary irreducible representations of the unitary group. We conceptually establish the consistency and stability of MFCA by inspecting the spectral properties of a generalized local parallel transport operator through the lens of Wigner $D$ -matrices. We demonstrate the efficacy of the proposed algorithm with numerical experiments.
{"title":"Representation theoretic patterns in multi-frequency class averaging for three-dimensional cryo-electron microscopy","authors":"Yifeng Fan;Tingran Gao;Zhizhen Zhao","doi":"10.1093/imaiai/iaab012","DOIUrl":"https://doi.org/10.1093/imaiai/iaab012","url":null,"abstract":"We develop in this paper a novel intrinsic classification algorithm—multi-frequency class averaging (MFCA)—for classifying noisy projection images obtained from three-dimensional cryo-electron microscopy by the similarity among their viewing directions. This new algorithm leverages multiple irreducible representations of the unitary group to introduce additional redundancy into the representation of the optimal in-plane rotational alignment, extending and outperforming the existing class averaging algorithm that uses only a single representation. The formal algebraic model and representation theoretic patterns of the proposed MFCA algorithm extend the framework of Hadani and Singer to arbitrary irreducible representations of the unitary group. We conceptually establish the consistency and stability of MFCA by inspecting the spectral properties of a generalized local parallel transport operator through the lens of Wigner \u0000<tex>$D$</tex>\u0000-matrices. We demonstrate the efficacy of the proposed algorithm with numerical experiments.","PeriodicalId":45437,"journal":{"name":"Information and Inference-A Journal of the Ima","volume":"10 3","pages":"723-771"},"PeriodicalIF":1.6,"publicationDate":"2021-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1093/imaiai/iaab012","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50224070","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper establishes non-asymptotic concentration bound and Bahadur representation for the quantile regression estimator and its multiplier bootstrap counterpart in the random design setting. The non-asymptotic analysis keeps track of the impact of the parameter dimension $d$ and sample size $n$ in the rate of convergence, as well as in normal and bootstrap approximation errors. These results represent a useful complement to the asymptotic results under fixed design and provide theoretical guarantees for the validity of Rademacher multiplier bootstrap in the problems of confidence construction and goodness-of-fit testing. Numerical studies lend strong support to our theory and highlight the effectiveness of Rademacher bootstrap in terms of accuracy, reliability and computational efficiency.
{"title":"Multiplier bootstrap for quantile regression: non-asymptotic theory under random design","authors":"Xiaoou Pan;Wen-Xin Zhou","doi":"10.1093/imaiai/iaaa006","DOIUrl":"https://doi.org/10.1093/imaiai/iaaa006","url":null,"abstract":"This paper establishes non-asymptotic concentration bound and Bahadur representation for the quantile regression estimator and its multiplier bootstrap counterpart in the random design setting. The non-asymptotic analysis keeps track of the impact of the parameter dimension \u0000<tex>$d$</tex>\u0000 and sample size \u0000<tex>$n$</tex>\u0000 in the rate of convergence, as well as in normal and bootstrap approximation errors. These results represent a useful complement to the asymptotic results under fixed design and provide theoretical guarantees for the validity of Rademacher multiplier bootstrap in the problems of confidence construction and goodness-of-fit testing. Numerical studies lend strong support to our theory and highlight the effectiveness of Rademacher bootstrap in terms of accuracy, reliability and computational efficiency.","PeriodicalId":45437,"journal":{"name":"Information and Inference-A Journal of the Ima","volume":"10 3","pages":"813-861"},"PeriodicalIF":1.6,"publicationDate":"2021-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1093/imaiai/iaaa006","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50224072","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper, we study the problem of learning the weights of a deep convolutional neural network. We consider a network where convolutions are carried out over non-overlapping patches. We develop an algorithm for simultaneously learning all the kernels from the training data. Our approach dubbed deep tensor decomposition (DeepTD) is based on a low-rank tensor decomposition. We theoretically investigate DeepTD under a realizable model for the training data where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted convolutional kernels. We show that DeepTD is sample efficient and provably works as soon as the sample size exceeds the total number of convolutional weights in the network.
{"title":"Learning a deep convolutional neural network via tensor decomposition","authors":"Samet Oymak;Mahdi Soltanolkotabi","doi":"10.1093/imaiai/iaaa042","DOIUrl":"https://doi.org/10.1093/imaiai/iaaa042","url":null,"abstract":"In this paper, we study the problem of learning the weights of a deep convolutional neural network. We consider a network where convolutions are carried out over non-overlapping patches. We develop an algorithm for simultaneously learning all the kernels from the training data. Our approach dubbed deep tensor decomposition (DeepTD) is based on a low-rank tensor decomposition. We theoretically investigate DeepTD under a realizable model for the training data where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted convolutional kernels. We show that DeepTD is sample efficient and provably works as soon as the sample size exceeds the total number of convolutional weights in the network.","PeriodicalId":45437,"journal":{"name":"Information and Inference-A Journal of the Ima","volume":"10 3","pages":"1031-1071"},"PeriodicalIF":1.6,"publicationDate":"2021-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1093/imaiai/iaaa042","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"50347111","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-01-25eCollection Date: 2022-06-01DOI: 10.1093/imaiai/iaaa041
Shiqing Yu, Mathias Drton, Ali Shojaie
Estimation of density functions supported on general domains arises when the data are naturally restricted to a proper subset of the real space. This problem is complicated by typically intractable normalizing constants. Score matching provides a powerful tool for estimating densities with such intractable normalizing constants but as originally proposed is limited to densities on [Formula: see text] and [Formula: see text]. In this paper, we offer a natural generalization of score matching that accommodates densities supported on a very general class of domains. We apply the framework to truncated graphical and pairwise interaction models and provide theoretical guarantees for the resulting estimators. We also generalize a recently proposed method from bounded to unbounded domains and empirically demonstrate the advantages of our method.
{"title":"Generalized score matching for general domains.","authors":"Shiqing Yu, Mathias Drton, Ali Shojaie","doi":"10.1093/imaiai/iaaa041","DOIUrl":"https://doi.org/10.1093/imaiai/iaaa041","url":null,"abstract":"<p><p>Estimation of density functions supported on general domains arises when the data are naturally restricted to a proper subset of the real space. This problem is complicated by typically intractable normalizing constants. Score matching provides a powerful tool for estimating densities with such intractable normalizing constants but as originally proposed is limited to densities on [Formula: see text] and [Formula: see text]. In this paper, we offer a natural generalization of score matching that accommodates densities supported on a very general class of domains. We apply the framework to truncated graphical and pairwise interaction models and provide theoretical guarantees for the resulting estimators. We also generalize a recently proposed method from bounded to unbounded domains and empirically demonstrate the advantages of our method.</p>","PeriodicalId":45437,"journal":{"name":"Information and Inference-A Journal of the Ima","volume":"11 2","pages":"739-780"},"PeriodicalIF":1.6,"publicationDate":"2021-01-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9203079/pdf/iaaa041.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"40041673","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"OUP accepted manuscript","authors":"","doi":"10.1093/imaiai/iaab020","DOIUrl":"https://doi.org/10.1093/imaiai/iaab020","url":null,"abstract":"","PeriodicalId":45437,"journal":{"name":"Information and Inference-A Journal of the Ima","volume":"47 1","pages":""},"PeriodicalIF":1.6,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88917000","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"OUP accepted manuscript","authors":"","doi":"10.1093/imaiai/iaab029","DOIUrl":"https://doi.org/10.1093/imaiai/iaab029","url":null,"abstract":"","PeriodicalId":45437,"journal":{"name":"Information and Inference-A Journal of the Ima","volume":"84 1","pages":""},"PeriodicalIF":1.6,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"86949626","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"OUP accepted manuscript","authors":"","doi":"10.1093/imaiai/iaab022","DOIUrl":"https://doi.org/10.1093/imaiai/iaab022","url":null,"abstract":"","PeriodicalId":45437,"journal":{"name":"Information and Inference-A Journal of the Ima","volume":"591 1","pages":""},"PeriodicalIF":1.6,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"77317120","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：