Linear-Gaussian systems and signal processing

Linear systems theory, based on the mathematics of vector spaces, is the backbone of all “classical” DSP and a large part of statistical machine learning. The basic idea -- that linear algebra applied to a signal can of substantial practical value -- has counterparts in many areas of science and technology. In other areas of science and engineering, linear algebra is often justified by the fact that it is often an excellent model for real-world systems. For example, in acoustics the theory of (linear) wave propagation emerges from the concept of linearization of small pressure disturbances about the equilibrium pressure in classical fluid dynamics. Similarly, the theory of electromagnetic waves is also linear. Except when a signal emerges from a justifiably linear system, in DSP and machine learning we do not have any particular correspondence to reality to back up the choice of linearity. However, the mathematics of vector spaces, particularly when applied to systems which are time-invariant and jointly Gaussian, is highly tractable, elegant and immensely useful.