A New Objective Measure Of Signal Complexity Using Bayesian Inference

Abstract

An objective Ockham prior which penalizes complexity in parametric signal hypotheses is derived from Bayesian fundamentals. Novel quantitative definitions of complexity are deduced under the procedure. This improves on current variants of the coding theoretic Minimum Message Length (MML) criterion where complexity definitions are imposed as heuristics. It is shown that the Ockham prior arises naturally in marginal Bayesian inference, but is excluded if joint inference is adopted. 1. I N T R O D U C T I O N : BAYESIAN SYSTEMATIC HYPOTHESES The Signal Identification problem arises whenever a systematic hypothesis is adopted to explain an observed data set. Let d = (d l , . . . , d ~ ) ~ be a finite set of one-dimensional observations. The prior hypothesis, 2, asserts that 2. O C K H A M ’ S R A Z O R Systematic hypotheses (1) must be assessed in the context of Ockham’s Razor (i.e. the Desideratum of Simplicity) [3] which, for the purposes of time series analysis, states that randomness must not be fitted with determinism. Consider a set of hypotheses, 21~12, . . ., parameterized with p.p. sets, 0 1 , 0 2 , . . . respectively. The likelihood function (LF) for the kth hypothesis is 1(& I d, 2,) = p(d I &, I k ) . The classical approach is to identify a model for d by maximizing l ( . ) . The resulting point inference, arg . sup, . supel I(& I d, lk), identifies both a mode1 (solving the Model Selection problem) and a set of parameter values (solving the Parameter Estimation problem). In open-ended inference, we may posit models with increasing numbers of degrees of heedom (e.g. increasing model order). In this manner, llell is reduced (l), 11 . 11 being the norm implied by the p.d.f. of e [l]. Ultimately, the likelihood-based inference machine fails because of its insensitivity to Ockham’s Razor. d = s + e 2.1. Subjec t ive vs. Objec t ive Complexi ty s # 0 is the unknown deterministic component (i.e. the ‘signal’) and e is the vector of unknown, non-systematic residuals. If the hypothesis is parametric, then s = s(8). The validity of the decomposition (1) must be assessed for a particular d, since all subsequent steps in the inference taskmodel selection, parameter estimationdepend upon it. 1.1. Probabi l i ty as Belief Calculus The fundamental concept underlying the Bayesian Paradigm is that the beliefs associated with inductive inference are uniquely quantified as probabilities and are consistently manipulated using the Probability Calculus [l]. The equivalence of the Belief Calculus and the Probability Calculus has been deduced from fundamentals [’I. Any unknowneither fixed or random-in a hypothesis has a domain, R, of possible values. The distribution of beliefs across R is expressed by a p.d.f. This constitutes tlie definition of a probabilistic parameter (p.p.) [ I ] which is tlie appropriate Bayesian extension of the random variable (r.v.) concept of orthodox inference. The two definitions merge when the uiikiiown is i~iherently random. The problem of admitting Ockham to an inference tends to be treated in two distinct stages [3-71: (i) an acceptable (quantifiable) definition of complexity is proposed; (ii) an inference procedure which includes a monotonically decreasing function of this complexity is adopted. Jeffreys [3] sought to specify the rate of ezchange between simplicity and closeness of fit to observations. His ‘Simplicity Postulate” stated that ‘simpler laws have the greater prior probabilities”. This subjective approach overlooks the prospect of engendering the principle objectively.