MDL model selection using the ML plug-in code

Abstract

We analyse the behaviour of the ML plug-in code, also known as the Rissanen-Dawid prequential ML code, relative to single parameter exponential families M. If the data are i.i.d. according to an (essentially) arbitrary P, then the redundancy grows at 1/2c log n. We find that, in contrast to other important universal codes such as the 2-part MDL, Shtarkov and Bayesian codes where c = 1, here c equals the ratio between the variance of P and the variance of the element of M that is closest to P in KL-divergence. We show how this behaviour can impair model selection performance in a simple setting in which we select between the Poisson and geometric models