A stronger version of the redundancy-capacity theorem of universal coding

Abstract

The capacity of the channel induced by a given class of sources is well known to be an attainable lower bound on the redundancy of universal codes w.r.t this class, both in the minimax sense and in the Bayesian (maximin) sense. We show that this capacity is essentially a lower bound also in a stronger sense, that is, for "most" sources in the class. This result extends Rissanen's lower bound for parametric families. We demonstrate its applicability in several examples and discuss its implications in statistical inference.