The f -Divergence and Coupling of Probability Distributions

Abstract

We consider the problem of finding the minimum and maximum values of f-divergence for discrete probability distributions P and Q provided that one of these distributions and the value of their coupling are given. An explicit formula for the minimum value of the f-divergence under the above conditions is obtained, as well as a precise expression for its maximum value. This precise expression is not explicit in the general case, but in many special cases it allows us to write out both explicit formulas and simple upper bounds, which are sometimes optimal. Similar explicit formulas and upper bounds are also obtained for the Kullback–Leibler and χ² divergences, which are the most important cases of the f-divergence.