Other useful definitions

The relative entropy or Kullback-Leibler divergence

between two probability distributions P(x) and Q(x) that are defined over the same alphabet is
 
The relative entropy satisfies (Gibbs' inequality) with equality only if . Note that in general , so this is not strictly a `distance', though it is sometimes called the `K-L distance'. This quantity is important in pattern recognition and neural networks.

Convex function.

A function f(x) is convex over (a,b) if for all and ,

(See figure 1.15.) A function f is strictly convex if, for all , the equality holds only for and .

  
Figure: A convex function.

Some strictly convex functions are

• and for all x;
• and for x>0.

Jensen's inequality.

If f is a convex function and x is a random variable then:

where denotes expectation. If f is strictly convex and , then the random variable x is a constant (with probability 1).