Definition of Entropy and related functions

The number of elements in a set

is denoted by .

For a random variable X, |X| denotes the number of elements in the set .

The entropy of X

is defined by:

with the convention for that , since .

The entropy is a measure of the information content or `uncertainty' of x. The question of why entropy is a fundamental measure of information content will be discussed in the forthcoming chapters. Here we note some properties of this mathematical function.

• with equality iff for one i.
• with equality iff for all i. (|X| denotes the number of elements in the set .)

The joint entropy of X,Y

is then:

Entropy is additive for independent random variables:
 

The conditional entropy of X given

is the entropy of the probability distribution .
 

The conditional entropy of X given Y

is the average over y of the conditional entropy of X given y.
 
This measures the average uncertainty that remains about x when y is known.

Chain rule for Entropy.

The joint entropy, conditional entropy and marginal entropy are related by:
 

In words, this says that the information content of XY is the information content of X plus the information content of Y given X.

The mutual information between X and Y

is
 
and satisfies H(X;Y) = H(Y;X), and . It measures the average reduction in uncertainty about x that results from learning the value of y, or vice versa. Equivalently, it measures the amount of information that y conveys about x.

The `entropy distance' between two random variables

can be defined to be the difference between their joint entropy and their mutual information:
 

This quantity satisfies the axioms for a distance -- , , , and .

Figure 1.14 shows how the total information content H(X,Y) of a joint ensemble can be broken down.

  
Figure: The relationship between joint information, marginal information, conditional information and mutual information.