Information Theory, Pattern Recognition and Neural Networks

Introduction to information theory [1]

The possibility of reliable communication over unreliable channels. The (7,4) Hamming code and repetition codes.

Entropy and data compression [3]

Entropy, conditional entropy, mutual information, Shannon information content. The idea of typicality and the use of typical sets for source coding. Shannon's source coding theorem. Codes for data compression. Uniquely decodeable codes and the Kraft-MacMillan inequality. Completeness of a symbol code. Prefix codes. Huffman codes. Arithmetic coding.

Communication over noisy channels [3]

Definition of channel capacity. Capacity of binary symmetric channel; of binary erasure channel; of Z channel. Joint typicality, random codes, and Shannon's noisy channel coding theorem. Real channels and practical error-correcting codes. Hash codes.

Statistical inference, data modelling and pattern recognition [2]

The likelihood function and Bayes' theorem.

Approximation of probability distributions [2]

Laplace's method. (Approximation of probability distributions by Gaussian distributions.)
Monte Carlo methods: Importance sampling, rejection sampling, Gibbs sampling, Metropolis method. (Slice sampling, Hybrid Monte Carlo, Overrelaxation, exact sampling. *)
Variational methods and mean field theory. Ising models.

Data modelling with neural networks [2]

Interpolation and classification using a single neuron. (Multilayer perceptrons. *) Back- propagation algorithm. Learning algorithms viewed in terms of inference.

Neural networks as information storage devices [2]

Capacity of a single neuron. Hopfield network and its relationship to spin glasses. Hopfield network for optimization problems, e.g., travelling salesman problem. Boltzmann machine. Hopfield network as a mean field approximation to the Boltzmann machine. (Boltzmann machine learning algorithm.*) [* = non-examinable]