Information Theory, Pattern Recognition & Neural Networks Information Theory, Pattern Recognition & Neural Networks

David J.C. MacKay

 
• Introduction to Information Theory
  • How can one achieve perfect communication over an imperfect, noisy commmunication channel?
    • The physical solution
    • The `system' solution
  • Error correcting codes for the binary symmetric channel
    • Repetition codes
    • Block codes -- the (7,4) Hamming code
      • Decoding the (7,4) Hamming code
    • Pictorial view of the Hamming code
      • Encoding
      • Decoding
    • Summary of Hamming code properties
    • Summary of codes' performances
  • What performance could the best codes achieve?
    • Example: f=0.1
  • Summary
  • Homework
  • Probability and entropy -- definitions and notation
    • Probabilities and ensembles
    • Meaning of probability.
    • Definition of Entropy and related functions
    • Other useful definitions
  • Exercises
• About this document ...