Homepage for CSC 310, Spring 2008

Information Theory

University of Toronto at Mississauga



YOU SHOULD CHECK THE FOLLOWING ANNOUNCEMENTS REGULARLY
ANNOUNCEMENTS:
  • The final exam is out of 90. Approximately 40 points are for material covered before the midterm, and approximately 50 points are for material covered after the midterm, especially Lectures 12-16 (and the associated assignments and tutorials). Be sure you know the definitions of the various kinds of information channel (BSC, BEC, Z channel).
  • The final exam will cover the entire course (Lectures 1-16 inclusive). You are responsible for material covered in class, in tutorials, in the supplementary notes, in the assignments and on the midterm. You are NOT responsible for Lecture 17.
  • Solutions to Assignments 2 and 3 are now available, below.
  • For the final exam, you will be allowed to bring 20 sheets of paper with ONE lecture slide printed on each side (for a total of 40 lecture slides). No other aids are allowed: no pages from the textbook, none of the supplementary notes, no assignments or solutions, no personal notes, no electronic aids, and you must not write on the lecture slides.
  • Solutions to the midterm are now available, below.
  • Due to new privacy laws, grades can no longer be posted on the course web site and all marked work must be picked up personally.
  • For the lecture on typical sets, please see the modest probability refresher (under Background Material, below).
  • In addition to the lecture slides, supplementary notes are now available for some of the lectures. (Click on "Lecture slides" below)
    COURSE DESCRIPTION:

    Information theory is about representing information compactly and transmitting it reliably. It is central to such applications as lossless data compression (e.g. ZIP files), lossy data compression (e.g. MP3s), and channel coding (e.g. for DSL lines). It also lies at the heart of many exciting areas of contemporary science and engineering. Its impact has been crucial to the success of the Voyager missions to deep space, the invention of the CD, the feasibility of mobile phones, the development of the Internet, the study of linguistics and human perception, the understanding of black holes, and numerous other fields. For this reason, Information Theory is a core subject at universities around the world.

    This course provides an introduction to information theory and should be of interest to students in Computer Science, Statistics, Mathematics and Electrical Engineering. Topics to be covered include entropy, data compression, optimal compression, information channels, channel capacity, error-correcting codes and digital fountain codes (which are the twenty-first century standards for satellite communications, disk drives, and data broadcast). Information Theory has been offered on the St George campus for years. Now, for the first time, you can take it at UTM, with the added advantage of a smaller class size. The course itself will be similar to that offered on St George in 2006.

    Prerequisites:

  • CSC148H5; STA257H5; MAT223H5.
  • If you do not have these courses, but have a basic knowledge of probability, calculus and linear algebra and basic programming skills, please see the instructor.

    Text:

  • David MacKay, Information Theory, Inference and Learning Algorithms, Cambridge University Press.
  • Freely available on the web.

    Instructor:

  • Anthony Bonner
  • email: my last name [at] cs [dot] toronto [dot] edu
  • phone: 905-828-3813 (UTM), 416-978-7441 (St George)
  • office: CC 4004 (UTM), BA 4268 (St George)
  • office hours: by appointment

  • Lectures: Wednesday 4-6pm, NE 174.
  • Tutorials: Thursday 2-3pm, NE 236.
  • Tutorials may introduce new material not covered in lectures or in the text.
  • Teaching Assistant: Amin Farbod, email: a [dot] farbod [at] utoronto [dot] ca
  • Lecture slides
  • Course outline

    Midterm Test:

  • When: during the first hour of class on Weds February 27 (immediately after reading week).
  • What: everything up to the end of Lecture 8 (including the supplementary notes).
  • Closed book
  • Solutions

    Assignments:

  • Assignment 1 (Solutions)
  • Assignment 2 (Solutions)
  • Assignment 3 (Solutions)
  • Assignment 4

    Background Material:

  • A modest probability refresher.
  • F. R. Kschischang, Probability Refresher. Free Download

    Additional References:

  • G.A. Jones and J.M. Jones. Information and Coding Theory. Springer, 2000.
  • Thomas M. Cover, Joy A. Thomas. Elements of Information Theory, 2nd Edition. New York: Wiley-Interscience, 2006.
  • John R. Pierce. An Introduction to Information Theory. Dover Publications, 1980. (basic and cheap)
  • Fazlollah M. Reza. Introduction to Information Theory. Dover Publications, 1994. (Reprint of 1961 edition)
  • Wikipedia entry on information theory.

    Plagiarism and Cheating:

  • The academic regulations of the University are outlined in the Code of Behaviour on Academic Matters.
  • Advice on academic offences