CSC 310 - Information Theory

Fall 2009

Textbook and references

We use the following text:

  • Information and coding theory, by Jones and Jones, Springer, 2000 [publisher's website].


    Related textbooks

    The above text is sufficient to master the material of CSC310. Below, I have compiled a list of supplementary, sources which you may find useful.

  • Information Theory, Inference and Learning Algorithms, by MacKay (freely available here), Cambridge University Press.
    This text supplements the textbook we use. The presentation in our text (see above) is of the form Definition-Lemma-Theorem, and it is crystal clear on the subject. However, the motivation/intuition part is not well covered. This second text contains lots of examples and pretty pictures. It also has the advantage of being free. Do not pay attention to the terminology or the technical material covered in the text. Its presentation is rather focused to a particular (and of limited interest) area of computer science, with somewhat inappropriate for our purposes technical exposition.

  • Elements of Information Theory, by Cover and Thomas, Wiley.
    Good, rigorous, and a little more advanced text than what we need here. Also, it seems to be more biased towards communication networks (but with an overall serious technical exposition).

    Textbooks on prerequisites

    I have compiled some references which you may find useful in reminding yourselves about the prerequisites. For Probability prerequisites we require knowledge on discrete probability. Also, elementary enumeration must be well appreciated. I'd recommend the following classic text.

  • An Introduction to Probability Theory and Its Applications, Vol. 1, by Feller, Wiley.
    This is a classic. Excellent, rigorous, with interesting examples and clear intuition. Suggestion: use it as a reference whenever needed during this course.

    For the Linear Algebra prerequisites we require elementary knowledge with emphasis on proofs. Any text that focuses on proofs (avoid texts that focus on calculations with matrices and linear systems) should do the job. I'd recommend the following one:

  • Linear Algebra (3rd edition), by Lang, Springer.
    Excellent introductory book. Hides most of unnecessary details, it is compact, simple, and to the point. One thing that it is implicit is the treatment of vector spaces over arbitrary fields. In CSC310 we require very basic knowledge of vector spaces over finite fields.

    From Abstract Algebra we have minimal requirements. Knowing the definitions of field and vector space (over arbitrary fields) is sufficient. For example, you can check the relevant sections/paragraphs from:

  • Abstract Algebra (3rd edition), by Dummit and Foote, Wiley.

    Recommended advanced textbooks

  • Introduction to Coding Theory, by Lint, Springer.

  • An Introduction to Kolmogorov Complexity and Its Applications (3rd edition), by Li and Vitaniy, Springer.