This document is (c) David J.C. MacKay, 2001
It originates from http://www.inference.phy.cam.ac.uk/mackay/itprnn/book.html
It contains the text of David MacKay's book, Information theory, inference, and learning algorithms. (latex source)
Copying and distribution of this file are NOT PERMITTED.
The file is provided for convenience of anyone wishing to make a web-based search of the text of the book.
% This document is (c) David J.C. MacKay, 2001
%
% It originates from http://www.inference.phy.cam.ac.uk/mackay/itprnn/
% http://www.inference.phy.cam.ac.uk/mackay/itprnn/book.html
%
% It contains the text of David MacKay's book,
% Information theory, inference, and learning algorithms.
% (latex source)
%
% Copying and distribution of this file are NOT PERMITTED.
%
% The file is provided for convenience of anyone wishing to
% make a web-based search of the text of the book.
% was book2e.tex is now book.tex (and still latex2e)
\documentclass[11pt]{book}%
% last minute additions
\usepackage{DJCMamssymb}% needed for blacktriangleright Mon 10/11/03 (put in symbols instead)
\usepackage{ragged2e}% provides \justifying
% end last minute additions
\usepackage{floatflt}
%\usepackage{hangingsecnum}% makes sec numbers sit in the left margin (tried cutting out on Thu 6/11/03)
\usepackage{hangingsecnum2}% makes sec numbers sit in the left margin (modified Thu 6/11/03)
%\usepackage{mparhack}
\usepackage{mparhackright-209}% makes all margin pars go in right margin
\usepackage{marginfig}% Defines many macros for making various styles of figure with captions
%\usepackage{symbols}% Provides a few math symbols (replaced with DJCMamssymb)
%\usepackage{twoside}
\usepackage{myalgorith}% defines the Algorithm environment as a float
% Also forces fig,tab, and alg all to use a single counter
\usepackage{aside}% defines the {aside} environment
\usepackage{chapsummary}% helps me compile index-like objects (NOT USED)
\usepackage{chapternotes}% lots of assorted stuff
\usepackage{lsalike}% defines citation commands
\usepackage{booktabs}% makes nice quality tables
\usepackage{prechapter}% defines a chapter-like object
\usepackage{mycaption}% defines ``\indented''and \@makecaption; and the notindented style used in figure captions
% additions post-Sat 5/10/02
\usepackage{latexsym}% needed in order to make use of the \Box command
\usepackage{tocloft}% implements my look of table of contents
\usepackage{tocloftcomp2}% implements my look of table of contents (was tocloftcomp until Thu 6/11/03)
\usepackage{mychapter}% defines chapter command, including the look of the new chapter page
% also defines the look of the section and subsection commands
\usepackage{mycenter}% modifies center to reduce vertical space waste - useful for figures, etc.
\usepackage{mypart}% modifies part to not cleardoublepage (no longer Sat 5/4/03)
\usepackage{myheadings}% redefines the pagestyle ``headings''
% \usepackage{headingmods}% redefines the pagestyle ``headings'' (similar to myheadings)
% \usepackage{myindents}% defines parindent and leftmargin
\usepackage{graphics}% enables rotating of boxes
% \usepackage{boldmathgk}% provides bold alpha etc. (doesn't work)
% \usepackage{fixmath}% provides bold alpha etc. Also (I think) provides numerous sloping greeks that I don't like
\usepackage{fixmathDJCM}% provides bold alpha etc. Has Gamma definition cut out. and Omega
% suggested by DAG:
%\usepackage{amsmath}
%\usepackage{mathptmx}
\usepackage{DAGmathspacing}% provides smallfrac
\usepackage{boxedminipage}
\usepackage{fancybox}% Provides ability to put verbatim text inside boxes
\usepackage{bbold}% CTAN blackboard.ps was helpful for choosing this PROVIDES ``holey 1'' as \textbb{1}
\usepackage{epsf}% to allow use of metapost figures
%\usepackage{hyperref} % incompatible with something
%
\usepackage{multicol}% why does CTAN refer to multicols?
%\usepackage{myindex2}% overrides book definition of index
\usepackage{myindex}% overrides book definition of index
\usepackage{makeidx}
\usepackage{mybibliog}
\usepackage{mygaps}% defines \eq and \puncgap and \colonspace and \puncspace
\usepackage{mytoc}% suppresses the CONTENTS headings
\makeindex
%
\newcommand{\thedraft}{7.0}% 6.6 was 2nd printing. 6.8 was when I fixed errs Tue 24/2/04 % 6.9 = Mon 28/6/04 % 6.10 = Mon 2/8/04 % 6.11 Sun 22/8/04 % 7.0 final for 3rd printing
\renewcommand{\textfraction}{0.10}
\pagestyle{headings}
\begin{document}
\bibliographystyle{lsalikedjcmsc}%.bst
%\newcommand{\bf}{\textbf}
%\newcommand{\sf}{\textsf}
%%\newcommand{\em}{\textem}
%\newcommand{\rm}{\textrm}
%\newcommand{\tt}{\texttt}
%\newcommand{\sl}{\textsl}
%\newcommand{\sc}{\textsc}
%
% chapter.tex
%
% this contains a few common definitions for all chapters
% of the itprnn book
% for _l1.tex:
\hyphenation{left-multi-pli-ca-tion}
\hyphenation{multi-pli-ca-tion}
%
\newcommand{\partnoun}{Part}
\newcommand{\partone}{\partnoun\ I}
\newcommand{\datapart}{I}
\newcommand{\noisypart}{II}
\newcommand{\finfopart}{III}
\newcommand{\probpart}{IV}
\newcommand{\netpart}{V}
\newcommand{\sgcpart}{VI}
\newcommand{\hybrid}{Hamiltonian}
\newcommand{\Hybrid}{Hamiltonian}
%
% If sending book to readers -
\newcommand{\begincuttable}{}
\newcommand{\ENDcuttable}{}
% If sending to editor -
%\newcommand{\begincuttable}{\marginpar{\raisebox{-0.5in}[0in][0in]{$\downarrow$}CUTTABLE?}}
%\newcommand{\ENDcuttable}{\marginpar{\raisebox{0.5in}[0in][0in]{$\uparrow$}CUTTABLE?}}
%
\newcommand{\adhoc}{ad hoc}
\newcommand{\busstop}{bus-stop}
\newcommand{\mynewpage}{\newpage}% switch this off later Sun 3/2/02
% see also tex/inputs/itchapter.sty
% chapternotes.sty is where there is an index
\newcommand{\fN}{f\!N}
\newcommand{\exercisetitlestyle}{\sf}
%
% used in sumproduct.tex and gallager.tex
\newcommand{\Mn}{{\cal M}(n)}
\newcommand{\Nm}{{\cal N}(m)}
%\newcommand{\N}{{\cal N}}
%
% the delta function that is 1 if true (defined in notation.tex)
\newcommand{\truth}{\mbox{\textbb{1}}}
% requires:
% \usepackage{bbold}% CTAN blackboard.ps was helpful for choosing this
%
% used in gene.tex
\newcommand{\deltaf}{\delta\! f}
\newcommand{\tI}{\tilde{I}}
\newcommand{\Kp}{K_{\rm{p}}}
\newcommand{\Ks}{K_{\rm{s}}}
%
% end
% lang4.tex - distributions.tex
\newcommand{\lI}{I}
%
% clust.tex
\newcommand{\rnk}{r^{(n)}_k}
\newcommand{\hkn}{\hat{k}^{(n)}}
% good sizes:
% -0.45: 1.25
% -0.25: 0.65
% -0.4 0.8
\newcommand{\softfig}[1]{\hspace{-0.4in}\psfig{figure=octave/kmeansoft/ps1/#1.ps,width=0.8in,angle=-90}}
\newcommand{\softtfa}[3]{\begin{tabular}{c}{$t=#2$}\\
\hspace*{-0.4in}\mbox{\psfig{figure=octave/kmeansoft/#3/#1.ps,width=1.2in,angle=-90}\hspace*{-0.2in}}\\
\end{tabular}}
\newcommand{\softtfabig}[3]{\begin{tabular}{c}{$t=#2$}\\
\hspace*{-0.6in}\mbox{\psfig{figure=octave/kmeansoft/#3/#1.ps,width=1.5in,angle=-90}\hspace*{-0.2in}}\\
\end{tabular}}
\newcommand{\softtfabigb}[3]{\begin{tabular}{c}{$t=#2$}\\
\hspace*{-0.45in}\mbox{\psfig{figure=octave/kmeansoft/#3/#1.ps,width=1.625in,angle=-90}\hspace*{-0.2in}}\\
\end{tabular}}
\newcommand{\softtf}[2]{\softtfa{#1}{#2}{ps1}}
\newcommand{\softtfbig}[2]{\softtfabig{#1}{#2}{ps1}}
\newcommand{\softtfbigb}[2]{\softtfabigb{#1}{#2}{ps1}}
\newcommand{\softtfb}[2]{\softtfa{#1}{#2}{ps3}}
\newcommand{\softtfbbig}[2]{\softtfabigb{#1}{#2}{ps3}}
\newcommand{\softfc}[1]{\begin{tabular}{c}%
\hspace*{-0.2in}\mbox{\psfig{figure=octave/kmeansoft/ps5/#1.ps,width=1.32in,angle=-90}\hspace*{-0.2in}}\\
\end{tabular}}
% end
%
% used in _p1 and _l2
\newcommand{\hpheight}{26mm}
\newcommand{\wow}{\marginpar{{\Huge{$*$}}}}
%\newcommand{\wow}{\marginpar{\raisebox{-12pt}{\psfig{figure=figs/wow.eps,width=1in}}}}
%
% used in _l1.tex:::::::
\renewcommand{\q}{{f}}
\newcommand{\obr}[3]{\overbrace{{#1}\,{#2}\,{#3}}}
\newcommand{\ubr}[3]{\underbrace{{#1}\,{#2}\,{#3}}}
\newcommand{\nbr}[3]{{{#1}\,{#2}\,{#3}}}
%
% for \mid and gaps puncgap etc see mygaps.sty
\newcommand{\EM}{EM}
\newcommand{\ENDsolution}{\hfill \ensuremath{\epfsymbol}\par}
\newcommand{\ENDproof}{\hfill \ensuremath{\epfsymbol}\par}
\newcommand{\Hint}{{\sf{Hint}}}
\newcommand{\viceversa}{{\itshape{vice versa}}}
\newcommand{\analyze}{analyze}
\newcommand{\analyse}{analyze}
\newcommand{\fitpath}{/home/mackay/octave/fit/ps}% used in fit.tex (gaussian fitting, octave)
% CUP style:
\renewcommand{\cf}{cf.}
\renewcommand{\ie}{i.e.}
\renewcommand{\eg}{e.g.}
\renewcommand{\NB}{N.B.}
%
% symbols i e and d in maths (operators)
\newcommand{\im}{{\rm i}}
\newcommand{\e}{{\rm e}}
% \d is already defined
%
% needs
% \usepackage{boxedminipage}
\newenvironment{conclusionboxplain}%
{\begin{Sbox}\begin{minipage}{\textwidth}}%
{\end{minipage}\end{Sbox}\fbox{\TheSbox}}
\newenvironment{conclusionbox}%
%{\begin{Sbox}\begin{minipage}{\textwidth}}%
%{\end{minipage}\end{Sbox}\fbox{\TheSbox}}
{% see also marginfig.sty for conflicting use of this enironment and its params - and for defn of fatfboxsep
\fatfboxsep%
\setlength{\mylength}{\textwidth}%
\addtolength{\mylength}{-2\fboxsep}%
\addtolength{\mylength}{-2\fboxrule}%
\vskip8pt\noindent\begin{Sbox}\begin{minipage}{\mylength}\hspace*{-\fboxsep}\hspace*{-\fboxrule}%
\hspace*{\leftmargini}\begin{minipage}{\textwidthlessindents}}%
{\end{minipage}\end{minipage}\end{Sbox}\shadowbox{\TheSbox}\resetfboxsep\vskip 1pt}
\newenvironment{oldconclusionbox}%
{\vskip 0.1pt \noindent\rule{\textwidth}{0.1pt}\vskip -18pt\begin{quote}\vskip -8pt}%
{\end{quote}\vskip -14pt \noindent\rule{\textwidth}{0.1pt}\vskip 6pt}
% {\vskip 0.1pt \noindent\rule{\textwidth}{0.1pt}\vskip -12pt\begin{quote}}%
% {\end{quote}\vskip -12pt \noindent\rule{\textwidth}{0.1pt}}
\newcommand{\dy}{\d y}
\newcommand{\plus}{+}
\newcommand{\Wenglish}{Wenglish}% winglish
\newcommand{\wenglish}{\Wenglish}% winglish
\newcommand{\percent}{{per cent}}% in USA only: percent
%
%\newcommand{\nonexaminable}{$^{*}$}
\newcommand{\nonexaminable}{}
%
% for exact sampling chapter
\newcommand{\envelope}{summary state}
%
\def\unit#1{\,{\rm #1}}
\def\cm{\unit{cm}}
\def\grams{\unit{g}}
% this is a 209 versus 2e problem: (huffman.latex edited instead)
%\def\tenrm{\rm}
%\def\tenit{\it}
%
% other problems: \pem
\renewcommand{\textfraction}{0.1}
%
% for use in free text:
\newcommand{\bits}{{\rm bits}}
\newcommand{\bita}{{\rm bit}}
% for use in equations or in '1 bit'
\newcommand{\ubits}{\,{\bits}}
\newcommand{\ubit}{\,{\bita}}
%
%
%
% ch 2:
\newcommand{\sixtythree}{{\tt sixty-three}}
\newcommand{\aep}{`asymptotic equipartition' principle}
%
% used in alpha:
\newcommand{\sla}{\sqrt{\lambda_a}}
\newcommand{\kga}{\kappa\gamma}
\newcommand{\kkgg}{\kappa^2\gamma^2}
\newcommand{\skg}{\sqrt{\kappa\gamma}}
\newcommand{\TYP}{{\rm \scriptscriptstyle TYP}}
%
\newcommand{\bb}{{\bf b}}
%
% used in ising.tex and _s4.tex
% J=+1 are in states1, J=-1 are in states
%\newcommand{\risingsample}[1]{\psfig{figure=isingfigs/states1/#1.ps,width=1.82in}}
\newcommand{\risingsample}[1]{\psfig{figure=isingfigs/states1/#1.ps,width=1in}}% was 1.75
\newcommand{\smallrisingsample}[1]{\psfig{figure=isingfigs/states1/#1.ps,width=0.6in}}% was 1.2 was 0.9
\newcommand{\Hisingsample}[1]{\psfig{figure=isingfigs/states1/#1.ps,width=2.6in}}
\newcommand{\hisingsample}[1]{\psfig{figure=isingfigs/states/#1.ps,width=2.6in}}
\newcommand{\bighisingsample}[1]{\psfig{figure=isingfigs/states/#1.ps,width=3.86in}}
%
% used in _noiseless.tex
\newcommand{\Connectionmatrix}{Connection matrix}
\newcommand{\connectionmatrix}{connection matrix}
\newcommand{\connectionmatrices}{connection matrices}
%\newcommand{\cwM}{M}% codeword number
%\newcommand{\cwm}{m}% codeword number
\newcommand{\cwM}{S}% codeword number
\newcommand{\cwm}{s}% codeword number
\newcommand{\sa}{\alpha}% signal amplitude in gaussian channel
%
\newcommand{\cmA}{A}% connection matrix symbol
\newcommand{\bcmA}{{\bf \cmA}}% connection matrix symbol
\newcommand{\bAcm}{{\bcmA}}
\newtheorem{ctheorem}{Theorem}[chapter]
\newtheorem{definc}{Definition}[chapter]
\newcommand{\appendixref}[1]{Appendix \ref{#1}}
\newcommand{\appref}[1]{Appendix \ref{#1}}
\newcommand{\Appendixref}[1]{Appendix \ref{#1}}
\newcommand{\sectionref}[1]{section \ref{#1}}
\newcommand{\Sectionref}[1]{Section \ref{#1}}
\newcommand{\secref}[1]{section \ref{#1}}
\newcommand{\Secref}[1]{Section \ref{#1}}
\newcommand{\chapterref}[1]{Chapter \ref{#1}}
\newcommand{\Chapterref}[1]{Chapter \ref{#1}}
\newcommand{\chref}[1]{Chapter \ref{#1}}
\newcommand{\Chref}[1]{Chapter \ref{#1}}
\newcommand{\chone}{\ref{ch.one}}
\newcommand{\chtwo}{\ref{ch.two}}
\newcommand{\chthree}{\ref{ch.three}}
\newcommand{\chfour}{\ref{ch.four}}
\newcommand{\chfive}{\ref{ch.five}}
\newcommand{\chsix}{\ref{ch.six}}
\newcommand{\chseven}{\ref{ch.ecc}}
\newcommand{\cheight}{\ref{ch.bayes}}
\newcommand{\chthirteen}{\ref{ch.single.neuron.class}}% single neuron
\newcommand{\chfourteen}{\ref{ch.single.neuron.bayes}}% single neuron bayes?
\newcommand{\chtwelve}{\ref{ch.nn.intro}}% intro to nn
\newcommand{\chcover}{\ref{ch.cover}}
\newcommand{\chbayes}{\ref{ch.bayes}}
\newcommand{\secpulse}{\ref{sec.pulse}}% 7.2.1?}
\newcommand{\secthirteenthree}{13.3?}
\newcommand{\secmetrop}{\ref{sec.metrop}}% 11.3?}
\newcommand{\figooo}{?1.11?}
\newcommand{\eqgamma}{8.27?}
\newcommand{\TSP}{travelling salesman problem}
\newcommand{\Bayes}{Bayes'}
\newcommand{\vfe}{variational free energy}
\newcommand{\vfem}{variational free energy minimization}
% could make this \ch6 = \ref{ch6}
% author, title etc is in here....
% {headerinfo.tex}% uses special commands
\setcounter{secnumdepth}{2}%
\newcommand{\indep}{\bot}% upside down pi desired
\newcommand{\dbf}{\slshape}% boldface in definitions
\newcommand{\dem}{\slshape}% emphasized definitions in text
\newcommand{\solutionb}[2]{\setcounter{solution_number}{#1}
\solutiona{#2}}
\newcommand{\lsolution}[2]{\section{Solution to exercise {#1}}{#2}}
%
%
\newcommand{\FIGS}{/home/mackay/book/FIGS}
\newcommand{\bookfigs}{/home/mackay/book/figs}
\newcommand{\figsinter}{/home/mackay/handbook/figs/inter}
\newcommand{\exburglar}{\exerciseref{ex.burglar}}
\newcommand{\exnine}{\exerciseref{ex.invP}}%10}
\newcommand{\exseven}{\exerciseonlyref{ex.weigh}}% use deprecated!
% was \exseven .... \exerciseref{ex.expectn}}%9}
\newcommand{\exaseven}{\exerciseref{ex.R9}}%{7}
\newcommand{\exten}{\exerciseref{ex.expectng}}%{11}
\newcommand{\exfourteen}{\exerciseref{ex.Hadditive}}%{15}
\newcommand{\exfifteen}{\exerciseref{ex.Hcondnal}}%{16}
\newcommand{\exeighteen}{\exerciseref{ex.Hmutualineq}}%{19}
\newcommand{\extwenty}{\exerciseref{ex.rel.ent}}%{21}
\newcommand{\extwentyone}{\exerciseref{ex.joint}}%{22}% the joint ensemble
\newcommand{\extwentytwo}{\exerciseref{ex.dataprocineq}}%{23}
\newcommand{\extwentythree}{\exerciseref{ex.zxymod2}}%{24}
\newcommand{\extwentyfour}{\exerciseref{ex.waithead}}%{25}
\newcommand{\extwentyfive}{\exerciseref{ex.sumdice}}%{26}
\newcommand{\extwentysix}{\exerciseref{ex.RN}}%{27}
\newcommand{\extwentyseven}{\exerciseref{ex.RNGaussian}}%{28}
\newcommand{\exthirtyone}{\exerciseref{ex.logit}}%{32}% logistic
\newcommand{\exthirtysix}{\exerciseref{ex.exponential}}%{37}%
\newcommand{\exthirtyseven}{\exerciseref{ex.blood}}%{38}% forensic
\newcommand{\exfiftythree}{\exerciseref{ex.}}%{53}% integers
\newcommand{\eqsixteenfive}{16.5}
\newcommand{\Kraft}{Kraft}% Kraft--McMillan
\newcommand{\exrelent}{\exerciseref{ex.rel.ent}}%{20} %% \ref{ex.rel.ent}
\newcommand{\eqKL}{1.24} %% \eqref{eq.KL}
\newcommand{\bSigma}{{\mathbf{\Sigma}}}
\newcommand{\sumproduct}{sum-product}
%
% for cpi material
%
\newcommand{\sigbias}{\sigma_{\rm bias}}
\newcommand{\sigin}{\sigma_{\rm in}}
\newcommand{\sigout}{\sigma_{\rm out}}
\newcommand{\abias}{\alpha_{\rm bias}}
\newcommand{\ain}{\alpha_{\rm in}}
\newcommand{\aout}{\alpha_{\rm out}}
%\newcommand{\bff}{\bf}
\newcommand{\handfigs}{/home/mackay/handbook/figs}
\newcommand{\mjofigs}{/home/mackay/figs/mjo}
\newcommand{\FIGSlearning}{/home/mackay/book/FIGS/learning}
\newcommand{\codefigs}{/home/mackay/_doc/code/ps/ps}
%
% mncEL stuff
%
\newcommand{\ebnowide}[1]{\mbox{\psfig{figure=../../code/#1.ps,width=2.8in,angle=-90}}}
\newcommand{\fem}{m}
\newcommand{\feM}{M}
\newcommand{\fel}{n}
\newcommand{\feL}{N}
\renewcommand{\L}{N}
\newcommand{\feLm}{{\cal N}(m)}
\newcommand{\feMl}{{\cal M}(n)}
\newcommand{\feK}{N}
\newcommand{\fek}{n}
\newcommand{\feKn}{{\cal N}(m)}
\newcommand{\feNk}{{\cal M}(n)}
\newcommand{\feN}{M}
\newcommand{\fen}{m}
\newcommand{\fer}{r}
\newcommand{\GL}{GL}
\newcommand{\SMN}{GL}
\newcommand{\NMN}{MN}
\newcommand{\MN}{MN}
\renewcommand{\check}{check}% was relationship
\newcommand{\checks}{checks}% was relationship
\newcommand{\fs}{f_{\rm s}}
\newcommand{\fn}{f_{\rm n}}
\newcommand{\llncspunc}{.}
\newcommand{\query}{\mbox{{\tt{?}}}}
\newcommand{\lcA}{{H}}
\newcommand{\rmncNall}{/home/mackay/_doc/code/rmncNall}
\newcommand{\oneA}{1A}
\newcommand{\twoA}{2A}
\newcommand{\thrA}{2A}
\newcommand{\oneB}{1B}
\newcommand{\twoB}{2B}
\newcommand{\thrB}{2B}
\newcommand{\bndips}{/home/mackay/_doc/code/bndips}
\newcommand{\codeps}{/home/mackay/_doc/code/ps}
\newcommand{\equalnode}{\raisebox{-1pt}[0in][0in]{\psfig{figure=figs/gallager/equal.eps,width=8pt}\hspace{0mm}}}
\newcommand{\plusnode}{\raisebox{-1pt}[0in][0in]{\psfig{figure=figs/gallager/plus.eps,width=8pt}\hspace{0mm}}}
%
% Mon 26/5/03 modified this to try to centre the left heading
\newcommand{\fourfourtable}[9]{\begin{tabular}[b]{lcc@{\hspace{4pt}}c}
\multicolumn{1}{l}{#1:} & & \multicolumn{2}{c}{#2} \\[-0.1in]% \cline{1-1}
& & {#3} & {#4} \\ \cline{3-4}
\raisebox{-6.5pt}[0pt][0pt]{{#5}} &\multicolumn{1}{l|}{#3} & {#6} & {#7} \\[-7pt]
&\multicolumn{1}{l|}{#4} & {#8} & {#9} \\
\end{tabular}}
% Mon 26/5/03 extra version with heading right aligned and space reduced between col 1 and 2
\newcommand{\fourfourtabler}[9]{\begin{tabular}[b]{r@{}cc@{\hspace{4pt}}c}
\multicolumn{1}{l}{#1:} & & \multicolumn{2}{c}{#2} \\[-0.1in]% \cline{1-1}
& & {#3} & {#4} \\ \cline{3-4}
\raisebox{-6.5pt}[0pt][0pt]{{#5}} &\multicolumn{1}{l|}{#3} & {#6} & {#7} \\[-7pt]
&\multicolumn{1}{l|}{#4} & {#8} & {#9} \\
\end{tabular}}
\newcommand{\fourfourtablebeforemaythree}[9]{\begin{tabular}[b]{lcc@{\hspace{4pt}}c}
\multicolumn{1}{l}{#1:} & & \multicolumn{2}{c}{#2} \\[-0.1in]% \cline{1-1}
& & {#3} & {#4} \\ \cline{3-4}
{#5} &\multicolumn{1}{l|}{#3} & {#6} & {#7} \\[-7pt]
&\multicolumn{1}{l|}{#4} & {#8} & {#9} \\
\end{tabular}}
\newcommand{\fourfourtableb}[9]{\begin{tabular}[b]{l|c@{\hspace{1pt}}c@{\hspace{3pt}}c}
{#1} & {#2} & {#3} & {#4} \\ \cline{1-1}\cline{3-4}
\multicolumn{2}{l}{#5} & & \\
\multicolumn{1}{l|}{#3} & & {#6} & {#7} \\[-5pt]
\multicolumn{1}{l|}{#4} & & {#8} & {#9} \\
\end{tabular}}
\newcommand{\fourfourtableold}[9]{\begin{tabular}[b]{l|c|c|c|}
{#1} & {#2} & {#3} & {#4} \\ \cline{1-1}
\multicolumn{2}{l|}{#5} & & \\ \hline
\multicolumn{2}{l|}{#3} & {#6} & {#7} \\ \hline
\multicolumn{2}{l|}{#4} & {#8} & {#9} \\ \hline
\end{tabular}}
\newcommand{\mathsstrut}{\rule[-3mm]{0pt}{8mm}}
%
% for ra.tex
%
\newcommand{\halfw}{0.35in}
\newcommand{\onew}{0.9in}%{0.7in}% used in Gallager/MN figures in ra.tex% increased Wed 9/4/03
\newcommand{\onehalfw}{1.05in}
\newcommand{\twow}{1.4in}
\newcommand{\twohalfw}{1.75in}
\newcommand{\GHfig}[1]{\psfig{figure=GHps/#1,width=\onehalfw}}% for rate 1/3
\newcommand{\GHfigone}[1]{\psfig{figure=GHps/#1,width=\onew}}%
\newcommand{\GHfigthird}[1]{\psfig{figure=GHps/#1,width=\halfw}}
\newcommand{\GHfigquarter}[1]{\psfig{figure=GHps/#1,width=\twohalfw}}
\newcommand{\GHfigtwo}[1]{\psfig{figure=GHps/#1,width=\twow}}% for rate 1/2
\newcommand{\GHfigdouble}[1]{\psfig{figure=GHps/#1,width=\twohalfw}}% for five wide
% extra wide fitting::::::::::: (for turbo)
\newcommand{\GHfigdoubleE}[1]{\psfig{figure=GHps/#1,width=2in}}% for five wide
\newcommand{\GHfigE}[1]{\psfig{figure=GHps/#1,width=1.2in}}% for rate 1/3
%
\newcommand{\GHdrawfig}[1]{\psfig{figure=GHps/#1,width=1.5in}}% was 1.8
\newcommand{\standardfig}[1]{\psfig{figure=rirreg/#1,width=1.8in,angle=-90}}
\newcommand{\loopsfig}[1]{\psfig{figure=rirreg/loops.#1,height=1.85in,width=1.8in,angle=-90}}
\newcommand{\titledfig}[2]{\begin{tabular}{c}%
{#1}\\%
\standardfig{#2}\\%
\end{tabular}%
}
%
% for the single neuron chapters
%
\newcounter{funcfignum}
\setcounter{funcfignum}{1}
\newcommand{\funcfig}[2]{
\put(#1,#2){\makebox(0,0)[b]{
\begin{tabular}{@{}c@{}}
\psfig{figure=\FIGSlearning/f.#1.#2.ps,height=1.3in,width=1.3in,angle=-90} \\[-0.15in]
$\bw = (#1,#2)$
\\ \end{tabular}
}
}
}
\newcommand{\wflatfig}[1]{
\begin{tabular}{@{}c@{}}\setlength{\unitlength}{1in}\begin{picture}(1.5,1.3)(0.30,0.40)
\psfig{figure=\FIGSlearning/#1,height=2.43in,width=2.064in,angle=-90}
% was 1.3,1.3
\end{picture}\\\end{tabular}
}
\newcommand{\wsurfig}[1]{
\begin{tabular}{@{}c@{}}\setlength{\unitlength}{1in}\begin{picture}(1.5,1.5)(0,0)
\psfig{figure=\FIGSlearning/#1,height=1.8in,width=1.8in,angle=-90}
% was 1.5,1.5
\end{picture}\end{tabular}
}
\newcommand{\datfig}[1]{
\begin{tabular}{@{}c@{}}\setlength{\unitlength}{1in}\begin{picture}(1,1)(0.30,0.1)
\psfig{figure=\FIGSlearning/#1,height=1.2in,width=1.412in,angle=-90}
% was 1,1
\end{picture}\end{tabular}
}
\newcommand{\optens}{optimal input distribution}% used in l5.tex, l6.tex, s5.tex
\newcommand{\dilbertcopy}{{[Dilbert image Copyright\copyright{1997} United Feature Syndicate, Inc.,
used with permission.]}}
\newcommand{\Rnine}{\mbox{R}_9}
\newcommand{\Rthree}{\mbox{R}_3}
\newcommand{\eof}{{\Box}}
\newcommand{\teof}{\mbox{$\Box$}}% for use in text
\newcommand{\ta}{{\tt{a}}}
\newcommand{\tb}{{\tt{b}}}
%\newcommand{\dits}{dits}
%\newcommand{\dit}{dit}
\newcommand{\disc}{disk}
\newcommand{\dits}{bans}
\newcommand{\dit}{ban}
%
% used in l5
%
\newcommand{\BSC}{binary symmetric channel}
\newcommand{\BEC}{binary erasure channel}
\newcommand{\subsubpunc}{}% change to . if subsubsections are given in-line headings
%
% convolutional code definitions
%
\newcommand{\cta}{t^{(a)}}
\newcommand{\ctb}{t^{(b)}}
\newcommand{\z}{z}
\newcommand{\lfsr}{linear-feedback shift-register}
%
% definitions for including hinton diagrams from extended directory
%
\newcommand{\ecfig}[1]{\psfig{figure=extended/ps/#1.ps,silent=}}
% extra argument
\newcommand{\ecfigb}[2]{\psfig{figure=extended/ps/#1.ps,#2,silent=}}
%
% used in _s1 and in _linear maybe
%%%%%%%%%%% see /home/mackay/code/bucky
\newcommand{\buckypsfig}[1]{\mbox{\psfig{figure=buckyps/#1,width=1.2in}}}
\newcommand{\buckypsfigw}[1]{\mbox{\psfig{figure=buckyps/#1,width=1.75in}}}
\newcommand{\buckypsgraph}[1]{\mbox{\psfig{figure=buckyps/#1,width=1.2in,angle=-90}}}
\newcommand{\buckypsgraphb}[1]{\mbox{\psfig{figure=buckyps/#1,width=1.75in,angle=-90}}}
\newcommand{\buckypsgraphB}[1]{\mbox{\psfig{figure=buckyps/#1,width=2.2in,angle=-90}}}
%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%55
% for l1a
%%%%%%%%%%%%%%%%%%
% example
% \bigrampicture{3.538mm}{hd_conbigram.ps}
% \bigrampicture{3.538mm}{hd_conbigram.ps,width=278pt}%%%%%%% 278 is the original size
% This used to work fine in latex209 then needed rejigging in 2e.
% (alignment of g,j,p,q,y wrong at the bottom) (saved to graveyard.tex
\newcommand{\bigrampicture}[3]%args are unitlength,picturename-and-picturesize,font-request
{%%%%%%%%%
\setlength{\unitlength}{#1}
\begin{picture}(30,30)(0,-30)% was 28,28 0,-28
\put(0.15,-27.8){\makebox(0,0)[bl]{\psfig{figure=bigrams/#2,angle=-90}}}
\put(1,-29){\makebox(0,0)[b]{{#3\tt a}}}
\put(2,-29){\makebox(0,0)[b]{{#3\tt b}}}
\put(3,-29){\makebox(0,0)[b]{{#3\tt c}}}
\put(4,-29){\makebox(0,0)[b]{{#3\tt d}}}
\put(5,-29){\makebox(0,0)[b]{{#3\tt e}}}
\put(6,-29){\makebox(0,0)[b]{{#3\tt f}}}
\put(7,-29){\makebox(0,0)[b]{\raisebox{0mm}[0mm][0mm]{#3\tt g}}}
\put(8,-29){\makebox(0,0)[b]{{#3\tt h}}}
\put(9,-29){\makebox(0,0)[b]{{#3\tt i}}}
\put(10,-29){\makebox(0,0)[b]{\raisebox{0mm}[0mm][0mm]{#3\tt j}}}
\put(11,-29){\makebox(0,0)[b]{{#3\tt k}}}
\put(12,-29){\makebox(0,0)[b]{{#3\tt l}}}
\put(13,-29){\makebox(0,0)[b]{{#3\tt m}}}
\put(14,-29){\makebox(0,0)[b]{{#3\tt n}}}
\put(15,-29){\makebox(0,0)[b]{{#3\tt o}}}
\put(16,-29){\makebox(0,0)[b]{\raisebox{0mm}[0mm][0mm]{#3\tt p}}}
\put(17,-29){\makebox(0,0)[b]{\raisebox{0mm}[0mm][0mm]{#3\tt q}}}
\put(18,-29){\makebox(0,0)[b]{{#3\tt r}}}
\put(19,-29){\makebox(0,0)[b]{{#3\tt s}}}
\put(20,-29){\makebox(0,0)[b]{{#3\tt t}}}
\put(21,-29){\makebox(0,0)[b]{{#3\tt u}}}
\put(22,-29){\makebox(0,0)[b]{{#3\tt v}}}
\put(23,-29){\makebox(0,0)[b]{{#3\tt w}}}
\put(24,-29){\makebox(0,0)[b]{{#3\tt x}}}
\put(25,-29){\makebox(0,0)[b]{\raisebox{0mm}[0mm][0mm]{#3\tt y}}}
\put(26,-29){\makebox(0,0)[b]{{#3\tt z}}}
\put(27,-29){\makebox(0,0)[b]{{#3--}}}
% they used to be at height -29 and were aligned bottom
%\put(27,-29){\makebox(0,0)[b]{{#3\verb+-+}}}
%
\put(29,-29){\makebox(0,0)[r]{#3$y$}}
%
\put(-0.2,-1){\makebox(0,0)[r]{{#3\tt a}}}
\put(-0.2,-2){\makebox(0,0)[r]{{#3\tt b}}}
\put(-0.2,-3){\makebox(0,0)[r]{{#3\tt c}}}
\put(-0.2,-4){\makebox(0,0)[r]{{#3\tt d}}}
\put(-0.2,-5){\makebox(0,0)[r]{{#3\tt e}}}
\put(-0.2,-6){\makebox(0,0)[r]{{#3\tt f}}}
\put(-0.2,-7){\makebox(0,0)[r]{{#3\tt g}}}
\put(-0.2,-8){\makebox(0,0)[r]{{#3\tt h}}}
\put(-0.2,-9){\makebox(0,0)[r]{{#3\tt i}}}
\put(-0.2,-10){\makebox(0,0)[r]{{#3\tt j}}}
\put(-0.2,-11){\makebox(0,0)[r]{{#3\tt k}}}
\put(-0.2,-12){\makebox(0,0)[r]{{#3\tt l}}}
\put(-0.2,-13){\makebox(0,0)[r]{{#3\tt m}}}
\put(-0.2,-14){\makebox(0,0)[r]{{#3\tt n}}}
\put(-0.2,-15){\makebox(0,0)[r]{{#3\tt o}}}
\put(-0.2,-16){\makebox(0,0)[r]{{#3\tt p}}}
\put(-0.2,-17){\makebox(0,0)[r]{{#3\tt q}}}
\put(-0.2,-18){\makebox(0,0)[r]{{#3\tt r}}}
\put(-0.2,-19){\makebox(0,0)[r]{{#3\tt s}}}
\put(-0.2,-20){\makebox(0,0)[r]{{#3\tt t}}}
\put(-0.2,-21){\makebox(0,0)[r]{{#3\tt u}}}
\put(-0.2,-22){\makebox(0,0)[r]{{#3\tt v}}}
\put(-0.2,-23){\makebox(0,0)[r]{{#3\tt w}}}
\put(-0.2,-24){\makebox(0,0)[r]{{#3\tt x}}}
\put(-0.2,-25){\makebox(0,0)[r]{{#3\tt y}}}
\put(-0.2,-26){\makebox(0,0)[r]{{#3\tt z}}}
\put(-0.2,-27){\makebox(0,0)[r]{{#3--}}}
%\put(-0.2,-27){\makebox(0,0)[r]{{#3\verb+-+}}}
\put(-0.2,1){\makebox(0,0)[r]{#3$x$}}
\end{picture}
}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% used in ch 1:
\newcommand{\pB}{p_{\rm B}}
\newcommand{\pb}{p_{\rm b}}
% from theorems.tex for exact.tex
\newcommand{\PGB}{p^{\rm G}_{\rm B}}
\newcommand{\PGb}{p^{\rm G}_{\rm b}}
\newcommand{\PB}{p_{\rm B}}
\newcommand{\Pb}{p_{\rm b}}
%
% used in occam.tex (from nn_occam.tex)
\newlength{\minch}
\setlength{\minch}{0.82in}
\newcommand{\ostruta}{\rule[-0.07\minch]{0cm}{0.18\minch}}
\newcommand{\ostrutb}{\rule[-0.17\minch]{0cm}{0.14\minch}}
%
% sumproduct.tex
\newcommand{\gP}{P^*}
\newcommand{\xmwon}{\ensuremath{\bx_m \wo n}}
\newcommand{\xmwonb}{\ensuremath{\bx_{m \wo n}}}
% southeast.tex
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newcommand{\gridlet}[1]{\thinlines
\multiput(#1)(0,-2){4}{\line(1,0){7.22}}%
\multiput(#1)(2,0){4}{\line(0,-1){7.22}}}
%
\newcommand{\gridletfive}[1]{\thinlines
\multiput(#1)(0,-2){5}{\line(1,0){9.22}}%
\multiput(#1)(2,0){5}{\line(0,-1){9.22}}}
%
\newcommand{\piece}[1]{\put(#1){\circle*{0.872}}}
\newcommand{\opiece}[1]{\put(#1){\circle{0.872}}}
\newcommand{\movingpiece}[1]{%
\thinlines
\put(#1){\circle*{0.872}}
\put(#1){\vector(0,-1){2}}
\put(#1){\vector(1,0){2}}
}%end movingpiece
\newcommand{\lhnextposition}[2]{\hnextposition{#1}
\put(#1){\makebox(0,0)[bl]{\raisebox{2mm}{#2}}}}% labelled horizontal arrow
\newcommand{\ldnextposition}[2]{\dnextposition{#1}
\put(#1){\makebox(0,0)[tl]{\raisebox{0mm}{#2}}}}% labelled horizontal arrow
\newcommand{\hnextposition}[1]{\put(#1){\vector(1, 0){2}}}
\newcommand{\vnextposition}[1]{\put(#1){\vector(0, -1){2}}}
\newcommand{\dnextposition}[1]{\put(#1){\vector(-2,-1){4}}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% dfountain
\newcommand{\Ripple}{\ensuremath{S}}
% deconvoln.tex
\newcommand{\noisenu}{n}
% for _s13.tex and one_neuron
\newcommand{\hammingsymbol}[7]{\setlength{\unitlength}{1.4mm}%
\begin{picture}(1,2)(0,0)%
\ifnum #1=1 \put(0,2){\line(1,0){1}} \fi%
\ifnum #2=1 \put(0,2){\line(0,-1){1}} \fi%
\ifnum #3=1 \put(1,2){\line(0,-1){1}} \fi%
\ifnum #4=1 \put(0,1){\line(1,0){1}} \fi%
\ifnum #5=1 \put(0,1){\line(0,-1){1}} \fi%
\ifnum #6=1 \put(1,1){\line(0,-1){1}} \fi%
\ifnum #7=1 \put(0,0){\line(1,0){1}} \fi%
\end{picture}%
}
\newcommand{\hammingdigit}[1]{%
\ifnum #1=6 \hammingsymbol{0}{0}{0}{1}{0}{1}{1}\fi%
\ifnum #1=14 \hammingsymbol{0}{0}{1}{0}{1}{1}{1}\fi%
\ifnum #1=2 \hammingsymbol{0}{0}{1}{1}{1}{0}{0}\fi%
\ifnum #1=1 \hammingsymbol{0}{1}{0}{0}{1}{1}{0}\fi%
\ifnum #1=10 \hammingsymbol{0}{1}{0}{1}{1}{0}{1}\fi%
\ifnum #1=12 \hammingsymbol{0}{1}{1}{0}{0}{0}{1}\fi%
\ifnum #1=4 \hammingsymbol{0}{1}{1}{1}{0}{1}{0}\fi%
\ifnum #1=11 \hammingsymbol{1}{0}{0}{0}{1}{0}{1}\fi%
\ifnum #1=0 \hammingsymbol{1}{0}{0}{1}{1}{1}{0}\fi%
\ifnum #1=7 \hammingsymbol{1}{0}{1}{0}{0}{1}{0}\fi%
\ifnum #1=13 \hammingsymbol{1}{0}{1}{1}{0}{0}{1}\fi%
\ifnum #1=5 \hammingsymbol{1}{1}{0}{0}{0}{1}{1}\fi%
\ifnum #1=9 \hammingsymbol{1}{1}{0}{1}{0}{0}{0}\fi%
\ifnum #1=3 \hammingsymbol{1}{1}{1}{0}{1}{0}{0}\fi%
\ifnum #1=8 \hammingsymbol{1}{1}{1}{1}{1}{1}{1}\fi%
}
% here in binary order.
%6 &\hammingsymbol{0}{0}{0}{1}{0}{1}{1} \\
%14&\hammingsymbol{0}{0}{1}{0}{1}{1}{1} \\
%2 &\hammingsymbol{0}{0}{1}{1}{1}{0}{0} \\
%1 &\hammingsymbol{0}{1}{0}{0}{1}{1}{0} \\
%10&\hammingsymbol{0}{1}{0}{1}{1}{0}{1} \\
%12&\hammingsymbol{0}{1}{1}{0}{0}{0}{1} \\
%4 &\hammingsymbol{0}{1}{1}{1}{0}{1}{0} \\
%11&\hammingsymbol{1}{0}{0}{0}{1}{0}{1} \\
%0 &\hammingsymbol{1}{0}{0}{1}{1}{1}{0} \\
%7 &\hammingsymbol{1}{0}{1}{0}{0}{1}{0} \\
%13&\hammingsymbol{1}{0}{1}{1}{0}{0}{1} \\
%5 &\hammingsymbol{1}{1}{0}{0}{0}{1}{1} \\
%9 &\hammingsymbol{1}{1}{0}{1}{0}{0}{0} \\
%3 &\hammingsymbol{1}{1}{1}{0}{1}{0}{0} \\
%8 &\hammingsymbol{1}{1}{1}{1}{1}{1}{1} \\
\newcommand{\ldpcc}{low-density parity-check code}
%\newcommand{\Ldpc}{Low-density parity-check}% defined elsewhere
% included by l2.tex
% definitions for weighings.tex and for text
% shows weighing trees, ternary
%
% decisions of what to weigh are shown in square boxes with 126 over 345 (l:r)
% state of valid hypotheses are listed in double boxes
% or maybe dashboxes?
% three arrows, up means left heavy, straioght means right heavy, down is balance
%
\newcommand{\mysbox}[3]{\put(#1){\framebox(#2){\begin{tabular}{c}#3\end{tabular}}}}
\newcommand{\mydbox}[3]{\put(#1){\framebox(#2){\begin{tabular}{c}#3\end{tabular}}}}
\newcommand{\myuvector}[3]{\put(#1){\vector(#2){#3}}}
\newcommand{\mydvector}[3]{\put(#1){\vector(#2){#3}}}
\newcommand{\mysvector}[2]{\put(#1){\vector(1,0){#2}}}
\newcommand{\mythreevector}[4]{\myuvector{#1}{#2,#3}{#4}\mydvector{#1}{#2,-#3}{#4}\mysvector{#1}{#4}}
%
%\newcommand{\h1}{\mbox{$1^+$}}
%\newcommand{\l1}{\mbox{$1^-$}}
%\newcommand{\h2}{\mbox{$2^+$}}
%\newcommand{\l2}{\mbox{$2^-$}}
%\newcommand{\h3}{\mbox{$3^+$}}
%\newcommand{\l3}{\mbox{$3^-$}}
%\newcommand{\h4}{\mbox{$4^+$}}
%\newcommand{\l4}{\mbox{$4^-$}}
%\newcommand{\h5}{\mbox{$5^+$}}
%\newcommand{\l5}{\mbox{$5^-$}}
%\newcommand{\h6}{\mbox{$6^+$}}
%\newcommand{\l6}{\mbox{$6^-$}}
%\newcommand{\h7}{\mbox{$7^+$}}
%\newcommand{\l7}{\mbox{$7^-$}}
%\newcommand{\h8}{\mbox{$8^+$}}
%\newcommand{\l8}{\mbox{$8^-$}}
%\newcommand{\h9}{\mbox{$9^+$}}
%\newcommand{\l9}{\mbox{$9^-$}}
%\newcommand{\h10}{\mbox{$10^+$}}
%\newcommand{\l10}{\mbox{$10^-$}}
%\newcommand{\h11}{\mbox{$11^+$}}
%\newcommand{\l11}{\mbox{$11^-$}}
%\newcommand{\h12}{\mbox{$12^+$}}
%\newcommand{\l12}{\mbox{$12^-$}}
%\setlength{\parindent}{0mm}
\title{Information Theory, Inference, \& Learning Algorithms}
\shortlecturetitle{}
\shortauthor{David J.C. MacKay}
% the book - called by book.tex
%
% aiming for 696 pages total
%
% thebook.tex
% should run
% make book.ind
% by hand?
% Mon 7/10/02
\setcounter{exercise_number}{1} % set to imminent value
%
\setcounter{secnumdepth}{1} % sets the level at which subsection numbering stops
\setcounter{tocdepth}{0}
\newcommand{\mysetcounter}[2]{}%was {\setcounter{#1}{#2}}
% useful for forcing pagenumbers in drafts
%\setcounter{tocdepth}{1}
\renewcommand{\bs}{{\bf s}}
\newcommand{\figs}{/home/mackay/handbook/figs} % while in bayes chapter
% \addtocounter{page}{-1}
\pagenumbering{roman}
\setcounter{page}{2} % set to current value
\setcounter{frompage}{2}% this is used by newcommands1.tex dvips operator that helps make
\setcounter{page}{1} % set to current value
\setcounter{frompage}{1}% this is used by newcommands1.tex dvips operator that helps make
% individual chapters.
%
% PAGE ii
%
% \chapter*{Dedication}
%\input{tex/dedicationa.tex}
%\newpage
%
% TITLE PAGE iii
%
\thispagestyle{empty}
\begin{narrow}{0in}{-\margindistancefudge}%
\begin{raggedleft}
~\\[1.5in]
{\Large \bf Information Theory,
Inference,
and Learning Algorithms\\[1in]
}
{\Large\sf David J.C. MacKay }\\
\end{raggedleft}
\vfill
\mbox{}\epsfxsize=160pt\epsfbox{cuplogo.eps}% increased x size to compensate for 0.9 shrinkage later and another 10%
% \mbox{}\epsfxsize=128pt\epsfbox{cuplogo.eps}
\vspace*{-6pt}
\end{narrow}
\newpage
\thispagestyle{empty}
\begin{center}
~\\[1.5in]
{\Huge \bf Information Theory, \\[0.2in]
Inference,\\[0.2in]
and Learning Algorithms\\[1in]
}
{\Large\sf David J.C. MacKay }\\
{\tt{mackay@mrao.cam.ac.uk}}\\[0.3in]
\copyright 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004\\[0.1in]
\copyright Cambridge University Press 2003\\[1.3in]
Version \thedraft\ (third printing) \today\\
\medskip
\medskip
\medskip
\medskip
\medskip
Please send feedback on this book via
{\tt{http://www.inference.phy.cam.ac.uk/mackay/itila/}}
\medskip
\medskip
\medskip
Version 6.0 of this book was published by C.U.P.\ in September 2003.
It will remain viewable on-screen on the above website, in postscript, djvu,
and pdf formats.
\medskip
\medskip
In the second printing (version 6.6) minor typos were corrected,
and the book design was slightly altered to modify the placement of section numbers.
\medskip
\medskip
In the third printing (version 7.0) minor typos were corrected, and chapter 8 was
renamed `Dependent random variables' (instead of `Correlated').
\medskip
\medskip
\medskip
{\em (C.U.P. replace this page with their own page ii.)}
\end{center}
%\dvipsb{frontpage}
\newpage
% choose one of these:
% \input{cambridgefrontstuff.tex}
% \newpage
% {\em Page vi intentionally left blank.}
%
\newpage
% pages v and vi pages vii and viii
\mytableofcontents
\dvipsb{table of contents}
% alternate
%\fakesection{Roadmap}
%\input{roadmap.tex}
%
\subchaptercontents{Preface}%{How to Use This Book}% use subchapter because this
% marks the chapter name in the header, unlike chapter*{}
% \section*{How to use this book}
%{\em [This front matter is still being written. The remainder of the book is essentially finished,
% except for typographical corrections, April 18th 2003.]}
%
% a longer version of this is in
% longabout.tex
% \section*{How to use this book}
% \section{How to use this book}
% The first question we must address is:
This book is aimed at senior undergraduates and graduate students in
Engineering, Science, Mathematics, and Computing. It expects
familiarity with calculus, probability theory, and
linear algebra as taught in a first- or second-year
undergraduate course on mathematics for
scientists and engineers.
Conventional courses on information theory
cover not only the beautiful {\em theoretical\/} ideas of Shannon,
but also {\em practical\/} solutions to \ind{communication} problems.
This book
goes further, bringing in Bayesian data modelling,
Monte Carlo methods, variational
methods, clustering algorithms, and neural networks.
Why unify information theory and
machine learning?
% Well,
Because they
% Information theory and
% machine learning
are two sides of the same coin.
% , so it makes sense to unify them.
% These two fields were once unified:
% It was once so:
In the 1960s, a single field, cybernetics, was populated
by information theorists, computer scientists, and neuroscientists, all
studying common problems.
Information theory and machine learning still belong together.
Brains are the ultimate compression and \ind{communication} systems.
And the state-of-the-art algorithms
for both data compression and error-correcting codes
use the same tools as machine learning.
% Our brains are surely the ultimate in robust
% error-correcting information storage and recall systems.
\section*{How to use this book}
The essential dependencies between chapters are indicated in
the figure on the next page. An arrow from one chapter to another
indicates that the second chapter requires some of the first.
%\section*{General points}
% The pinnacles of the book, the key chapters with the really exciting bits,
% are first \chref{chone} (in which we meet Shannon's noisy-channel coding theorem);
% \chref{ch.six} (in which we prove it); \chref{ch.hopfield} (in which
% we meet a neural network that performs robust error-correcting
% content-addressable memory); and Chapters \ref{ch.ldpcc} and \ref{chdfountain}
% (in which we meet beautifully simple sparse-graph codes that solve
% Shannon's communication problem).
%% honorable mention - \chref{ch.ac}, ch.ra /////\ exact sampling - not central.
% Do not feel daunted by this book.
% You don't need to read all of this book.
Within {\partnoun}s \datapart, \noisypart, \probpart, and \netpart\ of this book, chapters
on advanced or optional topics are
towards the end.
% For example, \chref{ch.codesforintegers} (Codes for Integers), \chref{ch.xword} (Crosswords and Codebreaking)
% and \chref{ch.sex} (Why have Sex? Information Acquisition and Evolution)
% are provided for fun.
All chapters of {\partnoun} \finfopart\
are optional on a first reading, except perhaps for
\chref{ch.message} (Message Passing).
The same system sometimes applies within a chapter:
the final sections often deal with
advanced topics that can be skipped on a first reading.
For example in two key chapters --
\chref{chtwo} ({The Source Coding Theorem}) and \chref{ch.six} ({The Noisy-Channel Coding Theorem}) --
the first-time reader should detour
at \secref{sec.chtwoproof} and \secref{sec.ch6stop} respectively.
% \subsection*{Roadmaps}
Pages \pageref{map1}--\pageref{map4} show a few ways to use this book.
First, I give the roadmap for a course that I teach in Cambridge:
% which embraces both information theory and machine learning.
`Information theory, pattern recognition, and neural networks'.
%
The book is also intended as a textbook for
traditional courses in information theory.
The second roadmap
shows the chapters for
an introductory information theory course
and the third
for a course aimed at an understanding of
state-of-the-art error-correcting codes.
%
The fourth roadmap shows how to use the text in a
conventional course on machine learning.
% The diagrams on the following pages will indicate
% the dependences between chapters and
% a few possible routes through the book.
\newpage
\begin{center}\hspace*{-0.2cm}\raisebox{2cm}{\epsfbox{metapost/roadmap.2}}\end{center}
\newpage
% \input{tex/cambroadmap.tex}
% \newpage
\begin{center}\raisebox{2cm}{\epsfbox{metapost/roadmap.3}}\end{center}
\label{map1}
\newpage
\begin{center}\raisebox{2cm}{\epsfbox{metapost/roadmap.4}}\end{center}
\newpage
\begin{center}\raisebox{2cm}{\epsfbox{metapost/roadmap.5}}\end{center}
\newpage
\begin{center}\raisebox{2cm}{\epsfbox{metapost/roadmap.6}}\end{center}
\label{map4}
\newpage
\section*{About the exercises}
% I firmly believe that
You can understand a subject only by
creating it for yourself.
% To this end, you should
% I think it is essential to
The exercises
play an essential role in this book.
% on each topic.
For guidance, each
% exercise
has a rating (similar to that used by \citeasnoun{KnuthAll})
from 1 to 5 to indicate its difficulty.
\noindent\ratfull\hspace*{\parindent}In addition, exercises that are especially recommended
are marked by a marginal encouraging rat.
Some exercises that require the use of a computer are
marked with a {\sl C}.
% will have
% a rating such as A1, A5, C1 or C5.
% The letter indicates how important I think the exercise is:
% A = very important $\ldots$ C = not essential to the flow of the
% book. The number indicates the difficulty of the problem:
% 1 = easy, 5 = research project.
% I'll circulate detailed recommendations on exercises
% as the course progresses.
Answers to many exercises are provided. Use them
wisely. Where a solution is provided, this is indicated
by including its page number
% of the solution with
alongside the difficulty rating.
Solutions to many of the other exercises
will be supplied to instructors using this book in their
teaching; please email {\tt{solutions@cambridge.org}}.
%\begin{table}[htbp]
%\caption[a]
\begin{realcenter}
\fbox{
\begin{tabular}{ll}
%\begin{minipage}{3in}
{\sf Summary of codes for exercises}\\[0.2in]
% \hspace{0.2in}
\begin{tabular}[b]{cl}
\dorat & Especially recommended \\[0.2in]
{\ensuremath{\triangleright}} & Recommended \\
{\sl C} & Parts require a computer \\
{\rm [p.$\,$42]}& Solution provided on page 42 \\
\end{tabular}
%\end{minipage}
&
\begin{tabular}[b]{cl}
\pdifficulty{1} & Simple (one minute) \\
\pdifficulty{2} & Medium (quarter hour) \\
\pdifficulty{3} & Moderately hard \\
\pdifficulty{4} & Hard \\
\pdifficulty{5} & Research project \\[0.2in]
\end{tabular}
\\
\end{tabular}
}
\end{realcenter}
%\end{table}
\section*{Internet resources}
The website
\begin{realcenter}
{\tt{http://www.inference.phy.cam.ac.uk/mackay/itila}}
\end{realcenter}
contains several resources:
\ben
\item
{\em Software}.
Teaching software that I use in lectures,\index{software}
interactive software, and research software,
written in {\tt{perl}}, {\tt{octave}}, {{\tt{tcl}}}, {\tt{C}}, and {\tt{gnuplot}}.
Also some animations.
\item
{\em Corrections to the book}. Thank you in advance for emailing these!
\item
{\em This book}.
The book is provided in {\tt{postscript}}, {\tt{pdf}}, and {\tt{djvu}}
formats for on-screen viewing. The same copyright restrictions
apply as to a normal book.
% \item
% {\em Further worked solutions to some exercises}.
% If you would like to send in your own solutions for inclusion,
% please do.
\een
% {\em (I aim to add a table of software resources here.)}
\section*{About this edition}
This is the third printing of the first edition.
In the second printing,
% a small number of typographical errors were corrected,
% and
the design of the book was altered slightly.
% to allow a slightly larger font size.
Page-numbering generally remains unchanged,
% consistent between the two printings,
except in chapters 1, 6, and 28,
where
% with the exception of pages 7 to 13, where
% among which
a few paragraphs, figures, and equations have
moved around.
% on which text, figures, and equations have all been slightly rearranged.
All equation, section, and exercise numbers are unchanged.
In the third printing, chapter 8 has been renamed
`Dependent Random Variables', instead of `Correlated', which was sloppy.
% BEWARE, _RNGaussian.tex had to be changed for the asides.
%\input{tex/secondprint.tex}% about the second printing
\section*{Acknowledgments}
%\chapter*{Acknowledgments}
I am most grateful to the organizations who have supported
me while this book gestated: the Royal Society and Darwin College
who gave me a fantastic research fellowship
in the early years; the University of Cambridge;
the Keck Centre at the University of California in San Francisco,
where I spent a productive sabbatical;
% (and failed to finish the book);
and
the Gatsby Charitable Foundation, whose support gave me the
freedom to break out of the Escher staircase that book-writing had become.
My work has depended on the generosity of free software authors.\index{software!free}\index{Knuth, Donald}
I wrote the book in \LaTeXe. Three cheers for Donald Knuth and Leslie Lamport!
%\nocite{latex}
Our computers run the GNU/Linux operating system. I use {\tt{emacs}}, {\tt{perl}}, and
{\tt{gnuplot}} every day. Thank you Richard Stallman, thank you Linus Torvalds,
thank you everyone.
% I thank David Tranah of Cambridge University Press for his editorial support.
% ``cut, it's my job''
Many readers, too numerous to name here,
have given feedback on the book, and to
them all I extend my sincere acknowledgments.
%
I especially wish to thank all the students and colleagues
at Cambridge University who have attended my lectures on
information theory and machine learning over the last nine years.
% Without their enthusiasm and criticism, this book would surely
The members of the Inference research group have given immense support,
and I thank them all for their generosity and patience over the last ten years:
Mark Gibbs, Michelle Povinelli, Simon Wilson, Coryn Bailer-Jones, Matthew Davey,
Katriona Macphee, James Miskin, David Ward, Edward Ratzer, Seb Wills, John Barry,
John Winn, Phil Cowans, Hanna Wallach, Matthew Garrett, and especially Sanjoy Mahajan.
Thank you too to Graeme Mitchison, Mike Cates, and Davin Yap.
Finally I would like to express my debt to my personal heroes,
the mentors from whom I have learned so much:
Yaser Abu-Mostafa,
Andrew Blake,
John Bridle,
Peter Cheeseman,
Steve Gull,
Geoff Hinton,
John Hopfield,
Steve Luttrell,
Robert MacKay,
Bob McEliece,
Radford Neal,
Roger Sewell,
and
John Skilling.
%%%%%%%%%%%%%%
%\chapter*{Dedication}
%\vspace*{80pt}
\vfill
\begin{center}
\rule{\textwidth}{1pt} \par \vskip 18pt
{ \huge \sl
{Dedication} }
\par
%\end{center}
\nobreak \vskip 40pt
%\begin{center}
This book is dedicated to the campaign against the arms trade.\\[0.3in]
%
% Their web page is
% , as overburdened with animated images as the world is with weapons, is here:
%\verb+http://www.caat.demon.co.uk/+\\[0.6in]
\verb+www.caat.org.uk+\\[0.6in]
\end{center}
\begin{quote}
\begin{raggedleft}
Peace cannot be kept by force.\\
It can only be achieved
% by understanding.
% Peace cannot be achieved through violence, it can only be attained
through understanding.\\
\hfill -- {\em Albert Einstein}\\
\end{raggedleft}
\end{quote}
\vspace*{2pt}
\rule{\textwidth}{1pt} \par
% Two things are infinite: the universe and human stupidity; and I'm not sure
% about the the universe.
%The important thing is not to stop questioning. Curiosity has its own reason for
% existing.
%Any intelligent fool can make things bigger, more complex, and more violent. It
% takes a touch of genius -- and a lot of courage -- to move in the opposite
% direction.
% \input{extrafrontstuff.tex}% aims dedication, about the author, etc
% see also tex/oldaims.tex
% for some good stuff.
% and tex/typicalreaders.tex
%
%% \input{tex/overview2001.tex}
%\dvipsb{preface}
\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\setcounter{page}{0} % set to current value
%Fake page % added to get draft.dps to look right
%\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\pagenumbering{arabic}
\prechapter{About Chapter}
\setcounter{page}{1} % set to current value
\label{pch.one}
%
% pre-chapter 1
%
\fakesection{Before ch 1}
In the first chapter, you will need to be familiar with the \ind{binomial distribution}.
% , reviewed below.
And to solve the exercises in the text --
which I urge you to do -- you will need to know {\dem\ind{Stirling's
approximation}\/}\index{approximation!Stirling}
for the factorial function, $%\beq
x! \simeq x^{x} \, e^{-x}
$,
and be able to
apply it to ${{N}\choose{r}} =
\smallfrac{N!}{(N-r)!\,r!}$.\marginpar{\small\raggedright{Unfamiliar notation?\\ See
\appref{app.notation}, \pref{app.notation}.}}
% $x!$
These topics are reviewed below.
\subsection*{The binomial distribution}
\label{sec.first.binomial}
\exampl{ex.binomial}{
A bent coin has probability $f$ of coming up heads.
The coin is tossed $N$ times.
What is the probability
distribution of the number of heads, $r$?
What are the \ind{mean} and \ind{variance} of $r$?
}
\amarginfig{t}{%
\begin{tabular}{r}
% $P(r\given f,N)$\\
\mbox{\psfig{figure=bigrams/urn.f.g.ps,angle=-90,width=1.51in}}%
%\\
%\mbox{\psfig{figure=bigrams/urn.f.l.ps,angle=-90,width=1.64in}}%
\\[-0.1in]
\multicolumn{1}{c}{\small$r$}
\\
\end{tabular}
%}{%
\caption[a]{The binomial distribution $P(r \given f\eq 0.3,\,N \eq 10)$.}
% , on a linear scale (top) and a logarithmic scale (bottom).}
\label{fig.binomial}
}
% see bigrams/README
\noindent
%\begin{Sexample}{ex.binomial}
{\sf Solution\colonspace}
\label{sec.first.binomial.sol}
The number of heads
has a binomial distribution.
\beq P(r \given f,N) = {N \choose r} f^{r} (1-f)^{N-r} . \eeq
The mean, $\Exp [ r ]$, and variance, $\var[r]$,
of this distribution are
defined by
\beq
\Exp [ r ] \equiv \sum_{r=0}^{N} P(r\given f,N) \, r
\label{eq.mean.def}
\eeq
\beqan
\var[r] & \equiv &
\Exp \left[ \left( r - \Exp [ r ] \right)^2 \right] \\
& = &
\Exp [ r^2 ] - \left( \Exp [ r ] \right)^2
= \sum_{r=0}^{N} P(r\given f,N) r^2 - \left( \Exp [ r ] \right)^2 .
\label{eq.var.sum}
\eeqan
%
Rather than evaluating the sums over $r$ in (\ref{eq.mean.def}) and (\ref{eq.var.sum}) directly,
it is easiest to obtain the mean and variance by noting that $r$
is the sum of $N$ {\em independent\/}
% , identically distributed
random variables, namely, the number of heads in the
first toss (which is either zero or one),
the number of heads in the second toss, and so forth.
In general,
\beq
\begin{array}{rcll}
\Exp [ x + y ] &=& \Exp [ x ] + \Exp [ y ] & \mbox{for any random variables $x$ and $y$};
\\
\var [ x + y ] &=& \var [ x ] + \var [ y ] & \mbox{if $x$ and $y$ are independent}.
\end{array}
\eeq
So the mean of $r$ is the sum of the means of those random
variables, and the variance of $r$ is the sum of their variances.\index{variances add}
% its mean and variance are given by adding the means and variances
% of those random variables, respectively.
The mean number of heads in a single toss
is $f\times 1 + (1-f)\times 0 = f$, and the variance of the
number of heads in a single toss is
\beq
\left[ f\times 1^2 + (1-f)\times 0^2 \right] - f^2 = f - f^2 = f(1-f),
\eeq
so the mean and variance of $r$ are:
\beq \Exp [ r ] = N f
%\eeq\beq
\hspace{0.35in} \mbox{and} \hspace{0.35in}
\var[r] = N f (1-f) . \hspace{0.35in}\epfsymbol\hspace{-0.35in}
\eeq
%\end{Sexample}
% ADD END PROOF SYMBOL HERE !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
\subsection*{Approximating $x!$ and ${{N}\choose{r}}$}
\amarginfig{t}{%
\begin{tabular}{r}
\mbox{\psfig{figure=bigrams/poisson.g.ps,angle=-90,width=1.5in}}%
%\\
%\mbox{\psfig{figure=bigrams/poisson.l.ps,angle=-90,width=1.64in}}%
\\[-0.1in]
\multicolumn{1}{c}{\small$r$}
\\
\end{tabular}
%}{%
\caption[a]{The Poisson distribution $P(r\,|\,\l\eq 15)$.}
% , on a linear scale (top) and a logarithmic scale (bottom).}
\label{fig.poisson}
}
% see bigrams/README
\label{sec.poisson}
% FAVOURITE BIT
\noindent
Let's derive Stirling's approximation by an unconventional route.
We start from the \ind{Poisson distribution} with mean $\l$,
\beq
P( r \given \l ) = e^{-\l} \frac{\l^r}{r!} \:\:\:\:
\:\: r\in \{ 0,1,2,\ldots\} .
\label{eq.poisson}
\eeq
%
% \noindent
For large $\l$, this distribution is well approximated -- at least\index{approximation!by Gaussian}
in the vicinity of $r \simeq \l$ -- by
a \ind{Gaussian distribution} with mean $\l$ and variance $\l$:
% So,
\beq
e^{-\l} \frac{\l^r}{r!} \,\simeq\, \frac{1}{\sqrt{2\pi \l}}
\, e^{{ -\smallfrac{(r-\l)^2}{2\l}}} .
\eeq
Let's plug $r=\l$ into this formula.\label{sec.stirling}
\beqan
e^{-\l} \frac{\l^{\l}}{\l!} &\simeq& \frac{1}{\sqrt{2\pi \l}}
\\
\Rightarrow \l! &\simeq& \l^{\l} \, e^{-\l} \sqrt{2\pi \l} .
\eeqan
This is {Stirling's approximation}
for the \ind{factorial} function.
\beq
x! \,\simeq\, x^{x} \, e^{-x} \sqrt{2\pi x} \:\:\:\Leftrightarrow\:\:\:
\ln x! \,\simeq\, x \ln x - x + {\textstyle\frac{1}{2}} \ln {2\pi x} .
\label{eq.stirling}
\eeq
We have derived not only the
leading order behaviour, $x! \simeq x^{x} \, e^{-x}$,
but also, at no cost, the next-order correction
term $\sqrt{2\pi x}$.
%
We now apply Stirling's approximation
% the approximation
%$%\beq
% x! \simeq x^{x} \, e^{-x} $
to\index{combination}
$%\beq
\ln {{N}\choose{r}}
$:%\eeq
\beqan
\ln {{N}\choose{r}}
\,\equiv\, \ln \frac{N!}{(N-r)!\,r!}
% & \simeq &
% N [ \ln N - 1 ] - (N-r) [ \ln (N-r) - 1 ] - r [ \ln r - 1 ]
%\\
& \simeq & (N-r) \ln\frac{N}{N-r} + r \ln\frac{N}{r}
.
\label{eq.choose.approx}
\eeqan
Since all the terms in this equation are logarithms,
this result can be rewritten in any base.\marginpar{\small Recall that
$\displaystyle{ \log_2 x = \frac{ \log_e x }{ \log_e 2} }$.\\[0.03in]
Note that $\displaystyle\frac{\partial \log_2 x }{\partial x} =
\frac{1}{\log_e 2}\,\frac{1}{x}$.
}
%\fakesubsection*{My rule about log and ln}
We will denote\index{conventions!logarithms}\index{notation!logarithms}
natural logarithms ($\log_e$) by `ln', and \ind{logarithms}
to base 2 ($\log_2$)
by `$\log$'.
If we introduce the {\dbf\ind{binary entropy function}},
\beq
H_2(x) \equiv x \log \frac{1}{x} + (1\! -\! x) \log \frac{1}{(1\! -\! x)} ,
\eeq
then we can rewrite the approximation (\ref{eq.choose.approx})
%\beq
%$ \log {{N}\choose{r}}
% \simeq (N-r) \log \frac{N}{N-r} + r \log \frac{N}{r}
%$
%\eeq
as
\amarginfig{t}{\small%
\begin{center}
\mbox{
\hspace{-6mm}
% \hspace{6.2mm}
\raisebox{\hpheight}{$H_2(x)$}
% to put H at left:
\hspace{-7.5mm}
% \hspace{-20mm}
\mbox{\psfig{figure=figs/H2.ps,%
width=42mm,angle=-90}}$x$
}
% see also H2p.tex
\end{center}
\caption[a]{The binary entropy function.}
% $H_2(x)$.}
\label{fig.h2x}
}
\beq
\log {{N}\choose{r}}
\, \simeq \, N H_2(r/N) ,
\label{eq.stirling.choose.l}
\eeq
or, equivalently,
% \:\:\:\Leftrightarrow\:\:\:
\beq
{{N}\choose{r}}
\, \simeq \, 2^{N H_2(r/N)} .
\label{eq.stirling.choose}
\eeq
If we need a more accurate approximation, we
can include terms of the next order from
Stirling's approximation
(\ref{eq.stirling}):
\beq
\log {{N}\choose{r}}
\,\simeq\, N H_2(r/N) -
{\textstyle\frac{1}{2}} \log \left[ {2\pi N \, \frac{N\!-\!r}{N} \,
\frac{r}{N}} \right]
.
\label{eq.H2approxaccurate}
\eeq
%
% - {\textstyle\frac{1}{2}} \ln {2\pi N}
% + {\textstyle\frac{1}{2}} \ln {2\pi N-r}
% + {\textstyle\frac{1}{2}} \ln {2\pi r}
%
% ln += {\textstyle\frac{1}{2}} \ln {2\pi (N-r)(r)/N}
% log_2 += {\textstyle\frac{1}{2}} \log_2 {2\pi (N-r)(r)/N}
% or
% log_2 += {\textstyle\frac{1}{2}} \log_2 {2\pi N}
% + {\textstyle\frac{1}{2}} \log_2 {\frac{(N-r)}{N}\frac{r}{N}}
% log_2 += {\textstyle\frac{1}{2}} \log_2 {2\pi \frac{(N-r)}{N}\frac{r}{N} N}
\ENDprechapter
\chapter{Introduction to Information Theory}
\label{ch.one}
\label{chone}
% % \part{Information Theory}
% \chapter{Introduction to Information Theory}
\label{ch1}
%\section{Communication over noisy channels}
% One of the principal questions addressed by information theory is
% Shannon's ground-breaking paper on `The Mathematical Theory of
% Communication' opens thus:
\begin{quotation}
\noindent
The fundamental problem of \index{communication} is that of reproducing at one point
either exactly or approximately a message selected at another point.
\\
\mbox{~} \hfill {\em (Claude Shannon, 1948)}\index{Claude Shannon} \\
%
\end{quotation}
\noindent
In the first half of
this book we
%are going to
study how to measure information content;
we
% are going to
% learn by how much data from a given source
% can be compressed; we
% are going to
learn how
% , practically, to
% achieve data compression;
to compress data; and we
% are going to
learn how to communicate
perfectly over imperfect communication channels.
We start by getting a feeling for this last problem.
\section[How can we achieve perfect communication?]{How
can we achieve perfect communication over an imperfect, noisy
communication channel?}
Some examples of noisy communication channels are:
\bit
\item
an analogue telephone
line,\marginpar{\footnotesize
\setlength{\unitlength}{1mm}%
\begin{picture}(45,10)(0,5)
\put(0,10){\makebox(0,0)[l]{\shortstack{modem}}}
\put(21,10){\makebox(0,0)[l]{\shortstack{phone\\line}}}
\put(39,10){\makebox(0,0)[l]{\shortstack{modem}}}
\put(15,10){\vector(1,0){3}}
\put(32,10){\vector(1,0){3}}
\end{picture}
}
over which two modems communicate digital information;
\item
the radio communication link from
Galileo,\marginpar{\footnotesize
\setlength{\unitlength}{1mm}%
\begin{picture}(45,10)(0,5)
\put(0,10){\makebox(0,0)[l]{\shortstack{Galileo}}}
\put(21,10){\makebox(0,0)[l]{\shortstack{radio\\waves}}}
\put(39,10){\makebox(0,0)[l]{\shortstack{Earth}}}
\put(15,10){\vector(1,0){3}}
\put(32,10){\vector(1,0){3}}
\end{picture}
}
the Jupiter-orbiting spacecraft,
to earth;
\item
\marginpar[c]{\footnotesize
\setlength{\unitlength}{1mm}%
\begin{picture}(30,20)(0,0)
\put(0,10){\makebox(0,0)[l]{\shortstack{parent\\cell}}}
\put(16,2){\makebox(0,0)[l]{\shortstack{daughter\\cell}}}
\put(16,16){\makebox(0,0)[l]{\shortstack{daughter\\cell}}}
\put(10,10){\vector(1,1){5}}
\put(10,10){\vector(1,-1){5}}
\end{picture}
}reproducing cells, in which the daughter cells' \ind{DNA}
contains information from the parent
% cell or
cells;
\item
\marginpar{\footnotesize
\setlength{\unitlength}{1mm}%
\begin{picture}(45,10)(0,5)
\put(0,10){\makebox(0,0)[l]{\shortstack{computer\\ memory}}}
\put(20,10){\makebox(0,0)[l]{\shortstack{\disc\\drive}}}
\put(33,10){\makebox(0,0)[l]{\shortstack{computer\\ memory}}}
\put(15,10){\vector(1,0){3}}
\put(29,10){\vector(1,0){3}}
\end{picture}
}a \disc{} drive.
\eit
The last example shows that \ind{communication} doesn't have to involve
information going from one {\em place\/} to another. When
we write a file on a \disc{} drive, we'll
% typically
read it off
% again
in the same location -- but at a later {\em time}.
These channels are noisy.\index{noise}\index{channel!noisy} A telephone line suffers
from cross-talk with other lines; the hardware in the
line distorts and adds noise to the transmitted signal. The deep
space network that listens to Galileo's puny transmitter
% fairy-bulb power
receives background radiation from
terrestrial and cosmic sources.
DNA is subject to mutations and damage.
A \ind{disk drive}, which writes
a binary digit (a one or zero, also known as a {\dbf bit}) by aligning a patch of magnetic
material in one of two orientations, may later
% , with some probability,
fail to read out the stored binary digit:
% that was stored
the patch of material might spontaneously flip
magnetization, or
a glitch of
background noise might cause the reading circuit
to report the wrong
value for the binary digit, or the writing head might not induce
the magnetization in the first place because of interference
from neighbouring bits.
In all these cases, if we transmit data, \eg, a string
of bits, over the channel, there is some probability that
the received message will not be identical to the transmitted message.
% And in all cases,
We would prefer to have a communication channel for
which this probability was zero -- or so close to zero that
for practical purposes it is indistinguishable from zero.
Let's consider
% the example of
a noisy \disc{} drive
% having the property
that transmits each bit correctly
% transmitted
with probability
$(1\!-\!f)$ and incorrectly with probability $f$.
This model
% favourite
communication channel\index{channel!binary symmetric} is known
as the {\dbf{\ind{binary symmetric channel}}} (\figref{fig.bsc1}).
\begin{figure}[htbp]
\figuremargin{%
\[
\begin{array}{c}
\setlength{\unitlength}{0.46mm}
\begin{picture}(30,20)(-5,0)
\put(-4,9){{\makebox(0,0)[r]{$x$}}}
\put(5,2){\vector(1,0){10}}
\put(5,16){\vector(1,0){10}}
\put(5,4){\vector(1,1){10}}
\put(5,14){\vector(1,-1){10}}
\put(4,2){\makebox(0,0)[r]{1}}
\put(4,16){\makebox(0,0)[r]{0}}
\put(16,2){\makebox(0,0)[l]{1}}
\put(16,16){\makebox(0,0)[l]{0}}
\put(24,9){{\makebox(0,0)[l]{$y$}}}
\end{picture}
\end{array}
\:\:\:
\begin{array}{ccl}%%%%% {c@{}c@{}l} %%%%% (for twocolumn style)
P(y\eq 0 \given x\eq 0) &= & 1 - \q ; \\ P(y\eq 1 \given x\eq 0) &= & \q ;
\end{array}
\begin{array}{ccl}
P(y\eq 0 \given x\eq 1) &= & \q ; \\ P(y\eq 1 \given x\eq 1) &= & 1 - \q .
\end{array}
\]
}{%
\caption[a]{The binary symmetric channel. The
transmitted symbol is $x$ and the
received symbol $y$. The noise level, the probability
% of a bit's being
that a bit is
flipped, is $f$.}
\label{fig.bsc1}
}%
\end{figure}
\begin{figure}[htbp]
\figuremargin{%
\begin{mycenter}
\begin{tabular}{rcl}
\psfig{figure=bitmaps/dilbert.ps,width=1.2in}
&\hspace{0.1in}%
\raisebox{0.22in}{%
\setlength{\unitlength}{1.2mm}%
\begin{picture}(20,20)(0,0)%
\put(10,1){\makebox(0,0)[t]{$(1-f)$}}
\put(10,17){\makebox(0,0)[b]{$(1-f)$}}
\put(12,9.5){\makebox(0,0)[l]{$f$}}
% \put(10,16.5){\makebox(0,0)[b]{$(1-f)$}}
\put(5,2){\vector(1,0){10}}
\put(5,16){\vector(1,0){10}}
\put(5,4){\vector(1,1){10}}
\put(5,14){\vector(1,-1){10}}
\put(4,2){\makebox(0,0)[r]{{1}}}
\put(4,16){\makebox(0,0)[r]{{0}}}
\put(16,2){\makebox(0,0)[l]{{1}}}
\put(16,16){\makebox(0,0)[l]{{0}}}
\end{picture}%
}%
\hspace{0.385in}&
\psfig{figure=_is/10000.10.ps,width=1.2in} \\
% & & \makebox[0in][l]{\large 10\% of bits are flipped} \\
\end{tabular}
\end{mycenter}
}{%
\caption[a]{A binary data sequence of length $10\,000$ transmitted over
a binary symmetric channel with noise level $f=0.1$.
\dilbertcopy}
\label{fig.bsc.dil}
}%
\end{figure}
\noindent
As an example,
% For the sake of argument,
let's imagine that $f=0.1$, that is, ten \percent\ of the bits are
flipped (figure \ref{fig.bsc.dil}).
% For a \disc{} drive to be useful, we would prefer that it should
% flip no bits at all in its entire lifetime.
A useful \disc{} drive would flip no bits at all in its entire lifetime.
%
If we expect to read and write a
gigabyte per day for ten years, we require a bit error
probability of the order of $10^{-15}$, or smaller.
There are two approaches to this goal.
\subsection{The physical solution}
The physical solution is to improve the physical characteristics of
the communication channel to reduce its error probability. We could
improve our \disc{} drive by
% , for example,
\ben
\item
using more reliable components in its circuitry;
\item
evacuating the air from the \disc{} enclosure so as
to eliminate the turbulence that perturbs the
reading head from the track;
\item
using a larger magnetic patch to represent each bit; or
\item
using higher-power signals or cooling the
circuitry in order to reduce thermal noise.
\een
These physical modifications
typically
increase the cost of the communication
channel.
% unit of area making the \disc{} spin at a slower rate
%
% the system solution
%
\begin{figure}%[htbp]
\figuremargin{%
\setlength{\unitlength}{1.25mm}
\begin{mycenter}
\begin{picture}(50,40)(-10,5)
\put(0,5){\framebox(25,10){\begin{tabular}{c}Noisy\\ channel\end{tabular}}}
\put(-20,20){\framebox(25,10){\begin{tabular}{c}Encoder\end{tabular}}}
\put(20,20){\framebox(25,10){\begin{tabular}{c}Decoder\end{tabular}}}
%\put(-20,40){\framebox(25,10){\begin{tabular}{c}Compressor\end{tabular}}}
%\put(20,40){\framebox(25,10){\begin{tabular}{c}Decompressor\end{tabular}}}
%\put(-50,20){\makebox(25,10){\begin{tabular}{c}{\sc Source}\\{\sc coding}\end{tabular}}}
% \put(-50,40){\makebox(25,10){\begin{tabular}{c}{\sc Channel}\\{\sc coding}\end{tabular}}}
\put(-20,37){\makebox(25,12){Source}}
%
\put(-10,14){\makebox(0,0){$\bt$}}
\put(-10,34){\makebox(0,0){$\bs$}}
\put(35,14){\makebox(0,0){$\br$}}
\put(35,34){\makebox(0,0){$\hat{\bs}$}}
\put(-7.5,18){\line(0,-1){8}}
\put(-7.5,10){\vector(1,0){6}}
\put(32.5,10){\vector(0,1){8}}
\put(32.5,10){\line(-1,0){6}}
%
\put(32.5,31){\vector(0,1){8}}
%\put(32.5,51){\vector(0,1){5}}
\put(-7.5,39){\vector(0,-1){8}}
%\put(-7.5,55){\vector(0,-1){5}}
\end{picture}
\end{mycenter}
}{%
\caption[a]{The `system' solution for
achieving
% almost perfect
reliable communication
over a noisy channel. The encoding system introduces
systematic redundancy
% in a systematic way
into the transmitted vector $\bt$. The decoding system
uses this known redundancy to deduce
from the
received vector $\br$
{\em both\/}
the original source vector
{\em and\/}
the noise introduced by the channel.
}
\label{system.solution}
}%
\end{figure}
\subsection{The `system' solution}
Information theory\index{information theory} and
\ind{coding theory}\index{system} offer
an alternative (and much more exciting)
approach: we accept the given noisy channel as it is
and
add communication {\dem systems\/} to it so that we
can {detect\/} and {correct\/} the errors introduced by the
% noise.
channel.
As shown in \figref{system.solution}, we add an
{\dem\ind{encoder}\/} before the channel and a {\dem\ind{decoder}\/} after
it. The encoder encodes the source message $\bs$
into a {\dem transmitted\/} message $\bt$,
% the idea is that the encoder adds
adding {\dem\ind{redundancy}\/} to the original message in some way. The
channel adds noise to the transmitted message, yielding a received
message $\br$. The decoder uses the known redundancy
introduced by the encoding system to infer both the original signal
$\bs$ and the added noise.
% added by the channel was.
Whereas physical solutions give incremental channel improvements
only at an ever-increasing cost,
% we hope to find
% there exist
system solutions can turn noisy channels into reliable
communication channels
with the only cost being a {\em computational\/} requirement
at the encoder and decoder.
% (and the delay associated with those computations.
%
% suggested addition:
% So, as the cost of computation falls, the cost of reliability will fall as well.
{\dbf Information theory} is concerned with the theoretical limitations and
% theoretical
potentials of such systems. `What is the best error-correcting
performance we could achieve?'
{\dbf Coding theory} is concerned with the creation of practical
encoding and decoding systems.
% Some
\section{Error-correcting codes for the binary symmetric channel}
We now consider examples of encoding and decoding systems.
What is the simplest way to add useful redundancy to a transmission?
[To make the rules of the game clear:
we want to be able to detect {\em and\/} correct errors;
and retransmission is not an option. We get only
one chance to encode, transmit,
and decode.]
\subsection{Repetition codes}
\label{sec.r3}
A straightforward idea is to repeat every bit of the message a prearranged
number of times -- for example, three times, as shown in \tabref{fig.r3}.
We call this {\dem \ind{repetition code}\/} `$\Rthree$'.
%\begin{figure}[htbp]
%\figuremargin{%
\amargintab{c}{
\begin{mycenter}
\begin{tabular}{c@{\hspace{0.3in}}c} \toprule % \hline
% Source sequence $\bs$ & Transmitted sequence $\bt$ \\ \hline
Source & Transmitted \\[-0.02in] % was -0.1, which was to much
sequence & sequence \\
$\bs$ & $\bt$ \\ \midrule % \hline
\tt 0 &\tt 000 \\
\tt 1 &\tt 111 \\ \bottomrule % \hline
\end{tabular}
\end{mycenter}
%}{%
\caption[a]{The repetition code {$\Rthree$}.}
\label{fig.r3}
}%
%\end{figure}
% \noindent
%
Imagine that
% what might happen if
we transmit the source message
\[
\bs = \mbox{\tt 0 0 1 0 1 1 0}
\]
over a binary
symmetric channel with noise level $f=0.1$ using this repetition code.
We can describe the channel as `adding' a sparse noise vector $\bn$ to the
transmitted vector -- adding in modulo 2 arithmetic, \ie, the binary algebra in which
{\tt 1}+{\tt 1}={\tt 0}. A possible noise
vector $\bn$ and received vector $\br = \bt + \bn$
are shown in
\figref{fig.r3.transmission}.
\begin{figure}[htbp]
%
% here i should switch the \[ \] for a display that oes not introduce
% white space at the top (about 0.1in)
%
\figuremargin{%
\[
\begin{array}{rccccccc}
\bs & {\tt 0}&{\tt 0}&{\tt 1}&{\tt 0}&{\tt 1}&{\tt 1}&{\tt 0} \\
\bt & \obr{{\tt 0}}{{\tt 0}}{{\tt 0}}&\obr{{\tt 0}}{{\tt 0}}{{\tt 0}}&\obr{{\tt 1}}{{\tt 1}}{{\tt 1}}&\obr{{\tt 0}}{{\tt 0}}{{\tt 0}}&
\obr{{\tt 1}}{{\tt 1}}{{\tt 1}}&\obr{{\tt 1}}{{\tt 1}}{{\tt 1}}& \obr{{\tt 0}}{{\tt 0}}{{\tt 0}} \\
\bn & \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 1}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}}&
\nbr{{\tt 1}}{{\tt 0}}{{\tt 1}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}} \\ \cline{2-8}
\br & \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 1}}& \nbr{{\tt 1}}{{\tt 1}}{{\tt 1}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}}&
\nbr{{\tt 0}}{{\tt 1}}{{\tt 0}}& \nbr{{\tt 1}}{{\tt 1}}{{\tt 1}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}}
\end{array}
\]
}{%
\caption{An example transmission using $\mbox{R}_3$.}
\label{fig.r3.transmission}
}
\end{figure}
%\noindent
How should we decode this received vector?
%
% optimality not clear - should justify?
%
% Perhaps you can see that
The optimal algorithm looks at the received
bits three at a time and takes
a \ind{majority vote} (\algref{alg.r3}).
\begin{algorithm}[htbp]
\algorithmmargin{%
\begin{mycenter}
\begin{tabular}{ccc} % \toprule % \hline
Received sequence $\br$ &
Likelihood ratio $\frac{P(\br\,|\, s\eq {\tt 1})}{P(\br\,|\, s\eq {\tt 0})}$
&
Decoded sequence $\hat{\bs}$ \\ \midrule
\tt 000 & $\gamma^{-3}$ &\tt 0 \\
\tt 001 & $\gamma^{-1}$ &\tt 0 \\
\tt 010 & $\gamma^{-1}$ &\tt 0 \\
\tt 100 & $\gamma^{-1}$ &\tt 0 \\
\tt 101 & $\gamma^{1}$ &\tt 1 \\
\tt 110 & $\gamma^{1}$ &\tt 1 \\
\tt 011 & $\gamma^{1}$ &\tt 1 \\
\tt 111 & $\gamma^{3}$ &\tt 1 \\
% \bottomrule
\end{tabular}
\end{mycenter}
}{%
\caption[a]{Majority-vote decoding algorithm for {$\Rthree$}.
Also shown are the likelihood ratios (\ref{eq.likelihood.bsc}), assuming
% This is the optimal decoder if
the channel is a binary symmetric channel; $\gamma \equiv (1-f)/f$.}
%
\label{fig.r3d}
\label{alg.r3}
}%
\end{algorithm}
%
\begin{aside}
%
At the risk of explaining the obvious, let's prove this result.
The optimal decoding decision
(optimal in the sense
of having the smallest probability of being wrong)
is to find which value of $\bs$
is most probable, given $\br$.\index{maximum {\em a posteriori}}
% to make clear the assumptions.
Consider the decoding of a single bit $s$, which was encoded
as
% after encoding as
$\bt(s)$
and gave rise to three received bits $\br = r_1r_2r_3$.
By \ind{Bayes' theorem},\label{sec.bayes.used} the {\dem posterior
probability\/} of $s$ is
\beq
P(s \,|\, r_1r_2r_3 ) = \frac{ P( r_1r_2r_3 \,|\, s ) P( s ) }
{ P( r_1r_2r_3 ) } .
\label{eq.bayestheorem}
\eeq
We can spell out the posterior probability of the two alternatives thus:
\beq
P(s\eq {\tt 1} \,|\, r_1r_2r_3 ) = \frac{ P( r_1r_2r_3 \,|\, s\eq {\tt 1} )
P( s\eq {\tt 1} ) }
{ P( r_1r_2r_3 ) } ;
\label{eq.post1}
\eeq
\beq
P(s\eq {\tt 0} \,|\, r_1r_2r_3 ) = \frac{ P( r_1r_2r_3 \,|\, s\eq {\tt 0} )
P( s\eq {\tt 0} ) }
{ P( r_1r_2r_3 ) } .
\label{eq.post0}
\eeq
%
This \ind{posterior probability} is determined by two factors:
the
{\dem{\ind{prior} probability\/}} $P(s)$, and
the data-dependent term $P( r_1r_2r_3 \,|\, s )$, which is called
the {\dem{\ind{likelihood}\/}} of $s$.
The normalizing constant $P( r_1r_2r_3 )$
% is irrelevant to
needn't be computed when finding
the optimal decoding decision,
which is to guess $\hat{s}\eq {\tt 0}$
if $P(s\eq {\tt 0} \,|\, \br ) > P(s\eq {\tt 1} \,|\, \br )$,
and $\hat{s}\eq {\tt 1}$ otherwise.
To find
$P(s\eq {\tt 0} \,|\, \br )$ and $P(s\eq {\tt 1} \,|\, \br )$,
% the optimal decoding decision,
we must make an assumption about the prior probabilities of the
two hypotheses ${s}\eq {\tt 0}$ and ${s}\eq {\tt 1}$, and we
must make an assumption about the probability of $\br$ given
$s$.
% $\bt(s)$.
We assume that the prior probabilities are equal:
$P( {s}\eq {\tt 0}) = P( {s}\eq {\tt 1}) = 0.5$;
then maximizing the posterior probability $P(s\,|\,\br)$ is
equivalent to maximizing the likelihood $P(\br\,|\,s)$.\index{maximum likelihood}
And we assume that the
channel is a binary symmetric channel with noise level $f<0.5$, so that
the likelihood is
\beq
P( \br \,|\, s ) = P(\br \,|\, \bt(s) ) = \prod_{n=1}^N
P(r_n \,|\, t_n(s) ) ,
\eeq
where $N=3$ is the number of transmitted bits in the block
we are considering, and
\beq
P(r_n\,|\,t_n) = \left\{ \begin{array}{lll}
(1\!-\!f) & \mbox{if} & r_n=t_n \\
f & \mbox{if} & r_n \neq t_n. \end{array} \right.
\eeq
Thus the likelihood ratio for the
two hypotheses is
% if we define $
\beq
\frac{P(\br\,|\, s\eq {\tt 1})}{P(\br\,|\, s\eq {\tt 0})}
% = \left( \frac{ (1-f) }{f} \right)^{
= \prod_{n=1}^N
\frac{P(r_n \,|\, t_n({\tt 1}) )}{P(r_n \,|\, t_n({\tt 0}) )} ;
\label{eq.likelihood.bsc}
\eeq
each factor
% $P(r_n \,|\, t_n(s) )$
$\frac{P(r_n | t_n({\tt 1}) )}{P(r_n | t_n({\tt 0}) )}$
equals $\frac{ (1-f) }{f}$ if $r_n=1$ and $\frac{f}{ (1-f) }$ if
$r_n=0$.
The ratio $\gamma \equiv \frac{ (1-f) }{f}$ is greater than 1,
since $f<0.5$, so the winning hypothesis is the one with the most
`votes', each vote counting for a factor of $\gamma$ in the
% posterior probability.
likelihood ratio.
Thus the majority-vote decoder shown in \algref{fig.r3d}
is the optimal decoder if we assume that
the channel is a binary symmetric channel and that the
two possible source messages {\tt 0} and {\tt 1}
have equal prior probability.
\end{aside}
%\noindent
We now apply the majority vote decoder to the received vector of \figref{fig.r3.transmission}.
The first three received bits are all ${\tt 0}$, so
we decode this triplet
as a ${\tt 0}$.
In the second triplet of \figref{fig.r3.transmission},
there are two {\tt 0}s and one {\tt 1}, so we decode
this triplet as a ${\tt 0}$ -- which in this case corrects the error.
Not all errors are corrected, however. If we are unlucky and
two errors fall in a single block, as in the fifth triplet of
\figref{fig.r3.transmission},
then the decoding rule gets the wrong answer, as shown in
\figref{fig.decoding.R3}.
% \Figref{fig.decoding.R3}
% shows the result of decoding the received vector
% from \figref{fig.r3.transmission}.
\begin{figure}[htbp]
\figuremargin{%
\[
\begin{array}{rccccccc}
\bs & {\tt 0}&{\tt 0}&{\tt 1}&{\tt 0}&{\tt 1}&{\tt 1}&{\tt 0} \\
\bt & \obr{{\tt 0}}{{\tt 0}}{{\tt 0}}&\obr{{\tt 0}}{{\tt 0}}{{\tt 0}}&\obr{{\tt 1}}{{\tt 1}}{{\tt 1}}&\obr{{\tt 0}}{{\tt 0}}{{\tt 0}}&
\obr{{\tt 1}}{{\tt 1}}{{\tt 1}}&\obr{{\tt 1}}{{\tt 1}}{{\tt 1}}& \obr{{\tt 0}}{{\tt 0}}{{\tt 0}} \\
\bn & \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 1}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}}&
\nbr{{\tt 1}}{{\tt 0}}{{\tt 1}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}}& \nbr{{\tt 0}}{{\tt 0}}{{\tt 0}} \\ \cline{2-8}
\br & \ubr{{\tt 0}}{{\tt 0}}{{\tt 0}}& \ubr{{\tt 0}}{{\tt 0}}{{\tt 1}}& \ubr{{\tt 1}}{{\tt 1}}{{\tt 1}}& \ubr{{\tt 0}}{{\tt 0}}{{\tt 0}}&
\ubr{{\tt 0}}{{\tt 1}}{{\tt 0}}& \ubr{{\tt 1}}{{\tt 1}}{{\tt 1}}& \ubr{{\tt 0}}{{\tt 0}}{{\tt 0}} \\
\hat{\bs} & {\tt 0}&{\tt 0}&{\tt 1}&{\tt 0}&{\tt 0}&{\tt 1}&{\tt 0} \\
\mbox{corrected errors} &
&\star & & & & & \\
\mbox{undetected errors} &
& & & &\star & &
\end{array}
\]
}{%
\caption{Decoding
% Applying the maximum likelihood decoder for $\mbox{R}_3$ to
the received vector
from \protect\figref{fig.r3.transmission}.}
\label{fig.decoding.R3}
}%
\end{figure}
\noindent
% Thus the error probability is reduced by the use of this code.
% It is easy to compute the error probability.
% Exercise 1.1. Could this be made an Example, i.e. worked through in
% the text? -- for a beginner, there is a lot in it, and it seems to
% be important.
%
% see exercise.sty
\exercissx{2}{ex.R3ep}{%%%%%%%% keep this as A2, but cut it from the ITPRNN list
Show\marginpar{\small\raggedright The exercise's rating, \eg
% `{\em{A}}2'
`[{\em2\/}]',
indicates its difficulty:
`1' exercises are the easiest.
% An exercise rated {\em{A}}2 is important and should not prove too difficult.
Exercises that are accompanied by a marginal rat are especially recommended.
If a solution or partial solution is provided, the page is indicated after the difficulty rating;
for example, this exercise's solution is on page \pageref{ex.R3ep.sol}.
}
that the error probability is reduced by the use of {$\Rthree$}
by computing the error probability of
this code for a binary symmetric channel
with noise level $f$.
%Do so.
}
%
% This fig is 0.1 inch too wide, 9801
%
The error probability is dominated by the probability that two
bits in a block of three are flipped, which scales as $f^2$.
%
% JARGON??????
%
In the
case of the binary symmetric channel with $f=0.1$, the {$\Rthree$} code has a
probability of error, after decoding, of $\pb \simeq 0.03$ per bit.
\Figref{fig.r3.dilbert} shows the
result of transmitting a binary
image over a binary symmetric channel
using the repetition code.
\begin{figure}[hbtp]
%\fullwidthfigure{%
%\figuredangle{% this hung off the bottom of the page
\figuremarginb{% I think this may make a collision?
\begin{center}
\setlength{\unitlength}{0.8in}% was 0.75 98.12. changed to 0.8 99.01
\begin{picture}(7,4.3)(0,1.4)
\put(0,5){\makebox(0,0)[tl]{\psfig{figure=bitmaps/dilbert.ps,width=1in}}}
\put(0.625,5.4){\makebox(0,0){\Large$\bs$}}
\thicklines
\put(1.35,4.75){\vector(1,0){0.4}}
\put(1.55,5.4){\makebox(0,0){{\sc encoder}}}
\put(2,5){\makebox(0,0)[tl]{\psfig{figure=poster/10000.r3.ps,width=1in}}}
\put(2.625,5.4){\makebox(0,0){\Large$\bt$}}
\put(3.6,5.4){\makebox(0,0){{\sc channel}}}
\put(3.6,5.15){\makebox(0,0){$f={10\%}$}}
\put(3.4,4.75){\vector(1,0){0.4}}
\put(4,5){\makebox(0,0)[tl]{\psfig{figure=poster/10000.r3.0.10.ps,width=1in}}}
\put(4.625,5.4){\makebox(0,0){\Large$\br$}}
\put(5.6,5.4){\makebox(0,0){{\sc decoder}}}
%\put(5.6,3.4){\makebox(0,0)[tl]{\parbox[t]{1.75in}{{\em The decoder takes the majority vote of the three signals.}}}}
\put(5.4,4.75){\vector(1,0){0.4}}
\put(6,5){\makebox(0,0)[tl]{\psfig{figure=poster/10000.r3.0.10.d.ps,width=1in}}}
\put(6.625,5.4){\makebox(0,0){\Large$\hat{\bs}$}}
\end{picture}
\end{center}
}{%
\caption[a]{Transmitting $10\,000$ source bits over a binary symmetric channel
with $f=10\%$
% 0.1$
using a repetition code and the majority vote decoding
algorithm. The probability
of decoded bit error has fallen to about 3\%; the rate has fallen
to 1/3.}
% \dilbertcopy
\label{fig.r3.dilbert}
}%
\end{figure}
% Should `rate' be explicitly defined?
\newpage\indent
The repetition code $\Rthree$ has therefore reduced the probability of
error, as desired.
Yet we have lost something: our
{\em rate\/} of information transfer has fallen by a factor of
three. So if we use a repetition code to communicate data over a telephone
line, it will reduce the error frequency, but it will also reduce our
communication rate. We will have to pay three times as much for each
phone call.
% there will also be a delay
Similarly,
%As for our \disc{} drive,
we would need three of the original noisy gigabyte \disc{} drives
in order to create a one-gigabyte \disc{} drive with $\pb=0.03$.
Can we
% What happens as we try to
push the error probability lower, to the
values required for a
% quality
sellable \disc{} drive -- $10^{-15}$?
We could achieve lower error probabilities by using repetition
codes with more repetitions.
\exercissx{3}{ex.R60}{
\ben
\item
Show that the probability of error of $\RN$, the repetition
code with $N$ repetitions, is
\beq
p_{\rm b} = \sum_{n=(N+1)/2}^{N} {{N}\choose{n}} f^n (1-f)^{N-n} ,
\eeq
for odd $N$.
\item
Assuming $f = 0.1$, which of the terms in this sum is the biggest?
How much bigger is it than the second-biggest term?
\item
Use \ind{Stirling's approximation} (\pref{sec.stirling}) to approximate
% get rid of
the ${{N}\choose{n}}$
in the largest term, and find,
approximately, the probability of error of the repetition
code with $N$ repetitions.
\item
Assuming $f = 0.1$, find how many repetitions
are required
% show that it takes a repetition
% code with rate about $1/60$
to get the probability of error
down to $10^{-15}$. [Answer: about 60.]
\een
}
So to build a {\em single\/}
gigabyte \disc{} drive
with the required reliability from noisy gigabyte drives with $f=0.1$,
we would need {\em sixty\/} of the noisy \disc{} drives.
The tradeoff between error probability and rate for repetition
codes is shown in \figref{fig.pbR.R}.
%
% see end of l1.tex for method, also see poster1.gnu
%
\newcommand{\pbobject}{\hspace{-0.15in}\raisebox{1.62in}{$\pb$}%
\hspace{-0.05in}}
\begin{figure}
\figuremargin{%
\begin{center}
\begin{tabular}{cc}
\hspace{-0.2in}\psfig{figure=\codefigs/rep.1.ps,angle=-90,width=2.6in} &
\pbobject\psfig{figure=\codefigs/rep.1.l.ps,angle=-90,width=2.6in} \\
\end{tabular}
\end{center}
}{%
\caption[a]{Error probability $\pb$ versus rate for repetition codes
over a binary symmetric channel with $f=0.1$.
The right-hand figure shows $\pb$ on a logarithmic scale. We would like
the rate to be large and $\pb$ to be small.
}
\label{fig.pbR.R}
}%
\end{figure}
% see end of this file for method
\subsection{Block codes -- the $(7,4)$ Hamming code}
\label{sec.ham74}
We would like to communicate with\index{Hamming code}
tiny probability of error {\em and\/} at a substantial rate.
Can we improve on repetition codes? What if we add redundancy to
{\dem blocks\/} of data instead of
% redundantly
encoding one bit at a time?
% You may already have heard of the idea of `parity check bits'.
We now
study a simple {\dem{block code}}.
A {\dem \ind{block code}\/} is a rule\index{error-correcting code!block code}
for converting a sequence of source
bits $\bs$, of length $K$, say, into a transmitted sequence $\bt$ of length
$N$ bits. To add redundancy, we make $N$
greater than $K$. In a {\dem linear\/} block code,
the extra $N-K$ bits are linear functions of the
original $K$ bits; these extra bits are called {\dem\ind{parity-check bits}}.
An example of a \ind{linear block code} is the \mbox{\dem$(7,4)$
\ind{Hamming code}}, which transmits $N=7$ bits for every $K=4$ source
bits.
% \index{error-correcting code!linear}
\begin{figure}[htbp]
\figuremargin{\small%
\begin{center}
\begin{tabular}{cc}
(a)\psfig{figure=hamming/encode.eps,angle=-90,width=1.3in} &
(b)\psfig{figure=hamming/correct.eps,angle=-90,width=1.3in} \\
\end{tabular}
\end{center}
}{
\caption[a]{Pictorial representation of encoding for the $(7,4)$ Hamming
code.
% a and b are not explained in the caption. Does this matter?
%
% The parity check bits $t_5,t_6,t_7$ are set so that the parity within
%% each circle is even.
}
\label{fig.74h.pictorial}
\label{fig.hamming.pictorial}
}
\end{figure}
%
The encoding operation for the code is shown pictorially
in \figref{fig.74h.pictorial}.
%
% \subsubsection{Encoding}
We arrange the seven transmitted bits in three intersecting circles.
% as shown in \figref{fig.hamming.encode}.
The first four
transmitted bits,
$t_1 t_2 t_3 t_4$, are set equal to the four source bits,
$s_1 s_2 s_3 s_4$.
The parity-check bits\index{parity-check bits}
$t_5 t_6 t_7$ are set so that the {\dem\ind{parity}\/}
within each circle is even:
the first parity-check bit is the parity of the first three source bits
(that is, it is
%zero
{\tt 0} if the sum of those bits is even, and
% one
{\tt 1} if the sum is odd);
the second is the parity of the last three; and the third parity bit
is the parity of source bits one, three and four.
As an example, \figref{fig.74h.pictorial}b shows the transmitted
codeword for the case $\bs = {\tt 1000}$.
% idea for rewriting this: go straight to pictorial story, leave out the
% matrix description for another time.
%
%
%\noindent
%
Table \ref{tab.74h} shows the codewords generated
by each of the $2^4=$ sixteen settings of the four source bits.
% Notice that the first four transmitted bits are
% identical to the four source bits, and the remaining three bits
% are parity bits:
% The special property of these codewords is that
These codewords
have the special property that
any pair
differ from each other in at least three bits.
\begin{table}[htbp]
\figuremargin{%
\begin{center}
\mbox{\small
\begin{tabular}{cc} \toprule
% Source sequence
$\bs$ &
% Transmitted sequence
$\bt$ \\ \midrule
\tt 0000 &\tt 0000000 \\
\tt 0001 &\tt 0001011 \\
\tt 0010 &\tt 0010111 \\
\tt 0011 &\tt 0011100 \\ \bottomrule
\end{tabular} \hspace{0.02in}
\begin{tabular}{cc} \toprule
$\bs$ & $\bt$ \\ \midrule
\tt 0100 &\tt 0100110 \\
\tt 0101 &\tt 0101101 \\
\tt 0110 &\tt 0110001 \\
\tt 0111 &\tt 0111010 \\ \bottomrule
\end{tabular} \hspace{0.02in}
\begin{tabular}{cc} \toprule
$\bs$ & $\bt$ \\ \midrule
\tt 1000 &\tt 1000101 \\
\tt 1001 &\tt 1001110 \\
\tt 1010 &\tt 1010010 \\
\tt 1011 &\tt 1011001 \\ \bottomrule
\end{tabular} \hspace{0.02in}
\begin{tabular}{cc} \toprule
$\bs$ & $\bt$ \\ \midrule
\tt 1100 &\tt 1100011 \\
\tt 1101 &\tt 1101000 \\
\tt 1110 &\tt 1110100 \\
\tt 1111 &\tt 1111111 \\ \bottomrule
\end{tabular}
}%%%%%%%%% end of row of four tables
\end{center}
}{%
\caption[a]{The sixteen codewords
$\{ \bt \}$ of the $(7,4)$ Hamming code. Any pair of
codewords
% have the % beautiful % elegant property that they
differ from each other in at least three bits.}
%\label{fig.hamming.encode}
\label{tab.74h}
\label{tab.h74}
\label{fig.h74}
\label{fig.74h}
}
\end{table}
%
\begin{aside}
Because the Hamming code is a {linear\/} code, it can\indexs{error-correcting code!linear}
be written compactly in terms of matrices as follows.\index{linear block code}
% It is a
% {\em linear\/} code; that is, t
The transmitted codeword $\bt$ is
% can be
obtained
from the source sequence $\bs$ by a linear operation,
\beq
\bt = \bG^{\T} \bs,
\label{eq.encode}
\eeq
where $\bG$ is the {\dem\ind{generator matrix}} of the code,
\beq
\bG^{\T} = {\left[ \begin{array}{cccc}
\tt 1 &\tt 0 &\tt 0 &\tt 0 \\
\tt 0 &\tt 1 &\tt 0 &\tt 0 \\
\tt 0 &\tt 0 &\tt 1 &\tt 0 \\
\tt 0 &\tt 0 &\tt 0 &\tt 1 \\
\tt 1 &\tt 1 &\tt 1 &\tt 0 \\
\tt 0 &\tt 1 &\tt 1 &\tt 1 \\
\tt 1 &\tt 0 &\tt 1 &\tt 1 \end{array} \right] } ,
\label{eq.h74.gen}
\eeq
and the encoding operation (\ref{eq.encode}) uses
modulo-2 arithmetic (${\tt 1}+{\tt 1}={\tt{0}}$, ${\tt 0}+{\tt 1}={\tt 1}$, etc.).
%\footnote{My notational
% convention is that all vectors -- $\bs$, $\bt$, etc.\ --
% are column vectors, except that in the figures where many
% vectors are listed, they are displayed as row vectors. The
% generator matrix $\bG$ is written ..... as to retain
% consistency with established notation in coding texts.}
% \begin{aside}
In the encoding operation
(\ref{eq.encode}) I have assumed that $\bs$ and $\bt$ are
column vectors. If instead they are row vectors, then this equation
is replaced by
\beq
\bt = \bs \bG,
\label{eq.encodeT}
\eeq
where
\beq
\bG = \left[ \begin{array}{ccccccc}
\tt 1& \tt 0& \tt 0& \tt 0& \tt 1& \tt 0& \tt 1 \\
\tt 0& \tt 1& \tt 0& \tt 0& \tt 1& \tt 1& \tt 0 \\
\tt 0& \tt 0& \tt 1& \tt 0& \tt 1& \tt 1& \tt 1 \\
\tt 0& \tt 0& \tt 0& \tt 1& \tt 0& \tt 1& \tt 1 \\
\end{array} \right] .
\label{eq.Generator}
\eeq
% f you are like me, you may
I find it easier to relate to
the right-multiplication (\ref{eq.encode})
% hyphenation specified in itprnnchapter.tex did not work so I do it manually
than the left-multiplica-{\breakhere}tion (\ref{eq.encodeT}).
% -- I like my matrices to act to the right.
Many coding theory texts use the left-multiplying conventions
(\ref{eq.encodeT}--\ref{eq.Generator}), however.
The rows of the generator matrix (\ref{eq.Generator}) can be
viewed as defining four basis vectors lying in a seven-dimensional
binary space. The sixteen codewords are obtained by making all
possible linear combinations
% binary sums
of these vectors.
\end{aside}
%
% should I add a cast of characters here?
% s,t,r,s^
\subsubsection{Decoding the $(7,4)$ Hamming code}
When we invent a more complex encoder $\bs \rightarrow \bt$,
the task of decoding the
received vector $\br$ becomes less straightforward. Remember that
{\em any\/} of the bits may have been flipped, including the parity bits.
% We can't assume that the three extra parity bits
%(The reader who
% is eager to see the denouement of the plot may skip ahead to section
% \ref{sec.code.perf}.)
% General defn of optimal decoder
If we assume that the channel is a binary symmetric channel and that
all source vectors are equiprobable,
% {\em a priori},
then the
optimal decoder
% is one that
identifies the source vector $\bs$ whose
encoding $\bt(\bs)$ differs from the received vector $\br$ in the
fewest bits. [{Refer to the likelihood function
% equation
% {eq.bayestheorem}--\ref{eq.likelihood.bsc}}
\bref{eq.likelihood.bsc}} to see why this is so.]
We could solve the decoding problem by measuring how far $\br$
is from each of the
sixteen codewords in \tabref{tab.74h}, then picking the closest.
Is there a more efficient way of finding the most probable source vector?
\subsubsection{Syndrome decoding for the Hamming code}
\label{sec.syndromedecoding}
For the $(7,4)$ Hamming code there is a pictorial solution to the
% syndrome
decoding problem, based on the encoding picture,
\figref{fig.74h.pictorial}.
%
% \subsubsection{Decoding}
%
% sanjoy says this is CONFUSING - tried to improve it Sat 22/12/01
% also romke did not like it
As a first example, let's assume the transmission was
$\bt = {\tt 1000101}$ and the noise flips the second bit,
so the received vector is
$\br = {\tt 1000101}\oplus{\tt{0100000}} = {\tt{1100101}}$.
% \ie, $\bn=({\tt 0},{\tt 1},{\tt 0},{\tt 0},{\tt 0}, {\tt 0},{\tt 0})$,
% and the received vector
We write the received vector into the three circles
as shown in \figref{fig.hamming.decode}a, and
look at each of the three circles to see whether its parity is even.
The circles whose parity is {\em{not}\/} even are shown by
dashed lines in \figref{fig.hamming.decode}b.
% The fact that all codewords differ from each other in at least
% three bits means that if the noise has flipped any one or two bits,
% the received vector will no longer be a valid codeword, and some of
% the parity checks will be broken.
%
The decoding task is
%We want
to find the smallest
set of flipped bits that can account for these violations
of the parity rules.
% violated.
[The pattern of violations of the parity checks is called the {\dem\ind{syndrome}}, and can be written as a binary vector -- for example,
in \figref{fig.hamming.decode}b, the syndrome is $\bz = ({\tt1},{\tt1},{\tt0})$,
because the first two circles are `unhappy' (parity {\tt1}) and the
third circle is `happy' (parity {\tt0}).]
% RESTORE ME:
%, and the task of syndrome decoding
% syndrome (just as a
% \ind{doctor} might seek the most probable underlying \ind{disease} to account for
% the symptoms shown by a \ind{patient}).
\begin{figure}% [htbp]
\figuremargin{\small%
\begin{center}
\begin{tabular}{ccc}
(a)\psfig{figure=hamming/decode.eps,angle=0,width=1.3in} \\
(b)\psfig{figure=hamming/s2.eps,angle=-90,width=1.3in} &
(c)\psfig{figure=hamming/t5.eps,angle=-90,width=1.3in} &
(d)\psfig{figure=hamming/s3.eps,angle=-90,width=1.3in} \\[0.3in]
\multicolumn{3}{c}{%
(e)\psfig{figure=hamming/s3.t7.eps,angle=0,width=1.3in}
\setlength{\unitlength}{1in}
\begin{picture}(0.4,0.6)(0,0)
\put(0,0.6){\vector(1,0){0.6}}
\end{picture}
% \raisebox{0.6in}{$\rightarrow$}
(${\rm e}'$)\psfig{figure=hamming/s3.t7.d.eps,angle=0,width=1.3in}
}\\
\end{tabular}
\end{center}
}{%
\caption[a]{Pictorial representation of decoding of the Hamming $(7,4)$
code. The received vector is written into the diagram
as shown in (a).
In (b,c,d,e), the received vector is
shown, assuming that the transmitted vector was
as in
% The bits that are flipped relative to
\protect
\figref{fig.hamming.pictorial}b and the bits labelled by $\star$
were flipped. The violated
parity checks are highlighted by dashed circles. One of the seven bits
is the most probable suspect to account for each `\ind{syndrome}', \ie, each
pattern of violated and satisfied parity checks.
In examples (b), (c), and (d), the most probable suspect is
the one bit that was flipped.
In example (e), two bits have been flipped, $s_3$ and $t_7$.
The most probable suspect is $r_2$, marked by a circle in (${\rm e}'$),
which shows the output of the decoding algorithm.
% each circle is even.
}\label{fig.hamming.decode}
\label{fig.hamming.s2}% these labels were in the wrong place feb 2000
\label{fig.hamming.s3}
\label{fig.hamming.correct}
}
\end{figure}
%
% ACTION: sanjoy still thinks this part is hard to follow - fixed Sat 22/12/01?
To solve the decoding task,
% problem,
we ask the question:
can we find a unique bit that lies {\em inside\/}
all the `unhappy' circles and {\em outside\/} all the
`happy' circles? If so, the flipping of that bit
would account for the observed
syndrome.
In the case shown in \figref{fig.hamming.s2}b,
the bit $r_2$
% that was flipped
lies inside the two unhappy circles and outside the happy
circle;
no other single bit has this property, so
$r_2$ is the only single bit capable of explaining the syndrome.
Let's work through a couple more examples.
\Figref{fig.hamming.s2}c shows what happens if one of the
parity bits, $t_5$, is flipped by the noise. Just one of the checks
is violated. Only $r_5$ lies inside this unhappy circle and outside
the other two happy circles,
so $r_5$ is identified as the only single bit
capable of explaining the syndrome.
If the central bit $r_3$ is received flipped,
\figref{fig.hamming.s3}d shows that all three checks are violated;
only $r_3$ lies inside all three circles, so $r_3$ is
identified as the suspect bit.
If you try flipping any one of the seven bits, you'll find
that a different syndrome is obtained in each case -- seven non-zero syndromes,
one for each bit. There is only
one other syndrome, the all-zero syndrome. So if
the channel is a binary symmetric channel with a
small noise level $f$, the optimal
decoder unflips at most one bit, depending on the
syndrome, as shown in \algref{tab.hamming.decode}.
Each syndrome could have been caused by other noise patterns
too, but any other noise pattern that has the same syndrome
must be less probable because it involves a larger number of
noise events.
%\begin{figure}
%\figuremargin{%
\begin{algorithm}
\algorithmmargin{%
\begin{center}
\begin{tabular}{c*{8}{c}}
% Fri 4/1/02 removed toprule and bottomrule because algorithm has its own frame
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \toprule
Syndrome $\bz$ & {\tt 000} & {\tt 001} & {\tt 010} & {\tt 011} & {\tt 100} & {\tt 101} & {\tt 110} & {\tt 111} \\ \midrule
Unflip this bit & {\small{\em none}} & $r_7$ & $r_6$ & $r_4$ & $r_5$ & $r_1$ & $r_2$ & $r_3$ \\
% \bottomrule
% Unflip this bit & {\small{\em none}} & 7 & 6 & 4 & 5 & 1 & 2 & 3 \\
% \bottomrule
% this is appropriate only if z =z3,z2,z1:
% Unflip this bit & {\small{\em none}} & 5 & 6 & 2 & 7 & 1 & 4 & 3 \\ \hline
\end{tabular}
\end{center}
%\begin{center}
%\begin{tabular}{cc} \hline
%Syndrome $\bz$ & % 3 2 1 !!!!!!!!!!!!!!!!!!!
%Flip this bit \\ \hline
% 000 &{\small{\em none}} \\
% 001 &5\\
% 010 &6\\
% 011 &2\\
% 100 &7\\
% 101 &1\\
% 110 &4\\
% 111 &3 \\ \hline
%\end{tabular}
%\end{center}
}{%
\caption[a]{Actions taken by the optimal decoder for the $(7,4)$ Hamming
code, assuming a binary symmetric channel with small noise level $f$.
The syndrome vector $\bz$ lists whether each parity check is
violated ({\tt 1}) or satisfied ({\tt 0}),
going through the checks in the order
of the bits $r_5$, $r_6$,
and $r_7$. }
\label{tab.hamming.decode}
}%
\end{algorithm}
What happens if the noise actually flips more than one bit?
\Figref{fig.hamming.s3}e shows the situation when two bits,
$r_3$ and $r_7$, are received flipped. The syndrome, {\tt 110},
makes us suspect the single bit $r_2$; so our optimal decoding algorithm
flips this bit, giving a decoded pattern with three errors
as shown in \figref{fig.hamming.s3}${\rm e}'$.
If we use the optimal decoding algorithm,
any two-bit error pattern will lead to a decoded seven-bit vector
that contains three errors.
\subsection{General view of decoding for linear codes: syndrome decoding}
\label{sec.syndromedecoding2}
\begin{aside}
% {\em (Does some of this stuff belong earlier in the pictorial area?)}
We can also describe the decoding problem
for a linear code in terms of matrices.\index{syndrome decoding}\index{linear block code}
% In the case of a linear code and a symmetric channel,
% the decoding task can be re-expressed as {\bf syndrome decoding}.
% Let's assume that the noise level $f$ is less than $1/2$.
The first four received bits, $r_1r_2r_3r_4$, purport to be
the four source bits; and the received bits $r_5r_6r_7$ purport
to be the parities of the source bits, as defined by the generator
matrix $\bG$. We evaluate the three parity-check bits for the
received bits, $r_1 r_2r_3 r_4$, and see whether
they match the three received
bits, $r_5r_6r_7$. The differences (modulo 2) between
these two triplets are called the {\dbf\ind{syndrome}}
of the received vector.
If the syndrome is zero -- if all three parity checks are happy
% agree with the corresponding received bits
-- then the received vector is a codeword,
and the most probable decoding is given by reading out its first four
bits. If the syndrome is non-zero, then
% we are certain that
the noise
sequence for this block was non-zero, and the syndrome is our
pointer to the most probable error pattern.
The computation of the syndrome vector is a
linear operation. If we define the $3 \times 4$ matrix $\bP$
such that the matrix of
equation (\ref{eq.h74.gen})
is
\beq
\bG^{\T} = \left[ \begin{array}{c}{\bI_4}\\
\bP\end{array} \right],
\eeq
where $\bI_4$ is the $4\times 4$ identity matrix, then
the syndrome vector is $\bz = \bH \br$, where the {\dbf\ind{parity-check matrix}}
$\bH$ is given by $\bH = \left[ \begin{array}{cc} -\bP & \bI_3 \end{array}
\right]$; in modulo 2 arithmetic, $-1 \equiv 1$, so
\beq
\bH = \left[ \begin{array}{cc} \bP & \bI_3 \end{array}
\right] = \left[
\begin{array}{ccccccc}
\tt 1&\tt 1&\tt 1&\tt 0&\tt 1&\tt 0&\tt 0 \\
\tt 0&\tt 1&\tt 1&\tt 1&\tt 0&\tt 1&\tt 0 \\
\tt 1&\tt 0&\tt 1&\tt 1&\tt 0&\tt 0&\tt 1
\end{array} \right] .
\label{eq.pcmatrix}
\eeq
All the codewords $\bt = \bG^{\T} \bs$ of the code satisfy
\beq
\bH \bt = \left[ {\tt \begin{array}{c} \tt0\\ \tt0\\ \tt0 \end{array} } \right] .
% (0,0,0) .
\eeq
\exercisaxB{1}{ex.GHis0}{
Prove that this is so by evaluating the $3\times4$ matrix $\bH \bG^{\T}$.
}
Since the received vector $\br$ is given by $\br = \bG^{\T}\bs + \bn$,
% and $\bH \bG^{\T}$=0,
the syndrome-decoding problem is to find the
most probable noise vector $\bn$ satisfying
the equation
\beq
\bH \bn = \bz .
\eeq
A decoding algorithm that solves this problem is called
a {\dem maximum-likelihood decoder}. We will discuss
decoding problems like this in later chapters.
%\footnote{Somewhere in this book
% I need to spell out \Bayes\ theorem for decoding. Here would be
% a good spot; but on the other hand, people can understand decoding
% intuitively, they don't need Bayes theorem and they might find it
% a hindrance if they were not only being hit by
% Shannon's theorem but also by likelihoods and priors.}
%
% ACTION NEEDED ????????????????????????????????????????
%
\end{aside}
\begin{figure}
%\fullwidthfigure{%
\figuredanglenudge{%
\begin{center}
\setlength{\unitlength}{0.8in}% was 1in, with figures 1.25 wide % then was 0.8 with 1in
\begin{picture}(7,2.7)(0,2.8)
\put(0,5){\makebox(0,0)[tl]{\psfig{figure=bitmaps/dilbert.ps,width=1in}}}
\put(0.625,5.4){\makebox(0,0){\Large$\bs$}}
\thicklines
\put(1.35,4.75){\vector(1,0){0.4}}
\put(1.55,5.4){\makebox(0,0){{\sc encoder}}}
\put(2,5){\makebox(0,0)[tl]{\psfig{figure=poster/10000.h74.ps,width=1in}}}
\put(1.982,3.75){\makebox(0,0)[tr]{{parity bits} $\left.\rule[-0.342in]{0pt}{0.342in} \right\{$}}
\put(2.625,5.4){\makebox(0,0){\Large$\bt$}}
\put(3.6,5.4){\makebox(0,0){{\sc channel}}}
\put(3.6,5.15){\makebox(0,0){$f={10\%}$}}
\put(3.4,4.75){\vector(1,0){0.4}}
\put(4,5){\makebox(0,0)[tl]{\psfig{figure=poster/10000.h74.0.10.ps,width=1in}}}
\put(4.625,5.4){\makebox(0,0){\Large$\br$}}
\put(5.6,5.4){\makebox(0,0){{\sc decoder}}}
%\put(5.6,3.5){\makebox(0,0)[tl]{\parbox[t]{1.75in}{{\em The decoder picks the $\hat{\bs}$ with maximum likelihood.}}}}
\put(5.4,4.75){\vector(1,0){0.4}}
\put(6,5){\makebox(0,0)[tl]{\psfig{figure=poster/10000.h74.0.10.d.ps,width=1in}}}
\put(6.625,5.4){\makebox(0,0){\Large$\hat{\bs}$}}
\end{picture}
\end{center}
}{%
\caption[a]{Transmitting $10\,000$ source bits over a binary symmetric channel
with $f=10\%$
%0.1$
using a $(7,4)$ Hamming code. The probability
of decoded bit error is about 7\%.}
% \dilbertcopy}
\label{fig.h74.dilbert}
}{0.7in}% third argument is the upward nudge of the caption
\end{figure}
\subsection{Summary of the $(7,4)$ Hamming code's properties}
Every possible received vector of length 7 bits is either a codeword,
or it's one flip away from a codeword.\index{Hamming code}
Since there are three parity constraints, each of which might
or might not be violated, there are
$2\times 2\times 2= 8$
% eight
distinct syndromes. They can be divided
into seven non-zero syndromes -- one
for each of the one-bit error patterns --
and the all-zero syndrome, corresponding to the zero-noise case.
The optimal decoder takes no action if the syndrome is zero,
otherwise it uses this mapping of non-zero syndromes onto one-bit error
patterns to unflip the suspect bit.
There is a {\dbf decoding error} if the four decoded bits $\hat{s}_1,
\hat{s}_2, \hat{s}_3, \hat{s}_4$ do not all match the source bits ${s}_1,
{s}_2, {s}_3, {s}_4$. The {\dbf probability of block error} $\pB$ is
the probability that one or more of the decoded bits in one block fail to
match the corresponding source bits,
\beq
\pB = P( \hat{\bs} \neq \bs ) .
\eeq
The {\dbf probability of bit error} $\pb$ is
the average probability
% per decoded bit
that a decoded bit fails to
match the corresponding source bit,
\beq
\pb = \frac{1}{K} \sum_{k=1}^K P( \hat{s}_k \neq s_k ) .
\eeq
In the case of the Hamming code,
a decoding error will occur whenever the noise has flipped more than
one bit in a block of seven.
% Any noise pattern that flips more than one bit will give rise to one of
% these syndromes, and our decoder will make an erroneous decision.
%
The probability of block error is thus the probability that two or more
bits are flipped in a block. This probability scales as $O(f^2)$, as did the
probability of error for the repetition code
$\Rthree$. But notice that the Hamming code
communicates at a greater rate, $R=4/7$.
\Figref{fig.h74.dilbert} shows a binary image transmitted over
a binary symmetric channel using the $(7,4)$ Hamming code.
About 7\% of the decoded bits are in error. Notice that
the errors are correlated:
% with each other:
often two or three successive
decoded bits are flipped.
\exercisaxA{1}{ex.Hdecode}{
This exercise and the next three refer to the
$(7,4)$ \ind{Hamming code}. Decode the received strings:
\ben
\item $\br = {\tt 1101011}$ % 10
\item $\br = {\tt 0110110}$ % 4
\item $\br = {\tt 0100111}$ % 4
\item $\br = {\tt 1111111}$. % 15
\een
}
\exercissxA{2}{ex.H74p}{
\ben \item
Calculate the probability of block error $p_{\rm B}$ of the $(7,4)$ Hamming
code
as a function of the noise level $f$ and show that to leading order
% \footnote{Do I need to explain what this means? Or use a different
% terminology? Maybe only physicists are familiar?}
%
% ACTION!!!
%
it goes as $21 f^2$.
\item
% }
% \exercis{}{
\difficulty{3}
% $^{B3}$
Show that to leading order the probability of
bit error $\pb$ goes as $9 f^2$.
\een}
\exercissxA{2}{ex.H74zero}{
% Hamming $(7,4)$ code.
Find some noise vectors that give the all-zero syndrome (that is,
noise vectors that leave all the parity checks unviolated).
How many such noise vectors are there?
}
% they are the codewords.
\exercisaxB{2}{ex.H74detail}{
% Hamming $(7,4)$ code.
I asserted above that a block decoding error will result
whenever two or more bits are flipped in a single block.
Show that this is indeed so. [In principle, there might be
error patterns that, after decoding, led only to the corruption
of the parity bits, with no source bits incorrectly
decoded.]
}
\subsection{Summary of codes' performances}
\label{sec.code.perf}
Figure \ref{fig.pbR.RH} shows the performance of \ind{repetition code}s and
the \ind{Hamming code}. It also shows the performance of a family of linear
block codes that are generalizations of Hamming codes, called \ind{BCH codes}.
% Reed-Muller codes, and
% see end of this file for method
%
\begin{figure}[htbp]
\figuremargin{%
\begin{center}
\begin{tabular}{cc}
\hspace{-0.2in}\psfig{figure=\codefigs/rephambch.1.ps,angle=-90,width=2.6in} &
\pbobject\psfig{figure=\codefigs/rephambch.1.l.ps,angle=-90,width=2.6in} \\
\end{tabular}
\end{center}
}{%
\caption[a]{Error probability $\pb$ versus rate $R$ for repetition codes,
the $(7,4)$ Hamming code and BCH codes with blocklengths up to 1023
over a binary symmetric channel with $f=0.1$.
The righthand figure shows $\pb$ on a logarithmic scale.}
\label{fig.pbR.RH}
}
\end{figure}
%
%\noindent
% use this noindent if the ``h'' (here) works, otherwise new para.
This figure shows that we can, using linear block codes, achieve better
performance than repetition codes; but the asymptotic situation still
looks grim.
\exercissxA{4}{ex.makecode}{
% invent your own code
Design an error-correcting code and a decoding algorithm for it,
estimate its probability of error,
and add it to figure \ref{fig.pbR.RH}.
[Don't worry if you find it difficult to make a code better than the
Hamming code, or if you find it difficult to find a good
decoder for your code; that's the point of this exercise.]
}
\exercissxA{3}{ex.makecode2error}{
A $(7,4)$ Hamming code
can correct any {\em one\/} error;
might there be a
% (10,4)
$(14,8)$ code
that can correct any two errors?
% What about a (9,4) code?
{\sf Optional extra:} Does the answer to this question
depend on whether the code is linear or nonlinear?
}
\exercissxA{4}{ex.makecode2}{
Design an error-correcting code, other than
a repetition code, that can
correct any {\em two\/} errors in a block of size $N$.
}
\section{What performance can the best codes achieve?}
There seems to be a trade-off between the decoded bit-error
probability $\pb$ (which we would like to reduce) and the rate $R$ (which
we would like to keep large). How can this trade-off be
characterized?
% Can we do better than repetition codes?
What points in
the $(R,\pb)$ plane are achievable? This question was addressed by
Claude Shannon\index{Shannon, Claude} in his pioneering paper of 1948, in which he both created the
field of information theory and solved most of its fundamental
problems.
% in the same paper.
At that time there was a widespread belief that the
boundary between achievable and nonachievable points in the
$(R,\pb)$ plane was a curve passing through the origin $(R,\pb) = (0,0)$;
if this were so, then, in order to achieve a vanishingly small
error probability $\pb$, one would have to reduce the rate
correspondingly close to zero.
% (figure ref here).
% This would seem a reasonable guess,
% in accordance with the general rule that the better something works
% the more you have to pay for it.
%
% ACTION: sanjoy doesn't like This
%
`No pain, no gain.'
However, Shannon proved the remarkable result that\wow\
% , for any given channel,
the boundary
between achievable and nonachievable points meets the $R$
axis at a {\em non-zero\/} value $R=C$, as shown in \figref{fig.pbR.RHS}.
\begin{figure}[htbp]
\figuremargin{%
\begin{center}
\begin{tabular}{cc}
\hspace{-0.2in}\psfig{figure=\codefigs/repshan.1.ps,angle=-90,width=2.6in} &
\pbobject\psfig{figure=\codefigs/repshan.1.l.ps,angle=-90,width=2.6in} \\
\end{tabular}
\end{center}
}{%
\caption[a]{Shannon's noisy-channel coding theorem.\indexs{noisy-channel coding theorem}\index{Shannon, Claude}
The solid curve shows the Shannon limit
on achievable values of $(R,\pb)$ for
the binary symmetric channel with $f=0.1$.
Rates up to $R=C$ are achievable with arbitrarily small $\pb$.
The points show the performance of some textbook codes,
as in \protect\figref{fig.pbR.RH}.
%\indent MANUAL INDENT
\hspace{1.5em}The equation defining the Shannon limit (the solid curve) is
%\[
$R = \linefrac{C}{(1-H_2(\pb))},$
%\]
where $C$ and $H_2$ are defined in \protect \eqref{eq.capacity}.
}
\label{fig.pbR.RHS}
}
\end{figure}
% see end of this file for method
%
For any channel, there exist codes that make it possible to
communicate with {\em arbitrarily small\/} probability of
error $\pb$ at non-zero rates. The first half of this book ({\partnoun}s I--III) will be
devoted to understanding this remarkable result, which is called
the {\dbf{noisy-channel coding theorem}}.
\subsection{Example: $f=0.1$}% A few details}
The maximum rate at which communication is possible with arbitrarily
small $\pb$ is called the {\dbf\ind{capacity}} of the channel.\index{channel!capacity}
The formula for the capacity of a binary
symmetric channel with noise level $f$ is\index{binary entropy function}
\beq
C(f) = 1 - H_2(f) = 1 - \left[ f \log_2
\frac{1}{f} + (1-f) \log_2 \frac{1}{1-f} \right] ;
\label{eq.capacity}
\eeq
the channel we were discussing earlier with noise level $f=0.1$
has capacity $C \simeq 0.53$. Let us consider what this means in terms
of noisy \disc{} drives. The \ind{repetition code} $\Rthree$ could communicate over this
channel with $\pb=0.03$ at a rate $R = 1/3$. Thus we know how
to build a single gigabyte \disc{} drive with $\pb = 0.03$
from three noisy gigabyte \disc{} drives. We also know how to make
a single gigabyte \disc{} drive
with $\pb \simeq 10^{-15}$ from sixty
noisy one-gigabyte drives \exercisebref{ex.R60}.
And now Shannon\index{Shannon, Claude}
passes by, notices us
\ind{juggling}
% tinkering
with \disc{} drives and codes and says:
\begin{quotation}
\noindent
`What performance are you trying to achieve?
$10^{-15}$? You don't need {\em sixty\/} \disc{} drives --
you can get that performance with just
{\em two\/} \disc{} drives (since 1/2 is less than $0.53$).
% (The capacity is 0.53, so the number of \disc{} drives needed at
% capacity is 1/0.53.)
% `
And if you want $\pb = 10^{-18}$
% , or $10^{-21}$,
or $10^{-24}$ or anything,
you can get there with two \disc{} drives too!'
\end{quotation}
%\begin{aside}
[Strictly, the above statements might not be quite right, since,
as we shall see, Shannon
proved his
noisy-channel coding theorem
%proves the achievability of ever smaller
% error probabilities at a given rate $Ra$)
is defined to be $\int_{a}^{b} \! \d v \: P(v)$. $P(v)\d v$ is dimensionless.
The density $P(v)$ is a dimensional
quantity, having dimensions inverse to the dimensions of $v$ -- in contrast
to discrete probabilities, which are dimensionless. Don't be surprised
to see probability densities greater than 1. This is normal, and nothing
is wrong, as long as $\int_{a}^{b} \! \d v \: P(v) < 1$ for any interval $(a,b)$.
Conditional and joint probability densities
are defined in just the same way as conditional and joint probabilities.
% , which is why I choose not to use different notation for them.
\end{aside}
% More equations here.
%
% bring from chapter 4?
%
% at present ch 4 refers to this page as the first occurrence of
% Laplace's rule.
%
% Sort out this mess:::::::::::::::
% p30 Ex 2.8 : There claims to be a solution to this on p121 but this is
%actually a solution to Ex 6.2
%Generally would be helpful if notation in Chapters 2 and 6 was the same
%
% !!!!!!!!!!!!!!!!!!!! Idea: move this exe to the end of this subsection?
% THIS EX seems to have no solution
\exercisaxB{2}{ex.postpa}{% solution added Mon 10/11/03
Assuming a uniform prior on $f_H$, $P(f_H) = 1$,
solve the problem posed in \exampleref{exa.bentcoin}.
Sketch the posterior distribution of $f_H$
and compute the probability that the $N\!+\!1$th outcome will be a head,
for
\ben
\item $N=3$ and $n_H=0$;
\item $N=3$ and $n_H=2$;
\item $N=10$ and $n_H=3$;
\item
$N=300$ and $n_H=29$.
\een
You will find the \ind{beta integral} useful:
\beq
\int_0^1 \! \d p_a \: p_a^{F_a} (1-p_a)^{F_b} =
\frac{\Gamma(F_a+1)\Gamma(F_b+1)}{ \Gamma(F_a+F_b+2) }
= \frac{ F_a! F_b! }{ (F_a + F_b + 1)! } .
\eeq
You may also find it instructive to look back at
\exampleref{ex.ip.urns} and \eqref{eq.laplace.succession.first}.
}
People sometimes confuse assigning a prior distribution
to an unknown parameter such as $f_H$ with making an initial guess
of the {\em{value}\/} of the parameter.
% But priors are not values, they are distributions.
But the prior over $f_H$, $P(f_H)$, is not a simple statement
like `initially, I would guess $f_H = \dhalf$'.
The prior is a probability density over $f_H$ which
specifies the prior degree of belief that $f_H$ lies
in any interval $(f,f+\delta f)$. It may well be the case
that our prior for $f_H$ is symmetric about $\dhalf$, so that the
{\em mean\/} of $f_H$ under the prior is $\dhalf$.
%under our prior for $f_H$, the {\em mean\/} of $f_H$ is $\dhalf$
% -- on symmetry grounds for example.
In this case, the
predictive distribution {\em for the first toss\/} $x_1$ would indeed be
\beq
P(x_1 \eq \mbox{head}) =
\int \! \d f_H \: P(f_H) P(x_1 \eq \mbox{head} \given f_H)
= \int \! \d f_H \: P(f_H) f_H = \dhalf .
\eeq
But the prediction for subsequent tosses will depend on
the whole prior distribution, not just its mean.
\subsubsection{Data compression and inverse probability}
Consider the following task.
\exampl{ex.compressme}{
Write a computer program capable of compressing binary files like this
one:\par
\begin{center}{\footnotesize%was tiny
{\tt 0000000000000000000010010001000000100000010000000000000000000000000000000000001010000000000000110000}\\
{\tt 1000000000010000100000000010000000000000000000000100000000000000000100000000011000001000000011000100}\\
{\tt 0000000001001000000000010001000000000000000011000000000000000000000000000010000000000000000100000000}\\[0.1in]% added this space Sat 21/12/02
}
\end{center}
% This file contains N=300 and n_1 = 29
The string shown contains $n_1=29$ {\tt 1}s
and $n_0=271$ {\tt 0}s.
% What is the probability that the next character in this file
% is a {\tt 1}?
}
Intuitively, compression works by taking advantage of the predictability
of a file. In this case, the source of the file
appears more likely to emit
{\tt 0}s than {\tt 1}s. A data compression program that compresses
this file must, implicitly or explicitly, be addressing the
question `What is the probability that the next character in this file
is a {\tt 1}?'
Do you think this problem is similar in character
to \exampleref{exa.bentcoin}? I do. One of the themes
of this book is that data compression and
data modelling are one and the same, and that they should
both be addressed, like the urn of example \ref{ex.ip.urns},
using inverse probability.
\Exampleonlyref{ex.compressme} is solved in \chref{ch4}.
%
% SOLVE IT HERE???
%
\subsection{The likelihood principle}
\label{sec.lp}
Please solve the following two exercises.
\exampl{ex.lp1}{
Urn\amarginfig{c}{\begin{center}\psfig{figure=figs/urnsA.ps,width=1.6in}\end{center}
\caption[a]{Urns for \protect\exampleonlyref{ex.lp1}.}}
A contains three balls: one black, and two white;
\ind{urn} B contains three balls: two black, and one white.
One of the urns is selected at random and one ball
is drawn. The ball is black. What is the probability
that the selected urn is urn A?
}
%
\exampl{ex.lp2}{
Urn\amarginfig{c}{\begin{center}\psfig{figure=figs/urns.ps,width=1.6in}\end{center}%
\caption[a]{Urns for \protect\exampleonlyref{ex.lp2}.}}
A contains five balls: one black, two white, one green and one pink;
urn B contains five hundred balls:
two hundred black, one hundred white, 50 yellow, 40 cyan, 30 sienna,
25 green, 25 silver, 20 gold, and 10 purple.
[One fifth of A's balls are black; two-fifths of B's are black.]
One of the urns is selected at random and one ball
is drawn. The ball is black. What is the probability
that the urn is urn A?
}
%
What do you notice about your solutions? Does each answer
depend on the detailed contents of each urn?
The details of the other possible outcomes and their probabilities
are irrelevant. All that matters is the probability of the outcome
that actually happened (here, that the ball drawn was black) given the different
hypotheses. We need only to know the {\em likelihood}, \ie,
how the probability of the data that happened varies with the
hypothesis.
This simple rule about inference
is known as the {\dbf\ind{likelihood principle}}.\label{sec.likelihoodprinciple}
%
% NOTE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% { \em (connect back to this point when discussing
% early stopping and inference in problems where the stopping rule is not known.)}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% README NOTE!!!!!!!!!!
\begin{conclusionbox}
{\sf The likelihood principle:}
given a generative model for data $d$ given parameters $\btheta$, $P(d \given \btheta)$,
and having observed a particular outcome $d_1$, all inferences\index{key points!likelihood principle}
and predictions should depend only on the function $P(d_1 \given \btheta)$.
\end{conclusionbox}
\noindent
In spite of the simplicity of this principle,
many classical statistical methods violate it.\index{classical statistics!criticisms}\index{sampling theory!criticisms}
% \newpage
\section{Definition of entropy and related functions}
\begin{description}
\item[The Shannon information content of an outcome $x$] is defined to be
% We define for each $x \in \A_X$, $
\beq
h(x) = \log_2 \frac{1}{P(x)} .
\eeq
% We can interpret $h(a_i)$ as the information content of the event
% $x \eq a_i$.
It is measured in bits. [The word `bit' is also used to
denote a variable whose value is 0 or 1; I hope context will
always make clear which of the two meanings is intended.]
\noindent
In the next few chapters, we will establish that
the Shannon information content $h(a_i)$ is indeed a natural measure of
the information content of the event $x \normaleq a_i$.
At that point, we will shorten the name of this quantity to
`the information content'.
\margintab{%
\begin{center}\small%footnotesize
%
% vertical table of a-z with probabilities, and information contents too;
% four decimal place
\begin{tabular}[t]{cccr} \toprule
$i$ & $a_i$ & $p_i$ & \multicolumn{1}{c}{$h(p_i)$} \\ \midrule
% $i$ & $a_i$ & $p_i$ & \multicolumn{1}{c}{$\log_2 \frac{1}{p_i}$} \\ \midrule
%
1 & {\tt a} &.0575 & 4.1 \\
2 & {\tt b} &.0128 & 6.3 \\
3 & {\tt c} &.0263 & 5.2 \\
4 & {\tt d} &.0285 & 5.1 \\
5 & {\tt e} &.0913 & 3.5 \\
6 & {\tt f} &.0173 & 5.9 \\
7 & {\tt g} &.0133 & 6.2 \\
8 & {\tt h} &.0313 & 5.0 \\
9 & {\tt i} &.0599 & 4.1 \\
10 &{\tt j} &.0006 & 10.7 \\
11 &{\tt k} &.0084 & 6.9 \\
12 &{\tt l} &.0335 & 4.9 \\
13 &{\tt m} &.0235 & 5.4 \\
14 &{\tt n} &.0596 & 4.1 \\
15 &{\tt o} &.0689 & 3.9 \\
16 &{\tt p} &.0192 & 5.7 \\
17 &{\tt q} &.0008 & 10.3 \\
18 &{\tt r} &.0508 & 4.3 \\
19 &{\tt s} &.0567 & 4.1 \\
20 &{\tt t} &.0706 & 3.8 \\
21 &{\tt u} &.0334 & 4.9 \\
22 &{\tt v} &.0069 & 7.2 \\
23 &{\tt w} &.0119 & 6.4 \\
24 &{\tt x} &.0073 & 7.1 \\
25 &{\tt y} &.0164 & 5.9 \\
26 &{\tt z} &.0007 & 10.4 \\
27 &{\tt{-}}&.1928 & 2.4 \\ \midrule
%27 &\verb+-+&.1928 & 2.4 \\ \midrule
& & & \\[-0.1in]
\multicolumn{3}{r}{
$\displaystyle \sum_i p_i \log_2 \frac{1}{p_i}$
} & 4.1 \\ \bottomrule % 4.11
\end{tabular}\\
\end{center}
% vertical table of a-z with probabilities, and information contents too;
\caption[a]{Shannon information contents of the outcomes {\tt a}--{\tt z}.}
\label{fig.monogram.log}
}
%
The fourth column in \tabref{fig.monogram.log} shows the Shannon
information content of the 27 possible outcomes when
a
random character is picked from an English document. The
outcome
% character
$x={\tt z}$ has a Shannon information content of
10.4 bits, and $x={\tt e}$ has an information content of 3.5 bits.
\item[The entropy of an ensemble $X$] is defined to be the average Shannon information
content of an outcome:
% from that ensemble:
\beq
H(X) \equiv \sum_{x \in \A_X} P(x) \log \frac{1}{P(x)},
\eeq
%\beq
% H(X) = \sum_i p_i \log \frac{1}{p_i},
%\eeq
with the convention for $P(x) \normaleq 0$ that \mbox{$0 \times \log 1/0 \equiv 0$},
since \mbox{$\lim_{\theta\rightarrow 0^{+}} \theta \log 1/\theta \normaleq 0 $}.
Like the information content, entropy is measured in bits.
When it is convenient, we may also write $H(X)$ as $H(\bp)$,
where $\bp$ is the vector $(p_1,p_2,\ldots,p_I)$.
Another name for the entropy of $X$ is the uncertainty of $X$.
\end{description}
\noindent
% The entropy is a measure of the information content or
% `uncertainty' of $x$. The question of why entropy is a
% fundamental measure of information content will be discussed in the
% forthcoming chapters. Here w
% was continued example
\exampl{eg.mono}{
The entropy of a
randomly selected letter in an English document
is about 4.11 bits, assuming its probability
is as given in \tabref{fig.monogram.log}.
%, p.\ \pageref{fig.monogram}.
% \tabref{tab.mono}.
We obtain this number by averaging $\log 1/p_i$ (shown in the fourth
column) under the probability distribution $p_i$ (shown in the third column).
}
We now note some properties of the entropy function.
\bit
\item
$H(X) \geq 0$ with equality iff $p_i \normaleq 1$ for one $i$.
[{`iff' means
`if and only if'.}]
\item Entropy is maximized if $\bp$ is uniform:
\beq
H(X) \leq \log(|\A_X|)
\:\: \mbox{ with equality iff $p_i \normaleq 1/|\A_X|$ for all $i$. }
\eeq
% \footnote{Exercise: Prove this assertion.}
{\sf Notation:}\index{notation!absolute value}\index{notation!set size}
the vertical bars `$|\cdot|$'
have two meanings.
% If $X$ is an ensemble, then
If $\A_X$ is a set,
$|\A_X|$ denotes the number of elements in $\A_X$;
if $x$ is a number,
% for example, the value of a random variable,
then $|x|$ is the absolute value of $x$.
\eit
%
% Mon 22/1/01
The {\dem\ind{redundancy}}
measures the fractional difference
between $H(X)$ and its maximum possible value,
$\log(|\A_X|)$.
\begin{description}%
\item[The redundancy of $X$] is:
\beq
1 - \frac{H(X)}{\log |\A_X|} .
\eeq
We won't make use of `redundancy'
% need this definition
in this book, so
I have not assigned a symbol to it.
% -- it would be redundant.
\end{description}
% ha ha
% funny but true.
% example: X is select a codeword from a code - H(X) = K, but |X| = 2^N
%
% Redundancy = 1 - R
% of code
\begin{description}% duplicated in _l1a and _p5A
\item[The joint entropy of $X,Y$] is:
\beq
H(X,Y) = \sum_{xy \in \A_X\A_Y} P(x,y) \log \frac{1}{P(x,y)}.
\eeq
Entropy is additive for independent random variables:
\beq
H(X,Y) = H(X) +H(Y) \:\mbox{ iff }\: P(x,y)=P(x)P(y).
\label{eq.ent.indep}% also appears in p5a (.again)
\eeq
\end{description}
\label{sec.entropy.end.parta}
Our definitions for information content
so far apply only to discrete probability distributions
over finite sets $\A_X$. The definitions can be extended
to infinite sets, though the entropy may then be infinite.
The case of a probability {\em density\/} over a continuous set is
addressed in section \ref{sec.entropy.continuous}.\index{probability!density}
Further important definitions and exercises to do with entropy
will come along in section \ref{sec.entropy.contd}.
\section{Decomposability of the entropy}
The entropy function satisfies a recursive property
that can be very useful when computing entropies.
For convenience, we'll stretch our notation\index{notation!entropy}
so that we can write $H(X)$ as $H(\bp)$, where
$\bp$ is the probability vector associated with the ensemble $X$.
Let's illustrate the property by an example first.
Imagine that a random variable $x \in \{ 0,1,2 \}$
is created by first flipping a fair coin to determine
whether $x = 0$; then, if $x$ is not 0,
flipping a fair coin a second time to determine whether
$x$ is 1 or 2.
The probability distribution of $x$ is
\beq
P( x\! =\! 0 ) = \frac{1}{2} ; \:\:
P( x\! =\! 1 ) = \frac{1}{4} ; \:\:
P( x\! =\! 2 ) = \frac{1}{4} .
\eeq
What is the entropy of $X$? We can either compute it by brute
force:
\beq
H(X) = \dfrac{1}{2} \log 2 + \dfrac{1}{4} \log 4 + \dfrac{1}{4} \log 4
= 1.5 ;
\eeq
or we can use the following decomposition, in which the value of $x$
is revealed gradually.
Imagine first learning whether $x\! =\! 0$, and then,
if $x$ is not $0$, learning which non-zero value is the case. The revelation
of whether $x\! =\! 0$ or not entails revealing a
binary variable whose probability distribution is $\{\dhalf,\dhalf \}$.
This revelation has an entropy $H(\dhalf,\dhalf) = \frac{1}{2} \log 2 +\frac{1}{2} \log 2 = 1\ubit$.
If $x$ is not $0$, we learn the value of the second coin flip.
This too is a
binary variable whose probability distribution is $\{\dhalf,\dhalf\}$, and whose entropy is
$1\ubit$.
We only get to experience the second revelation half the time, however,
so the entropy can be written:
\beq
H(X) = H( \dhalf , \dhalf ) + \dhalf \, H( \dhalf , \dhalf ) .
\eeq
Generalizing, the observation we are making about the entropy
of any probability distribution $\bp = \{ p_1, p_2, \ldots , p_I \}$
is that
\beq
H(\bp) =
H( p_1 , 1\!-\!p_1 )
+ (1\!-\!p_1)
H \! \left(
\frac{p_2}{1\!-\!p_1} ,
\frac{p_3}{1\!-\!p_1} , \ldots ,
\frac{p_I}{1\!-\!p_1}
\right) .
\label{eq.entropydecompose}
\eeq
When it's written as a formula, this property
looks regrettably ugly; nevertheless it is a simple
property and one that you should make use of.
Generalizing further, the entropy has the property for any $m$
that
\beqan
H(\bp) &=&
H\left[ ( p_1+p_2+\cdots+p_m ) , ( p_{m+1}+p_{m+2}+\cdots+p_I ) \right]
\nonumber
\\
&&+ ( p_1+
% p_2+
\cdots+p_m )
H\! \left(
\frac{p_1}{ ( p_1+\cdots+p_m ) } ,
% \frac{p_2}{ ( p_1+\cdots+p_m ) } ,
\ldots ,
\frac{p_m}{ ( p_1+\cdots+p_m ) }
\right)
\nonumber
\\
&& + ( p_{m+1}+
%p_{m+2}+
\cdots+p_I )
H \! \left(
\frac{p_{m+1}}{ ( p_{m+1}+\cdots+p_I ) } ,
% \frac{p_{m+2}}{ ( p_{m+1}+\cdots+p_I ) } ,
\ldots ,
\frac{p_I}{ ( p_{m+1}+\cdots+p_I ) }
\right) .
\nonumber
\\
\label{eq.entdecompose2}
\eeqan
\exampl{example.entropy}{
A source produces a character $x$
from the alphabet $\A = \{ {\tt 0}, {\tt 1}, \ldots, {\tt 9}, {\tt a}, {\tt b}, \ldots, {\tt z} \}$;
with probability $\dthird$, $x$ is a numeral (${\tt 0}, \ldots, {\tt 9}$);
with probability $\dthird$, $x$ is a vowel (${\tt a}, {\tt e}, {\tt i}, {\tt o}, {\tt u}$);
and with probability $\dthird$ it's one of the 21 consonants. All numerals are equiprobable,
and the same goes for vowels and consonants.
Estimate the entropy of $X$.
}
\solution\ \
$\log 3 + \frac{1}{3} ( \log 10 + \log 5 + \log 21 )= \log 3 + \frac{1}{3} \log 1050 \simeq \log 30\ubits$.\ENDsolution
%> pr log(36)/log(2)
%5.16992500144231
%> pr log(30)/log(2)
%4.90689059560852
%> pr (log(3) +log(1050)/3.0 )/log(2)
%4.93035370490565
% This may be compared with the maximum entropy for an alphabet
% of 36 characters, $\log 36\ubits$.
\section{Gibbs' inequality}
% We will also find useful the following:
\begin{description}
% SPACE PROBLEM HERE ...
\item[The \ind{relative entropy} {\em or\/} \ind{Kullback--Leibler divergence}]
\marginpar[t]{\small\raggedright{The `ei' in L{\bf{ei}}bler is pronounced\index{pronunciation}
the same as in h{\bf{ei}}st.}}between two probability distributions $P(x)$ and $Q(x)$
that are defined over the same alphabet $\A_X$ is\index{entropy!relative}\index{divergence}
\beq
D_{\rm KL}(P||Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)} .
\label{eq.KL}
\label{eq.DKL}
\eeq
The relative entropy satisfies {\dem\ind{Gibbs' inequality}}
\beq
D_{\rm KL}(P||Q) \geq 0
\eeq
with equality only if $P \normaleq Q$. Note that in general
the relative entropy is not symmetric under interchange of the
distributions $P$ and $Q$:
in general
$D_{\rm KL}(P||Q) \neq D_{\rm KL}(Q||P)$, so $D_{\rm KL}$,
although it is sometimes called the `\ind{KL distance}',
is not strictly a
distance\index{distance!$D_{\rm KL}$}.\index{distance!relative entropy}
% `distance\index{distance!$D_{\rm KL}$}'.
% It is also known as the `discrimination' or `divergence',
The \ind{relative entropy} is important in pattern recognition and neural networks,
as well as in information theory.
%
% could include that aston guy's stuff here on (pq)^1/2?
%
% see also ../notation.tex
%
\end{description}
Gibbs' inequality is probably the most important inequality in this book.
It, and many other inequalities, can be proved
using the concept of convexity.
\section{Jensen's inequality for convex functions}
\begin{aside}
The
words `\ind{\convexsmile}'
and `\ind{\concavefrown}' may be pronounced `convex-smile'
and `concave-frown'.
This terminology has useful redundancy: while one
may forget which way up `convex' and `concave' are,
it is harder to confuse a smile with a frown.\index{notation!convex/concave}
\end{aside}
\begin{description}
%
\item[{\Convexsmile\ functions}\puncspace] A function $f(x)$ is {\dem \ind{\convexsmile}\/}
over $(a,b)$ if
\amarginfig{c}{%
\footnotesize
\setlength{\unitlength}{0.75mm}
\begin{tabular}{c}
\begin{picture}(60,60)(0,0)
\put(0,0){\makebox(60,65){\psfig{figure=figs/convex.eps,angle=-90,width=45mm}}}
\put(10,8){\makebox(0,0){$x_1$}}
\put(48,8){\makebox(0,0){$x_2$}}
\put(17,2){\makebox(0,0)[l]{$x^* = \lambda x_1 + (1-\lambda)x_2$}}
\put(31,23){\makebox(0,0){$f(x^*)$}}
\put(35,39){\makebox(0,0){$\lambda f(x_1) + (1-\lambda)f(x_2)$}}
\end{picture}
\end{tabular}
\caption[a]{Definition of convexity.}
\label{fig.convex}
}\
every chord of the function
lies above the function,
as shown in \figref{fig.convex}; that is,
for all $x_1,x_2
\in (a,b)$ and $0\leq \lambda \leq 1$,
\beq
f( \lambda x_1 + (1-\lambda)x_2 ) \:\:\leq \:\:\
\lambda f(x_1) + (1-\lambda) f(x_2 ) .
\eeq
A function $f$ is {\dem strictly
\convexsmile\/} if, for all $x_1,x_2 \in (a,b)$, the equality holds only
for $\lambda \normaleq 0$ and $\lambda\normaleq 1$.
Similar definitions apply to \concavefrown\ and strictly \concavefrown\
functions.
\end{description}
\newcommand{\tinyfunction}[2]{
\begin{tabular}{@{}c@{}}
{\small{#1}}
\\[-0.25in]
\psfig{figure=figs/#2.ps,width=1.06in,angle=-90}
\\
\end{tabular}
}
Some strictly \convexsmile\ functions are
\bit
\item $x^2$, $e^x$ and $e^{-x}$ for all $x$;
\item $\log (1/x)$ and $x \log x$ for $x>0$.
\eit
\begin{figure}[htbp]
\figuremargin{%
\begin{center}
\raisebox{0.4in}{%
\begin{tabular}[c]{c@{}c@{}c@{}c}
\tinyfunction{$x^2$}{convex_xx} &
\tinyfunction{$e^{-x}$}{convex_exp-x} &
\tinyfunction{$\log \frac{1}{x}$}{convex_logix} &
\tinyfunction{$x \log x$}{convex_xlogx} \\[0.2in]
%\tinyfunction{$x^2$}{convex_xx} &
%\tinyfunction{$e^{-x}$}{convex_exp-x} \\[0.42in]
%\tinyfunction{$\log \frac{1}{x}$}{convex_logix} &
%\tinyfunction{$x \log x$}{convex_xlogx} \\[0.2in]
\end{tabular}
}
\end{center}
}{%
\caption[a]{\Convexsmile\ functions.}
\label{fig.convexf}
}%
\end{figure}
\begin{description}
\item[Jensen's inequality\puncspace] If $f$ is a \convexsmile\ function
and $x$ is a random variable then:
\beq
\Exp\left[ f(x) \right] \geq f\!\left( \Exp[x] \right) ,
\label{eq.jensen}
\eeq
where $\Exp$ denotes \ind{expectation}. If $f$ is strictly \convexsmile\ and
$\Exp\left[ f(x) \right] \normaleq f\!\left( \Exp[x] \right)$, then the random
variable $x$ is a constant.
% (with probability 1).
% |!!!!!!!!!!!!!!!!! removed pedantry
\ind{Jensen's inequality} can also be rewritten for a
\concavefrown\ function, with the direction of the inequality
reversed.
\end{description}
A physical version of Jensen's \ind{inequality} runs as follows.
\amarginfignocaption{b}{\mbox{\psfig{figure=figs/jensenmass.ps,width=1.75in,angle=-90}}}
\begin{quote}
If a collection of
masses $p_i$ are placed on a
\convexsmile\ curve $f(x)$
at locations $(x_i, f(x_i))$, then the
\ind{centre of gravity} of those masses, which is at $\left( \Exp[x],
\Exp\left[ f(x) \right] \right)$, lies above the curve.
\end{quote}
If this fails to convince you, then feel free to
do the following exercise.
\exercissxC{2}{ex.jensenpf}{
Prove \ind{Jensen's inequality}.
}
\exampl{ex.jensen}{
Three squares have average area $\bar{A} = 100\,{\rm m}^2$.
The average of the lengths of their sides is $\bar{l} = 10\,{\rm m}$.
What can be said about the size of the largest of the
three squares? [Use Jensen's inequality.]
}
\solution\ \
Let $x$ be the length of the side of a square, and let the
probability of $x$ be $\dthird,\dthird,\dthird$ over the
three lengths $l_1,l_2,l_3$. Then the information that we have is
that $\Exp\left[ x \right]=10$ and $\Exp\left[ f(x) \right]=100$,
where $f(x) = x^2$ is the function mapping lengths to areas.
This is a strictly \convexsmile\ function.
We notice that the equality
$\Exp\left[ f(x) \right] \normaleq f\!\left( \Exp[x] \right)$ holds,
therefore $x$ is a constant, and the three lengths
must all be equal. The area of the largest square is 100$\,{\rm m}^2$.\ENDsolution
\subsection{Convexity and concavity also relate to maximization}
If $f(\bx)$ is \convexfrown\ and there exists a point at which
\beq
\frac{\partial f}{\partial x_k} = 0 \:\: \mbox{for all $k$},
% \forall k
\eeq
then $f(\bx)$ has its maximum value at that point.
The converse does not hold: if a \convexfrown\ $f(\bx)$ is maximized at
some $\bx$ it is not necessarily true that the gradient
$\grad f(\bx)$ is equal
to zero there. For example, $f(x) = -|x|$ is maximized at $x=0$
where its derivative is undefined; and $f(p) = \log(p),$ for
a probability
$p \in (0,1)$, is maximized on the boundary of the range,
at $p=1$, where the gradient $\d f(p)/\d p =1$.
%, since $f$ might for example
% be an increasing function with no maximum such as $\log x$,
% or its maximum might be located at a point $\bx$
% on the boundary of the range of $\bx$.
%
%{\em (is this use of range correct?)}
% exercises from that.
%
% exercises that belong between old chapters 1 and 2.
%
% see also _p5a.tex for moved exercises.
%
\section{Exercises}
\subsection*{Sums of random variables}
% sums of random variables.
% dice questions
\exercissxA{3}{ex.sumdice}{
\ben
\item
Two ordinary dice with faces labelled $1,\ldots,6$
are thrown. What is the probability distribution of
the sum\index{law of large numbers}
of the values? What is the probability distribution of the
absolute difference between the values?
\item
One\marginpar[c]{\small\raggedright{This exercise
is intended to help you think about the \ind{central-limit theorem}, which says
that if independent random variables $x_1, x_2, \ldots, x_N$
have means $\mu_n$ and finite variances $\sigma_n^2$, then, in the
limit of large $N$, the sum $\sum_n x_n$ has a distribution that tends
to a normal (\index{Gaussian distribution}Gaussian) distribution
with mean $\sum_n \mu_n$ and variance $\sum_n \sigma_n^2$.}}
hundred ordinary dice are thrown. What,
roughly, is the probability distribution of the sum of the values?
Sketch the probability distribution and estimate its mean and
standard deviation.
\item
How can two cubical dice be labelled using the numbers $\{0,1,2,3,4,5,6\}$
so that when the two dice are thrown the sum has a uniform
probability distribution over the integers 1--12?
% Can you prove your solution is unique?
\item
Is there any way that one hundred dice
could be labelled with integers
such that the probability distribution of the sum is uniform?
\een
}
% answer, one normal, one 060606
% uniqueness proved by noting that every outcome 1-12 has
% to be made from 3 microoutcomes, and 12 can only be made
% from 6,6, so there must be a six on each die, indeed 3 on 1, and
% 1 on the other. 1 can only be mae from 1,0, and don't want 0,0,
% so there must be three 0s. (M Gardner)
%
\subsection*{Inference problems}
\exercissxA{2}{ex.logit}{
If $q=1-p$ and $a = \ln \linefrac{p}{q}$, show that
\beq
p = \frac{1}{1+\exp(-a)} .
\label{eq.sigmoid}
\label{eq.logistic}
\eeq
Sketch this function and find its relationship to the hyperbolic tangent
function $\tanh(u)=\frac{e^{u} - e^{-u}}{e^{u} + e^{-u}}$.
It will be useful to be fluent in base-2 logarithms also.
If $b = \log_2 \linefrac{p}{q}$, what is $b$ as a function of $p$?
}
%
% is this exercise inappropriate now because we have not defined
% joint ensembles yet?
%
\exercissxB{2}{ex.BTadditive}{
Let $x$ and $y$ be dependent
% correlated
random variables with
$x$ a binary variable taking values in $\A_X = \{ 0,1 \}$.
Use \Bayes\ theorem to show that the log posterior probability
ratio for $x$ given $y$ is
\beq
\log \frac{P(x\eq 1 \given y)}{P(x\eq 0 \given y)} = \log \frac{P(y \given x\eq 1)}{P(y \given x\eq 0)}
+ \log \frac{P(x\eq 1)}{P(x\eq 0)} .
\eeq
}
% define ODDS ?
\exercissxB{2}{ex.d1d2}{
Let $x$, $d_1$ and $d_2$ be random variables such that $d_1$
and $d_2$ are conditionally independent given a binary variable $x$.
% (That is, $P(x,d_1,d_2)
% = P(x)P(d_1 \given x)P(d_2 \given x)$.)
%
% somewhere I need to introduce graphical repns and define
%
% TO DO!!! TODO
%
% (\ind{conditional independence} is discussed further in section XXX.)
%
% and give examples. A and C children of B. and A->B->C
% Jensen defn is
% A is cond indep of B given C if
% A|B,C = A|C
% which is symmetric, implying by BT
% B|A,C = B|C
% pf
% B|A,C = A|B,C B|C / A|C = B|C
% my defn here is
% A,B,C = C A|C B|C
% proof:
% A,B,C = C A|C B|C,A = .
% NB graphical model and decomposition are not 1-1 related. The two
% graphs A and C children of B. and A->B->C both have a joint prob
% that can be factorized in either way.
%
% $x$ is a binary variable taking values in $\A_X = \{ 0,1 \}$.
Use \Bayes\ theorem to show that the posterior probability
ratio for $x$ given $\{d_i \}$ is
\beq
\frac{P(x\eq 1 \given \{d_i \} )}{P(x\eq 0 \given \{d_i \})} =
\frac{P(d_1 \given x\eq 1)}{P(d_1 \given x\eq 0)}
\frac{P(d_2 \given x\eq 1)}{P(d_2 \given x\eq 0)}
\frac{P(x\eq 1)}{P(x\eq 0)} .
\eeq
}
\subsection*{Life in high-dimensional spaces}
%{Life in $\R^N$}
\index{life in high dimensions}
\index{high dimensions, life in}
Probability distributions and volumes have some unexpected
properties in high-dimensional spaces.
% The real line is denoted by $\R$. An $N$--dimensional real space
% is denoted by $\R^N$.
\exercissxA{2}{ex.RN}{
Consider a sphere of radius $r$ in an $N$-dimensional real space.
% dimensions.
Show that the
fraction of the volume of the sphere that
is
in the surface shell lying
at values of the radius between $r- \epsilon$ and $r$, where $0 < \epsilon < r$, is:
\beq
f = 1 - \left( 1 - \frac{\epsilon}{r} \right)^{\!N} .
\eeq
% from Bishop p.29
Evaluate $f$ for the cases $N\eq 2$, $N\eq 10$
and $N\eq 1000$, with (a) $\epsilon/r \eq 0.01$; (b) $\epsilon/r \eq 0.5$.
{\sf Implication:} points that are uniformly distributed in a sphere in $N$
dimensions, where $N$ is large, are very likely to be in a \ind{thin shell}
near the surface.
% (From Bishop (1995).)
}
%
\label{sec.exercise.block1}
\subsection*{Expectations and entropies}
You are probably familiar with the idea of computing the \ind{expectation}\index{notation!expectation}
of a function of $x$,
\beq
\Exp\left[ f(x) \right] = \left< f(x) \right> = \sum_{x} P(x) f(x) .
\eeq
Maybe you are not so comfortable with computing this expectation
in cases where the function $f(x)$ depends on
the probability $P(x)$. The next few
examples address this concern.
\exercissxA{1}{ex.expectn}{
Let $p_a \eq 0.1$, $p_b \eq 0.2$, and $p_c \eq 0.7$.
Let $f(a) \eq 10$, $f(b) \eq 5$, and $f(c) \eq 10/7$.
What is $\Exp\left[ f(x) \right]$?
What is $\Exp\left[ 1/P(x) \right]$?
}
\exercissxA{2}{ex.invP}{
For an arbitrary ensemble, what is $\Exp\left[ 1/P(x) \right]$?
}
\exercissxB{1}{ex.expectng}{
Let $p_a \eq 0.1$, $p_b \eq 0.2$, and $p_c \eq 0.7$.
Let $g(a) \eq 0$, $g(b) \eq 1$, and $g(c) \eq 0$.
What is $\Exp\left[ g(x) \right]$?
}
\exercissxB{1}{ex.expectng2}{
Let $p_a \eq 0.1$, $p_b \eq 0.2$, and $p_c \eq 0.7$.
What is the probability that $P(x) \in [0.15,0.5]$?
What is
\[
P\left( \left| \log \frac{P(x)}{ 0.2} \right| > 0.05 \right) ?
\]
}
\exercissxA{3}{ex.Hineq}{
Prove the assertion that
$H(X) \leq \log(|\A_X|)$ with equality iff $p_i \normaleq 1/|\A_X|$ for all $i$.
($|\A_X|$ denotes the number of elements in the set $\A_X$.)
[Hint: use Jensen's inequality (\ref{eq.jensen}); if your
first attempt to use Jensen does not succeed, remember that
Jensen involves both a random variable and a function,
and you have quite a lot of freedom in choosing
these; think about whether
your chosen function $f$ should be convex or concave.]
% further hint: try $u\eq 1/p_i$ as the random variable.]
}
\exercissxB{3}{ex.rel.ent}{
Prove that the relative entropy (\eqref{eq.KL})
satisfies $D_{\rm KL}(P||Q) \geq 0$ (\ind{Gibbs' inequality})
with equality only if $P \normaleq Q$.
% You may find this result
% helps with the previous two exercises. Note (moved to _p5a.tex)
%
% refer to this in mean field theory chapter {ch.mft}
%
}
%
% Decomposability of the entropy
\exercisaxB{2}{ex.entropydecompose}{
Prove that the entropy is
indeed decomposable as described in
\eqsref{eq.entropydecompose}{eq.entdecompose2}.
}
\exercissxB{2}{ex.decomposeexample}{
A random variable $x \in \{0,1,2,3\}$ is selected
by flipping a bent coin with bias $f$ to determine whether
the outcome is in $\{0,1\}$ or $\{ 2,3\}$;
\amarginfignocaption{t}{%
\begin{center}\small%footnotesize
\setlength{\unitlength}{0.6mm}
\begin{picture}(30,50)(-10,-15)
\put(-6,25){{\makebox(0,0)[r]{$f$}}}
\put(-6,5){{\makebox(0,0)[r]{$1\!-\!f$}}}
\put(-10,15){\vector(1,1){17}}
\put(-10,15){\vector(1,-1){17}}
\put(10,35){\vector(1,1){10}}
\put(10,35){\vector(1,-1){10}}
\put(16,45){{\makebox(0,0)[r]{$g$}}}
\put(16,25){{\makebox(0,0)[r]{$1\!-\!g$}}}
\put(16,5){{\makebox(0,0)[r]{$h$}}}
\put(16,-15){{\makebox(0,0)[r]{$1\!-\!h$}}}
\put(10,-5){\vector(1,1){10}}
\put(10,-5){\vector(1,-1){10}}
\put(24,45){{\makebox(0,0)[l]{\tt 0}}}
\put(24,25){{\makebox(0,0)[l]{\tt 1}}}
\put(24,5){{\makebox(0,0)[l]{\tt 2}}}
\put(24,-15){{\makebox(0,0)[l]{\tt 3}}}
\end{picture}
\end{center}
}
then either flipping a second bent coin with bias $g$
or a third bent coin with bias $h$ respectively.
Write down the probability distribution of $x$.
Use the
decomposability of the entropy (\ref{eq.entdecompose2})
to find the entropy of $X$. [Notice how compact
an expression is obtained if you make use of the binary entropy
function $H_2(x)$, compared with writing out the four-term
entropy explicitly.]
Find the derivative of $H(X)$ with respect to $f$. [Hint: $\d H_2(x)/\d x = \log((1-x)/x)$.]
}
\exercissxB{2}{ex.waithead0}{
An unbiased coin is flipped until one head is thrown. What is the
entropy of the random variable $x \in \{1,2,3,\ldots\}$, the number of
flips?
Repeat the calculation for the case of a biased coin with probability $f$
of coming up heads.
[Hint: solve the problem both directly and by using the
decomposability of the entropy (\ref{eq.entropydecompose}).]
%
}
%
% removed joint entropy questions.
\section{Further exercises}
%
\subsection*{Forward probability}% problems}
\exercisaxB{1}{ex.balls}{
An urn contains $w$ white balls and $b$ black balls.
Two balls are drawn, one after the other, without replacement.
Prove that the probability that the first ball
is white is equal to the probability that the second is white.
}
%
\exercisaxB{2}{ex.buffon}{
A circular \ind{coin} of diameter $a$ is thrown onto a \ind{square} grid
whose squares are $b \times b$. ($aB$ given that $F>A$?)
}
\exercisaxB{2}{ex.liars}{
The inhabitants of an island tell the
truth one third of the time. They lie with probability 2/3.
On an occasion, after one of them made a statement,
you ask another `was that statement true?'
and he says `yes'.
What is the probability that the statement was indeed true?
% [Ans: 1/5].
}
%
\exercissxB{2}{ex.R3error}{
Compare two ways of computing the probability of error of
the repetition code $\Rthree$, assuming a binary
symmetric channel (you
did this once for \exerciseref{ex.R3ep}) and confirm that they
give the same answer.
\begin{description}
\item[Binomial distribution method\puncspace]
Add the probability that all three bits are
flipped to the probability that exactly two bits are flipped.
% Add the probability of all three bits'
% being flipped to the probability of exactly two bits' being flipped.
\item[Sum rule method\puncspace]
% Using the different possible inferences]
Using the \ind{sum rule},
compute the marginal probability that $\br$ takes on each of
the eight possible values, $P(\br)$.
[$P(\br) = \sum_s P(s)P(\br \given s)$.]
Then compute
the posterior probability of $s$ for each of the eight
values of $\br$. [In fact, by symmetry, only two example
cases
$\br = ({\tt0}{\tt0}{\tt0})$ and
$\br = ({\tt0}{\tt0}{\tt1})$ need be considered.]
\marginpar{\small\raggedright{\Eqref{eq.bayestheorem} gives the posterior probability of
the input $s$, given the received vector $\br$.
}}
% $\br = ({\tt1},{\tt1},{\tt0})$,
% $\br = ({\tt1},{\tt1},{\tt1})$,
Notice that some of the
inferred bits are better determined than others.
From the posterior probability $P(s \given \br)$ you can read out
the case-by-case error probability,
the probability that the more probable hypothesis
is not correct, $P(\mbox{error} \given \br)$.
Find the average error probability using the sum rule,
\beq
P(\mbox{error}) = \sum_{\br} P(\br) P(\mbox{error} \given \br) .
\eeq
\end{description}
}
%
\exercissxB{3C}{ex.Hwords}{
The frequency
% probability
$p_n$ of the
$n$th most frequent word in English is roughly approximated
by
\beq
p_n \simeq \left\{
\begin{array}{ll}
\frac{0.1}{n} & \mbox{for $n \in 1, \ldots, 12\,367$}
% 8727$.}
\\
0 & n > 12\,367 .
\end{array}
\right.
\eeq
[This remarkable $1/n$ law is known as \ind{Zipf's law},
and applies to the word frequencies of many languages
% cite Shannon collection p.197 - except he has the number 8727, wrong!
% could also cite Gell-Mann
\cite{zipf}.]
If we assume that English is generated by picking
words at random according to this distribution,
what is the entropy of English (per word)?
[This calculation can be found in `Prediction and entropy of printed English', C.E.\ Shannon,
{\em Bell Syst.\ Tech.\ J.}\ {\bf 30}, p\pdot50--64 (1950), but, inexplicably,
the great man made numerical errors in it.]
% , in bits per word?
}
%%% Local Variables:
%%% TeX-master: ../book.tex
%%% End:
% \input{tex/_e1A.tex}%%%%%%%%%%%%%%%%%%%%% inference probs to do with logit and dice and decay moved into _p8.tex
\dvips
% include urn.tex here for another forward probability exercise.
%
\section{Solutions}% to Chapter \protect\ref{ch.prob.ent}'s exercises}
\fakesection{_s1aa solutions}
%=================================
\soln{ex.independence.bigram}{
No, they are not independent. If they were then all the
conditional distributions $P(y \given x)$ would be identical
functions of $y$, regardless of $x$ (\cf\ \figref{fig.conbigrams}).
}
\soln{ex.fp.toss}{
We define the fraction $f_B \equiv B/K$.
\ben
\item
The number of black balls
has a binomial distribution.
\beq P(n_B\,|\,f_B,N) = {N \choose n_B} f_B^{n_B} (1-f_B)^{N-n_B} . \eeq
\item
The mean and variance of this distribution are:
\beq \Exp [ n_B ] = N f_B \eeq
\beq \var[n_B] = N f_B (1-f_B) .
\label{eq.variance.binomial}
\eeq
These results were derived in \exampleref{ex.binomial}.
The standard deviation of $n_B$ is $\sqrt{\var[n_B]} = \sqrt{N f_B (1-f_B)}$.
% on page \pageref{sec.first.binomial.sol}.
When $B/K = 1/5$ and $N=5$,
the expectation and variance of $n_B$ are
1 and 4/5. The standard deviation is 0.89.
When $B/K = 1/5$ and $N=400$,
the expectation and variance of $n_B$ are
80 and 64. The standard deviation is 8.
\een
}
\soln{ex.fp.chi}{
The numerator of the quantity
\[%beq
z = \frac{(n_B - f_B N)^2}{ {N f_B (1-f_B)} }
%\label{eq.chisquared}
\]%eeq
can be recognized as\index{chi-squared}\index{$\chi^2$}
$\left( n_B - \Exp [ n_B ] \right)^2$;
the denominator is equal to
the variance of $n_B$ (\ref{eq.variance.binomial}),
which is by definition the expectation of the numerator.
So the expectation of $z$ is 1. [A random variable like $z$,
which measures the deviation of data from the
expected
% average
value, is sometimes called $\chi^2$ (chi-squared).]
In the case $N=5$ and $f_B = 1/5$, $N f_B$ is 1, and
$\var[n_B]$ is 4/5. The numerator has five possible values, only
one of which is smaller than 1:
$(n_B - f_B N)^2 = 0$ has probability $P(n_B \eq 1)= 0.4096$;
% $(n_B - f_B N)^2 = 1$ has probability $P(n_B = 0)+P(n_B = 2)= $ ;
% $(n_B - f_B N)^2 = 4$ has probability $P(n_B = 3)= $ ;
% $(n_B - f_B N)^2 = 9$ has probability $P(n_B = 4)= $ ;
% $(n_B - f_B N)^2 = 16$ has probability $P(n_B = 5)= $ ;
so the probability that $z < 1$ is 0.4096.
%
}
%
% stole solution from here
%
%%%%%%%%%%%%%%%%%%%%%%%%%% added 99 9 14
\soln{ex.jensenpf}{
We wish to prove, given the property
\beq
f( \lambda x_1 + (1-\lambda)x_2 ) \:\: \leq \:\:
\lambda f(x_1) + (1-\lambda) f(x_2 ) ,
\label{eq.convexdefn}
\eeq
that, if $\sum p_i = 1$ and $p_i \geq 0$,
\beq%
% \Exp\left[ f(x) \right] \geq f\left( \Exp[x] \right) ,
\sum_{i=1}^I p_i f(x_i) \geq f\left( \sum_{i=1}^I p_i x_i \right) .
\eeq
We proceed by recursion, working from the right-hand side. (This proof
does not
% needs further work to
handle
% awkward
cases where some $p_i=0$; such
details are left to the pedantic reader.) At the first line we
use the definition of convexity (\ref{eq.convexdefn}) with
$\lambda = \frac{p_1}{\sum_{i=1}^I p_i } = p_1$; at the second line,
$\lambda = \frac{p_2}{\sum_{i=2}^I p_i }$.
% , and so forth.
\fakesection{temporary solution}
\begin{eqnarray}
\lefteqn{ f\left( \sum_{i=1}^I p_i x_i \right) =
% &=&
f\left( p_1 x_1 + \sum_{i=2}^I p_i x_i
\right) } \nonumber
\\
&\leq&
p_1 f(x_1) + \left[ \sum_{i=2}^I p_i \right]
\left[ f\left( \sum_{i=2}^I p_i x_i
\left/ \sum_{i=2}^I p_i \right. \right) \right]
\\
&\leq&
p_1 f(x_1) + \left[ \sum_{i=2}^I p_i \right]
\left[
\frac{p_2}
{\sum_{i=2}^I p_i } f\left( x_2 \right)
+ \frac{\sum_{i=3}^I p_i}
{\sum_{i=2}^I p_i }
f\left( \sum_{i=3}^I p_i x_i
\left/ \sum_{i=3}^I p_i \right. \right)
\right] ,
\nonumber
% probably cut this last line, just show one itn of recursion
%
\end{eqnarray}
and so forth. %
% this works if I want to restore it. Indeed I have restored it
\hfill $\epfsymbol$% $\Box$%\epfs% end proof symbol
}
%%%%%%%%%%%%%%%%%%%%
% main post-chapter exercise solution area:
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\soln{ex.sumdice}{
\ben \item For the outcomes $\{2,3,4,5,6,7,8,9,10,11,12\}$,
the probabilities are $\P = \{
\frac{1}{36},
\frac{2}{36},
\frac{3}{36},
\frac{4}{36},
\frac{5}{36},
\frac{6}{36},
\frac{5}{36},
\frac{4}{36},
\frac{3}{36},
\frac{2}{36},
\frac{1}{36}\}%
$.
\item The value of one die has mean $3.5$ and variance $35/12$.
So the sum of one hundred has mean $350$ and variance $3500/12 \simeq 292$,
and by the \ind{central-limit theorem} the probability distribution
is roughly Gaussian (but confined to the integers), with
this mean and variance.
\item
In order to obtain a sum that has a uniform distribution
we have to start from random variables some of which
have a spiky distribution
with the probability mass concentrated at the extremes.
The unique solution is to have one ordinary die and one with faces 6, 6, 6, 0, 0, 0.
% That this solution is unique can be proved with an argument
% that starts by noting
% that each of the 12 outcomes has to be realized
% by 3 distinct microstates (a microstate
% being one of the 36 particular orientations
% of the two dice). To create outcome `12'
% in three ways there must be one six on
% one dice and three sixes on the other;
% similarly to create outcome `1' three ways, there
% must be one die with three zeroes on it
% and one with one one.
\item
Yes, a uniform distribution can be created in several ways,\marginpar[t]{\small\raggedright{To think about:
does this uniform distribution contradict the \ind{central-limit theorem}?}}
for example by labelling the $r$th die with
the numbers $\{0,1,2,3,4,5\}\times 6^r$.
\een
}
% \subsection*{Inference problems}
%
\soln{ex.logit}{
\beqan
a = \ln \frac{p}{q}
\hspace{0.2in} & \Rightarrow & \hspace{0.2in} \frac{p}{q} = e^a
\label{logit.step1}
\eeqan
and $q=1-p$ gives
\beqan
\frac{p}{1-p} & =& e^a
\\ \Rightarrow \hspace{0.52in} p & = & \frac{e^a}{e^a+1} = \frac{1}{1+\exp(-a)} .
\label{logit.step2}
\eeqan
The hyperbolic tangent is
\beq
\tanh(a) = \frac{e^a -e^{-a}}{e^a + e^{-a}}
\eeq
so
\beqan
f(a)& \equiv& \frac{1}{1+\exp(-a)} =
\frac{1}{2} \left( \frac{1-e^{-a}}{1+e^{-a}} + 1 \right) \nonumber \\
&=& \frac{1}{2}\left( \frac{ e^{a/2} - e^{-a/2} }{
e^{a/2} + e^{-a/2}} +1 \right)
= \frac{1}{2} ( \tanh(a/2) + 1 ) .
\eeqan
In the case $b = \log_2 \linefrac{p}{q}$, we can repeat
steps (\ref{logit.step1}--\ref{logit.step2}), replacing $e$ by $2$, to
obtain
\beq
p = \frac{1}{1+2^{-a}} .
\label{eq.sigmoid2}
\label{eq.logistic2}
\eeq
}
\soln{ex.BTadditive}{
\beqan
P(x \given y) &=& \frac{P(y \given x)P(x) }{P(y)}
\\%\eeq\beq
\Rightarrow\:\:
\frac{P(x\eq 1 \given y)}{P(x\eq 0 \given y)} &=& \frac{P(y \given x\eq 1)}{P(y \given x\eq 0)}
\frac{P(x\eq 1)}{P(x\eq 0)}
\\%\eeq\beq
\Rightarrow\:\:
\log \frac{P(x\eq 1 \given y)}{P(x\eq 0 \given y)} &=& \log \frac{P(y \given x\eq 1)}{P(y \given x\eq 0)}
+ \log \frac{P(x\eq 1)}{P(x\eq 0)} .
\eeqan
}
\soln{ex.d1d2}{
The conditional independence of $d_1$ and $d_2$ given $x$
means
\beq
P(x,d_1,d_2) = P(x)P(d_1 \given x)P(d_2 \given x) .
\eeq
This gives a separation of the posterior probability ratio
into a series of factors, one for each data point, times
the prior probability ratio.
\beqan
\frac{P(x\eq 1 \given \{d_i \} )}{P(x\eq 0 \given \{d_i \})} &=&
\frac{P(\{d_i\} \given x\eq 1)}{P(\{d_i\} \given x\eq 0)}
\frac{P(x\eq 1)}{P(x\eq 0)}
\\ &=&
\frac{P(d_1 \given x\eq 1)}{P(d_1 \given x\eq 0)}
\frac{P(d_2 \given x\eq 1)}{P(d_2 \given x\eq 0)}
\frac{P(x\eq 1)}{P(x\eq 0)} .
\eeqan
}
%
%
\subsection*{Life in high-dimensional spaces}
\soln{ex.RN}{
The \ind{volume} of a \ind{hypersphere} of radius $r$ in $N$ dimensions is
in fact
\beq
V(r,N) = \frac{\pi^{N/2}}{(N/2)!} r^{N} ,
\eeq
but you don't need to know this.
For this question all that we need is the $r$-dependence,
$V(r,N) \propto r^{N} .$
So the fractional volume in $(r-\epsilon,r)$ is
\beq
\frac{ r^{N} - (r-\epsilon)^N }{ r^N} =
1 -\left( 1 -\frac{\epsilon}{r}\right)^N .
\eeq
The fractional volumes in the shells for the required cases are:
\begin{center}
\begin{tabular}[t]{cccc} \toprule
$N$ & 2 & 10 & 1000 \\ \midrule
$\epsilon/r = 0.01$ & 0.02 & 0.096 & 0.99996 \\
$\epsilon/r = 0.5\phantom{0}$ & 0.75 & 0.999 & $1 - 2^{-1000}$ \\ \bottomrule
\end{tabular}\\
\end{center}
\noindent Notice that no matter how small $\epsilon$ is, for large enough $N$
essentially all the probability mass is in the surface shell of thickness
$\epsilon$.
}
%\soln{ex.weigh}{
% See chapter \chtwo.
%}
%
\soln{ex.expectn}{
$p_a \eq 0.1$, $p_b \eq 0.2$, $p_c \eq 0.7$.
$f(a) \eq 10$, $f(b) \eq 5$, and $f(c) \eq 10/7$.
\beq
\Exp\left[ f(x) \right] = 0.1 \times 10 + 0.2 \times 5 + 0.7 \times 10/7 = 3.
\eeq
For each $x$, $f(x) = 1/P(x)$, so
\beq
\Exp\left[ 1/P(x) \right] = \Exp\left[ f(x) \right] = 3.
\eeq
}
%
\soln{ex.invP}{
For general $X$,
\beq
\Exp\left[ 1/P(x) \right] = \sum_{x\in \A_X} P(x) 1/P(x) =
\sum_{x\in \A_X} 1 = | \A_X | .
\eeq
}
%
\soln{ex.expectng}{
$p_a \eq 0.1$, $p_b \eq 0.2$, $p_c \eq 0.7$.
$g(a) \eq 0$, $g(b) \eq 1$, and $g(c) \eq 0$.
\beq
\Exp\left[ g(x) \right]=p_b = 0.2.
\eeq
}
\soln{ex.expectng2}{
\beq
P\left( P(x) \! \in \! [0.15,0.5] \right) = p_b = 0.2 .
\eeq
\beq
P\left( \left| \log \frac{P(x)}{ 0.2} \right| > 0.05 \right)
= p_a + p_c = 0.8 .
\eeq
}
%
\soln{ex.Hineq}{
This type of question can be approached in two ways:
either by differentiating
the function to be maximized, finding the maximum, and proving
it is a global maximum; this strategy is somewhat risky since it is possible
for the maximum of a function to be at the boundary of the space,
at a place where the derivative is not zero.
Alternatively, a carefully chosen inequality
can establish the answer. The second method is much neater.
\begin{Prooflike}{Proof by differentiation (not the recommended method)}
Since it is slightly easier to differentiate $\ln 1/p$ than $\log_2 1/p$,
we temporarily define $H(X)$ to be measured using natural logarithms, thus
scaling it down by a factor of $\log_2 e$.
\beqan
H(X) &=& \sum_i p_i \ln \frac{1}{p_i} \\
\frac{\partial H(X)}{\partial p_i} &=& \ln \frac{1}{p_i} - 1
\eeqan
we maximize subject to the constraint $\sum_i p_i = 1$ which can be enforced
with a Lagrange multiplier:
\beqan
G(\bp) & \equiv & H(X) + \lambda \left( \sum_i p_i - 1 \right) \\
\frac{\partial G(\bp)}{\partial p_i} &=& \ln \frac{1}{p_i} - 1 + \lambda .
\eeqan
At a maximum,
\beqan
\ln \frac{1}{p_i} - 1 + \lambda &=& 0 \\
\Rightarrow \ln \frac{1}{p_i} &=& 1 - \l ,
\eeqan
so all the $p_i$ are equal. That this extremum is indeed a maximum
is established by finding the curvature:
\beq
\frac{\partial^2 G(\bp)}{\partial p_i \partial p_j} = -\frac{1}{p_i}
\delta_{ij} ,
\eeq
which is negative definite. \hfill
\end{Prooflike}
\begin{Prooflike}{Proof using Jensen's inequality (recommended method)}
First a reminder of the inequality.
\begin{quotation}
\noindent
If $f$ is a \convexsmile\ function
and $x$ is a random variable then:
\[%beq
\Exp\left[ f(x) \right] \geq f\left( \Exp[x] \right) .
\]%eeq
If $f$ is strictly \convexsmile\ and
$\Exp\left[ f(x) \right] \eq f\left( \Exp[x] \right)$, then the random
variable $x$ is a constant
(with probability 1).
\end{quotation}
The secret of a proof using Jensen's inequality is to choose the
right function and the right random variable.
We could define
% $f(u) = \log \frac{1}{u}$ and
\beq
f(u) = \log \frac{1}{u} = - \log u
\eeq
(which is a convex function) and
think of $H(X) = \sum p_i \log \frac{1}{p_i}$ as the
mean of $f(u)$ where $u=P(x)$, but this
would not get us there -- it would give us an inequality in the
wrong direction. If instead we define
\beq
u = 1/P(x)
\eeq
then we find:
% this introduces an extra minus sign:
\beq
H(X) = - \Exp\left[ f( 1/P(x) ) \right]
\leq - f\left( \Exp[ 1/P(x) ] \right) ;
\eeq
now we know from \exerciseref{ex.invP}\ that $\Exp[ 1/P(x) ] = |\A_X|$, so
\beq
H(X) \leq - f\left( |\A_X| \right) = \log |\A_X| .
\eeq
Equality holds only if the random variable $u = 1/P(x)$ is a constant,
which means $P(x)$ is a constant for all $x$.
\end{Prooflike}
}
%
\soln{ex.rel.ent}{
\beq
D_{\rm KL}(P||Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)} .
% \label{eq.KL}
\eeq
\label{sec.gibbs.proof}% cross ref problem? Tue 12/12/00
We prove \ind{Gibbs' inequality} using \ind{Jensen's inequality}.
Let $f(u) = \log 1/u$ and $u=\smallfrac{Q(x)}{P(x)}$.
Then
\beqan
D_{\rm KL}(P||Q) & =& \Exp[ f( Q(x)/P(x) ) ]
\\ &\geq&
f\left(
\sum_x P(x) \frac{Q(x)}{P(x)} \right)
= \log \left( \frac{1}{\sum_x Q(x)} \right) = 0,
\eeqan
with equality only if $u=\frac{Q(x)}{P(x)}$ is a constant, that is,
if $Q(x) = P(x)$.\hfill$\epfsymbol$\\
\begin{Prooflike}{Second solution}
In the above proof the expectations were with respect to
the probability distribution $P(x)$. A second solution method
uses Jensen's inequality with $Q(x)$ instead.
We define $f(u) = u \log u$ and let $u = \frac{P(x)}{Q(x)}$.
Then
\beqan
D_{\rm KL}(P||Q)& =&
\sum_x Q(x) \frac{P(x)}{Q(x)} \log
\frac{P(x)}{Q(x)} = \sum_x Q(x) f\left( \frac{P(x)}{Q(x)} \right) \\
&\geq& f\left( \sum_x Q(x) \frac{P(x)}{Q(x)} \right) = f(1) = 0,
\eeqan
with equality only if $u=\frac{P(x)}{Q(x)}$ is a constant, that is,
if $Q(x) = P(x)$.
\end{Prooflike}
}
%
% solns moved to _s5A.tex
%
\soln{ex.decomposeexample}{
\beq
H(X)= H_2(f) + f H_2(g) + (1-f) H_2(h) .
\eeq
}
%
\soln{ex.waithead0}{
The probability that there are $x-1$ tails and then one head
(so we get the first head on the $x$th
toss) is
\beq
P(x) = (1-f)^{x-1} f .
\eeq
If the first toss is a tail, the probability distribution for
the future looks just like it did before we made the first toss.
Thus we have a recursive expression for the entropy:
\beq
H(X) = H_2( f ) + (1-f) H(X) .
\eeq
Rearranging,
\beq
H(X) = H_2( f ) / f .
\eeq
}
%
%
\fakesection{waithead solution}
\soln{ex.waithead}{
The probability of the number of tails $t$ is
\beq
P(t) = \left(\frac{1}{2}\right)^{\!t} \frac{1}{2}
\:\mbox{ for $t\geq 0$}.
\eeq
The expected number of heads is 1, by definition of the problem.
The expected number of tails is
\beq
\Exp[t] =
\sum_{t=0}^{\infty} t \left(\frac{1}{2}\right)^{\!t} \frac{1}{2} ,
\eeq
which may be shown to be 1 in a variety of ways. For example, since
the situation after one tail is thrown is equivalent to the opening
situation, we can write down the recurrence relation
\beq
\Exp[t] = \frac{1}{2} ( 1 + \Exp[t] ) + \frac{1}{2}0 \:\:
\Rightarrow \:\: \Exp[t] = 1.
\eeq
% if we define $S=\Exp[t]$ then we can subtract $S/2$ from $S$ to obtain
% a geometric series:
%\beq
% (1-1/2)S = \sum_{t=0}^{\infty} \left(\frac{1}{2}\right)^{t+1}
% = \frac{1/2}{1-1/2} = 1
%\eeq
% which gives $S=2$ --- what?
%%%%%%%%%%%%%%%%
%, for example, introducing
% $Z(\beta) \equiv \sum_t \left(\frac{1}{2}\right)^{\beta t} \frac{1}{2}
% = \frac{1}{2}/\left(1 - (\linefrac{1}{2})^{\beta}\right)$:
%\beq
% \sum_{t=0}^{\infty} t \left(\frac{1}{2}\right)^{t} \frac{1}{2}
% = \frac{\d}{\d\beta} \log Z
%\eeq
The probability distribution of the `estimator' $\hat{f} = 1/(1+t)$,
given that $f=1/2$, is plotted
in \figref{fig.f.estimator}. The probability of $\hat{f}$ is
simply the probability of the corresponding
value of $t$.
%
% gnuplot
% load 'figs/festimator.gnu'
%\begin{figure}
%\figuremargin{%
\marginfig{%
\begin{center}
\begin{tabular}{c}
$P(\hat{f})$\\[-0.3in]
\mbox{\psfig{figure=figs/festimator.ps,angle=-90,width=2in}}\\
\hspace{1.82in}$\hat{f}$
\end{tabular}
\end{center}
%}{%
\caption[a]{The probability distribution of the estimator $\hat{f} = 1/(1+t)$,
given that $f=1/2$.}
% , so that $P(t) = 1/2^{t+1}$.}
\label{fig.f.estimator}
%}
%\end{figure}
}
}
\soln{ex.waitbus}{
\ben
\item
The mean number of rolls from one six to the next six is six
(assuming
we
% don't count the first of the two sixes).
start counting rolls after the first of the two sixes).
The probability that the next six occurs on the $r$th
roll is the probability of {\em not\/} getting a six
for $r-1$ rolls multiplied by the probability of then
getting a six:
\beq
P(r_1 \eq r) = \left( \frac{5}{6} \right)^{\! r-1} \frac{1}{6}, \:\: \mbox{for $r\in \{1,2,3,\ldots \}$.}
\eeq
This probability distribution of the number of rolls, $r$,
may be called
an \ind{exponential distribution}, since
\beq
P(r_1 \eq r) = e^{-\alpha r} / Z,
\eeq
where $\alpha = \ln({6}/5)$, and $Z$ is a normalizing constant.
\item
The mean number of rolls from the clock until the next six is six.
\item
The mean number of rolls, going back in time,
until the most recent six is six.
\item
The mean number of rolls from the six before
the clock struck to the six after the clock struck
is the sum of the answers to (b) and (c), less one,
% (assuming we don't count the first of the two sixes),
that is, eleven.
\item
Rather than explaining the difference between (a)
% six and
and (d), let me give another hint.\index{bus-stop paradox}\index{waiting for a bus}
% see gnu/waitbus.gnu
Imagine that the buses in Poissonville arrive independently at random
(a \ind{Poisson process}), with, on average, one bus every six minutes.
Imagine that passengers turn up at {\busstop}s at a uniform rate,
% random also,
and are scooped up by the bus without delay, so the
interval between two buses remains constant.
Buses that follow gaps bigger than six minutes
become overcrowded. The passengers' representative complains that
two-thirds of all passengers found themselves on overcrowded buses.
The bus operator claims, `no, no -- only one third
of our buses are overcrowded'. Can both these claims be true?
\een
\amarginfig{b}{%
\begin{center}
\mbox{\hspace{-0.3in}\psfig{figure=figs/waitbus.ps,angle=-90,width=2.05in}}\\[-0.2in]
\end{center}
\caption[a]{The probability distribution of the number
of rolls $r_1$
from one 6 to the next
(falling solid line),
\[%\beq
P(r_1 \eq r) = \left( \frac{5}{6} \right)^{\! r-1} \frac{1}{6} ,
\]%\eeq
and the probability distribution (dashed line)
of
% the quantity $r_{\rm tot}=r_1+r_2-1$,
the number of rolls from the 6 before 1pm to the next 6,
% where $r_1$ and $r_2$ are the numbers of rolls before
% and after the clock strikes,
$r_{\rm tot}$,
\[%\beq
P(r_{\rm tot} \eq r) = r \, \left( \frac{5}{6} \right)^{\! r-1}
\left( \frac{1}{6} \right)^{\! 2 }
.
\]%\eeq
The probability $P(r_1>6)$ is about 1/3; the probability
$P(r_{\rm tot} > 6 )$ is about 2/3. The mean of $r_1$ is 6, and the
mean of $r_{\rm tot}$ is 11.
}
% other elegant ways of saying it:
% P( number rolls from one 6 to the next)
% P( number of rolls from the 6 before 1pm to the next)
}% end figure
}% end solbn
%
% \subsection{Move this solution}
%
% \subsection*{Conditional probability}
% \soln{ex.R3error}{
%
\fakesection{r3 error soln}
\soln{ex.R3error}{
\begin{description}
\item[Binomial distribution method\puncspace]
From the solution to \exerciseonlyref{ex.R3ep},
$p_B = 3 f^2 (1-f) + f^3$.\index{repetition code}
\item[Sum rule method\puncspace]
The marginal probabilities of the eight values of $\br$ are\index{sum rule}
illustrated by:
\beq
P(\br \eq {\tt0}{\tt0}{\tt0} ) = \dhalf (1-f)^3 + \dhalf f^3 ,
\eeq
\beq
P(\br \eq {\tt0}{\tt0}{\tt1} ) = \dhalf f(1-f)^2 + \dhalf f^2(1-f)
= \dhalf f(1-f) .
\eeq
The posterior probabilities are represented by
\beq
P( s\eq{\tt1} \given \br \eq {\tt0}{\tt0}{\tt0} ) = \frac{ f^3 }
{ (1-f)^3 + f^3 }
\eeq
and
\beq
P( s\eq{\tt1} \given \br \eq {\tt0}{\tt0}{\tt1} )
= \frac{ (1-f)f^2 }
{ f(1-f)^2 + f^2(1-f) }
= f .
\eeq
The probabilities of error in these representative cases are thus
\beq
P(\mbox{error} \given \br \eq {\tt0}{\tt0}{\tt0} ) = \frac{ f^3 }
{ (1-f)^3 + f^3 }
\eeq
and
\beq
P(\mbox{error} \given \br \eq {\tt0}{\tt0}{\tt1} ) = f .
\eeq
Notice that while the average probability of error of $\Rthree$ is
about $3 f^2$, the probability (given $\br$)
that any {\em{particular}\/} bit is
wrong is either about $f^3$ or $f$.
The average error probability, using the sum rule, is
\beqa
P(\mbox{error}) &=& \sum_{\br} P(\br) P(\mbox{error} \given \br) \\
&=& 2 [\dhalf (1-f)^3 + \dhalf f^3] \frac{ f^3 }
{ (1-f)^3 + f^3 }
+ 6 [\dhalf f(1-f)] f .
\eeqa
\marginpar{\vspace{-0.8in}\par\small\raggedright{The first two terms are for the cases $\br = \tt000$ and $\tt111$;
the remaining 6 are for the other outcomes, which share the
same
probability of occurring and identical error probability, $f$.}}%
So
\beqa
P(\mbox{error})
&=& f^3 + 3 f^2(1-f) .
\eeqa
\end{description}
}
%
%
% see also _s1A.tex
\soln{ex.Hwords}{
The entropy is 9.7
% 11.8
bits per word.
% , which is 2.6 bits per letter WRONG - shannon (p197) is in error
}
%\soln{ex.Hwords}{
%
% z := 1.000004301
%
%sum( 0.1/n * log(1.0/(0.1/n))/log(2.0) , n=1..12367) ;
% 9.716258456
% 9.716 bits.
%}
%\input{tex/_s1a.tex} nothing there any more
\fakesection{_s1A solutions}
%=================================
% quake
%
% \subsection*{Solutions to further inference problems}
%\soln{ex.exponential}{
% See chapter \chbayes.
%}
%\soln{ex.blood}{
% See chapter \chbayes.
%}
%
% The other exercises are discussed in the next chapter.
%%%%%%%%%%%%%%%%%%%%%%%%%%
\dvipsb{solutions 1a}
% now another inference chapter !
\prechapter{About Chapter}
\fakesection{About the first Bayes chapter}
If you are eager to get on to
% with data compression, information content and entropy,
information theory, data compression, and noisy channels,
you can skip to \chapterref{ch2}.
Data compression and data modelling are
intimately connected, however, so you'll probably
want to come back to this chapter
by the time you get to \chapterref{ch4}.
%
% move this later
%
% The exercises in this chapter are not a prerequisite for
% chapters \ref{ch2}--\ref{ch7}.
\fakesection{prerequisites for chapter 8}
Before reading \chapterref{ch.bayes},
it might be good to look at the following exercises.
% you
% should have worked on
% finished
% all the exercises in chapter \chone, in particular,
% \exerciserefrange{ex.logit}{ex.exponential}.
%
% \exthirtyone--\exthirtysix.
% uvw to HXY>0
\exercissxB{2}{ex.dieexponential}{
A die is selected at random from two twenty-faced dice
on which the symbols 1--10 are written with nonuniform frequency
as follows.
\begin{center}
\begin{tabular}{l@{\hspace{0.2in}}*{10}{l}} \toprule
Symbol & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 \\ \midrule
Number of faces of die A &
6 & 4 & 3 & 2 & 1 &1 &1 &1 &1 & 0 \\
Number of faces of die B &
3 & 3 & 2 & 2 & 2 &2 &2 &2 &1 & 1 \\ \bottomrule
\end{tabular}
\end{center}
The randomly chosen die is rolled 7 times, with the following
outcomes:
\begin{center}
5, 3, 9, 3, 8, 4, 7. % Sat 21/12/02 tried cutting this \\
\end{center}
What is the probability that the die is die A?
}
\exercissxB{2}{ex.dieexponentialb}{
Assume that there is a third twenty-faced die, die C, on which the symbols
1--20 are written once each.
As above, one of the three dice is selected at random and rolled
7 times, giving the outcomes:
% \begin{center}
3, 5, 4, 8, 3, 9, 7. \\
% \end{center}
What is the probability that the die is (a) die A, (b) die B, (c) die C?
}
% no normal solution pointer
\exercissxA{3}{ex.exponential}{ {\exercisetitlestyle Inferring a decay constant}\\
%\begin{quotation}
Unstable particles are emitted from a source and decay at a
distance $x$, a real number
that has an exponential probability distribution
with characteristic length $\lambda$. Decay events can only
be observed if they occur in a window extending from $x=1\cm$
to $x=20\cm$. $N$ decays are observed at locations $\{x_1 ,
\ldots , x_N\}$.
% ($x_n$ is a real number.)
What is $\lambda$?
%\end{quotation}
\begin{center}
\mbox{\psfig{figure=\FIGS/decay.ps,width=3in,angle=90,%
bbllx=154mm,bblly=147mm,bbury=257mm,bburx=175mm}}\\
\end{center}
}
% no normal solution pointer
% \subsection*{Genetic test evidence}
% \begin{quotation}
\exercissxB{3}{ex.blood}{ {\exercisetitlestyle Forensic evidence} \\
% Two people have left traces of their own blood at the scene of a
% crime. Their blood groups can be reliably identified from these
% traces and are found
% to be of type `O' (a common type in the local population, having
% frequency 60\%) and of type `AB' (a rare type, with frequency 1\%).
% A suspect is tested and found to have type `O' blood.
% A careless lawyer might claim that the fact that the suspect's
% blood type was found at the scene is positive evidence for the theory
% that he was present. But do these data
% $D=$ \{type `O' and `AB' blood were found at scene\} make it more
% probable that this suspect was one of the two people present at the
% crime?
Two people have left traces of their own blood at the scene of a
crime.
A suspect, Oliver, is tested and found to have type `O' blood.
The blood groups of the two traces
are found
to be of type `O' (a common type in the local population, having
frequency 60\%) and of type `AB' (a rare type, with frequency 1\%).
Do these data
(type `O' and `AB' blood were found at scene) give evidence in favour
of the proposition that Oliver was one of the two people present at the
crime?
}
% \end{quotation}
%%%%%%%%%% (many are repeated from _s1aa)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \prechapter{About Chapter}
\mysetcounter{page}{54}
\ENDprechapter
\chapter{More about Inference}
\label{ch.bayes}\label{ch1b}
% contains the decay problem, the bent coin, and blood.
%
%
% solutions to exercises are in _s8.tex
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\fakesection{Inference intro}
It is not a controversial statement that \Bayes\ theorem\index{Bayes' theorem}
provides the correct language for describing the inference of a
message communicated over a
noisy channel, as we used it in \chref{ch1} (\pref{sec.bayes.used}).
But strangely, when it comes to other
inference problems, the use of
% approaches based on
\Bayes\ theorem
is not so widespread.
%let's take a little tour of other applications of
% probabilistic inference.
% Coherent inference can always be mapped onto probabilities (Cox, 1946).
%% \cite{cox}.
% Many
% textbooks on statistics do not mention this fact, so maybe it is worth
% using an example to emphasize the contrast between Bayesian inference
% and the orthodox methods of statistical inference.
%% involving
%% estimators, confidence intervals, hypothesis testing, etc.
% If this topic interests you, excellent further reading is
% to be found in the works of Jaynes, for example,
% \citeasnoun{Jaynes.intervals}.
\section{A first inference problem}
\label{sec.decay}\label{ex.exponential.sol}% special label by hand
When I was an undergraduate in Cambridge, I was privileged to receive
supervisions from Steve Gull. Sitting at his desk in a dishevelled
office in St.\ John's College, I asked him how one ought to answer an
old Tripos question (\exerciseonlyref{ex.exponential}):
\begin{quotation}
Unstable particles are emitted from a source and decay at a
distance $x$, a real number
that has an exponential probability distribution
with characteristic length $\lambda$. Decay events can only
be observed if they occur in a window extending from $x=1\cm$
to $x=20\cm$. $N$ decays are observed at locations $\{x_1 ,
\ldots , x_N\}$.
% ($x_n$ is a real number.)
What is $\lambda$?
\end{quotation}
\begin{center}
\mbox{\psfig{figure=\FIGS/decay.ps,width=3in,angle=90,%
bbllx=154mm,bblly=147mm,bbury=257mm,bburx=175mm}}\\
\end{center}
I had scratched my head over this for some time.
My education had provided me with a couple of approaches to solving
such inference problems: constructing `\ind{estimator}s'
of the unknown parameters; or `fitting' the model to
the data, or to a processed version of the data.
Since the mean of an unconstrained exponential distribution is $\l$,
it seemed reasonable to examine the sample mean $\bar{x} = \sum_n x_n / N$
and see
if an estimator $\hat{\l}$ could be obtained from it.
It was evident that the {estimator}
$\hat{\l}=\bar{x}-1$ would be appropriate for
$\lambda \ll 20\,$cm, but not for cases where the
truncation of the distribution at the right-hand side
is significant; with a little ingenuity and the introduction of
ad hoc bins, promising estimators for $\lambda \gg 20$ cm could be
constructed. But there was no obvious estimator that would work
under all conditions.
Nor could I find a satisfactory
approach based on fitting the density $P(x\given \lambda)$ to
a histogram derived from the data. I was stuck.
What is the general solution to this problem and others like it?
Is it always necessary, when confronted by a new inference problem,
to grope in the dark for appropriate `estimators' and worry
about finding the `best' estimator (whatever that means)?
%% I hope you have already stopped and thought about this question.
% problem.
% \\ \mbox{~}\dotfill\ \mbox{~} \\
% \newpage
Steve
% Gull
wrote down the probability of one data point, given $\l$:
\beq
P(x\given \lambda) =\left\{ \begin{array}{ll}
{\textstyle \smallfrac{1}{\l}} \,
e^{-x/\lambda } / Z(\lambda) & 1 < x < 20 \\
0 & {\rm otherwise }
\end{array} \right.
\label{basic.likelihood}
\eeq
where
\beq
Z(\l) = \int_1^{20} \d x \: \smallfrac{1}{\l} \,
e^{-x/\lambda } = \left(e^{-1/\l} - e^{-20 /\l} \right).
\label{basic.likelihood.Z}
\eeq
This seemed obvious enough.
Then he wrote {\dem{\ind{\Bayes\ theorem}}}:
\beqan
\label{bayes.theorem}
% \begin{array}{l}
P(\l\given \{x_1, \ldots, x_N\}) &=&
\frac{P(\{x\}\given \lambda) P(\l)}{P(\{x\}) } \\
%&& \hspace{0.5in}
&\propto& \frac{1}{\left( \l Z(\l) \right)^N}
\exp \left( \textstyle - \sum_1^N x_n / \l \right) P(\l)
.
% \end{array}
\label{basic.posterior}
\eeqan
Suddenly, the straightforward distribution $P(\{x_1 ,\ldots, x_N \}\given
\l)$, defining the probability of the data given the hypothesis $\l$,
was being turned on its head so as to define the probability of a
hypothesis given the data. A simple figure showed the probability of
a single data point $P(x\given \l)$ as a familiar function of $x$, for
different values of $\l$ (figure \ref{decay.like.1}). Each curve was
an innocent exponential, normalized to have area 1. Plotting the
same function as a function of $\l$ for a fixed value of $x$,
something remarkable happens: a peak emerges (figure
\ref{decay.like.2}). To help understand these two points
of view of the one function, \figref{decay.probandlike}
shows a surface plot of $P(x\given \l)$ as a function of $x$ and $\l$.
\begin{figure}
\figuremargin{%
\begin{center}
\mbox{\psfig{figure=\FIGS/decay.like.1.ps,%
width=2 in,angle=-90}\ \ \ \raisebox{-3mm}[0in][0in]{$x$}}
\end{center}
}{%
\caption{{The probability density $P(x\given \l)$ as a function of $x$.}}
\label{decay.like.1}
}%
\end{figure}
\begin{figure}
\figuremargin{%
\begin{center}
\mbox{\psfig{figure=\FIGS/decay.like.2.ps,%
width=2 in,angle=-90}\ \ \ \raisebox{-3mm}[0in][0in]{$\lambda$}}
\end{center}
}{%
\caption[a]{{The probability density $P(x\given \l)$ as a function of $\l$,
for three different values of $x$.}
\small
When plotted this way round, the function is known as
the {\dem\ind{likelihood}\/} of $\l$.
The marks indicate the three values of $\l$, $\l=2,5,10$,
that were used in the preceding figure.
}
\label{decay.like.2}
}
\end{figure}
%\begin{figure}
%\figuremargin{%
\marginfig{
\begin{center}
\begin{tabular}{c}
\makebox[0pt][l]{\hspace*{0.21in}\raisebox{0.435in}{$x$}}%
\mbox{\psfig{figure=\FIGS/probandlike.ps,%
width=2in,angle=-90}%
\makebox[0pt][l]{\hspace*{-0.352in}\raisebox{0.435in}{$\l$}}}\\[-0.3in]% was -0.6 Sat 5/10/02
\end{tabular}\end{center}
%}{%
\caption[a]{{The probability density $P(x\given \l)$ as a function of $x$
and $\l$. Figures \ref{decay.like.1} and \ref{decay.like.2} are
vertical sections through this surface.}
}
\label{decay.probandlike}
}
%\end{figure}
\begin{figure}
\figuremargin{%
\begin{center}
\mbox{\psfig{figure=\FIGS/decay.like.xxx.ps,%
width=2in,angle=-90}}
\end{center}
}{%
\caption[a]{{The likelihood function in the case of a six-point dataset,
$P(\{x\} = \{1.5,2,3,4,5,12\}\given \lambda)$, as a function of $\l$.}
}
\label{decay.like.xxx}
}
\end{figure}
For a dataset consisting of several points, \eg, the
six points
$\{x\}_{n=1}^{N} = \{1.5,2,3,4,5,12\}$, the likelihood function
$P(\{x\}\given \lambda)$ is the product of the $N$ functions of $\l$,
$P(x_n\given \l)$ (\figref{decay.like.xxx}).
%
Steve summarized \Bayes\ theorem
% (equation \ref{bayes.theorem})
as
embodying the fact that
\begin{conclusionbox}
what you know about $\lambda$
after the data arrive is what
you knew before [$P(\lambda)$], and what the data told you
[$P(\{x\}\given \lambda)$].
\end{conclusionbox}
Probabilities are used here to
quantify degrees of belief.
% The probability
% of $\lambda$ is a quantification of what you know about $\lambda$.
To nip possible confusion in the bud, it must be
emphasized that the hypothesis $\lambda$ that correctly describes
the situation is {\em not\/} a {\em stochastic\/} variable, and the fact that
the Bayesian uses a probability\index{probability!Bayesian}
distribution $P$ does {\em not\/} mean
that he thinks of the world as stochastically changing its nature
between the states described by the different hypotheses. He uses the
notation of probabilities to represent his {\em beliefs\/} about the mutually
exclusive micro-hypotheses (here, values of $\l$),
of which only one is actually true. That
probabilities can denote degrees of belief, given assumptions, seemed
reasonable to me.
% , and is proved by Cox (1946).
% \citeasnoun{cox}.
% . Anyone who does not find it reasonable to use
% probabilities to quantify degrees of belief can read
% paper, where it is proved to be
% valid.
\label{sec.decayb}
The posterior probability distribution
% of equation
(\ref{basic.posterior}) represents
the unique and complete solution to the problem.
There is no need to invent\index{classical statistics!criticisms}
`estimators'; nor do we need to invent
criteria for comparing alternative estimators with each other.
Whereas orthodox statisticians offer twenty ways of solving a
problem, and another twenty different criteria for deciding which of
these solutions is the best, Bayesian statistics only offers one
answer to a well-posed problem.
% Added Mon 4/2/02
\marginpar{\small\raggedright{If you have any difficulty understanding this chapter I recommend
ensuring you are happy with
exercises \ref{ex.dieexponential} and \ref{ex.dieexponentialb} (\pref{ex.dieexponentialb})
then noting their similarity to
\exerciseonlyref{ex.exponential}.}}
\subsection{Assumptions in inference}
Our inference is conditional on our assumptions [for example, the
prior $P(\lambda)$]. Critics view such priors as a difficulty because
they are `subjective', but I
don't see how it could be otherwise. How can one perform inference
without making assumptions?
I believe that it is of great value that Bayesian
methods force one to make these tacit assumptions explicit.
First,
once assumptions are made, the inferences are objective and unique,
reproducible with complete agreement by anyone who has the same
information and makes the same assumptions. For example, given the
assumptions listed above, $\H$, and the data $D$,
% from an experiment
% measuring decay lengths,
everyone will agree about the posterior
probability of the decay length $\l$:
\beq
P(\l\given D,\H) = \frac{ P(D\given \l,\H) P(\l\given \H) }{ P(D\given \H) } .
\eeq
Second, when the assumptions are explicit, they are easier to
criticize, and easier to modify -- indeed,
we can quantify the sensitivity of our inferences to
the details of the assumptions. For example,
we can note from the likelihood curves
in figure \ref{decay.like.2} that in the case of a single data point at
$x=5$, the likelihood
function is less strongly peaked than in the case $x=3$; the
details of the prior $P(\lambda)$ become increasingly important as the sample
mean $\bar{x}$ gets closer to the middle of the window, 10.5. In the case
$x=12$, the likelihood function doesn't have a peak at all -- such data
merely rule out small values of $\lambda$, and don't give any information
about the relative probabilities of large values of $\lambda$. So
in this case, the details of the prior at the small--$\lambda$ end
of things are not important, but at the large--$\lambda$ end, the prior
is important.
% is whatever we knew before
% the experiment, \ie, our prior.
Third, when we are not sure which of various alternative assumptions
is the most appropriate for a problem, we can treat this question as
another inference task. Thus, given data $D$, we can\index{Bayes' theorem}
% learn from the data
compare alternative assumptions $\H$ using \Bayes\ theorem:
\beq
P(\H\given D,\I) = \frac{ P(D\given \H,\I) P(\H\given \I) }{ P(D\given \I) } ,
\label{basic.ev}
\eeq
where $\I$ denotes the highest assumptions, which we are not
questioning.
Fourth, we can take into account our uncertainty regarding such
assumptions when we make subsequent predictions. Rather than choosing
one particular assumption $\H^{*}$, and working out our predictions
about some quantity $\bt$, $P(\bt\given D,\H^{*},\I)$, we obtain
predictions that take into account our uncertainty about $\H$ by
using the sum rule:
\beq
P(\bt \given D, \I) = \sum_{\H} P(\bt \given D, \H , \I ) P(\H\given D,\I) .
\label{basic.marg}
\eeq
This is another contrast with orthodox statistics, in which it is
conventional to `test' a default model, and then, if the test\index{test!statistical}\index{statistical test}
`accepts the model' at some `\ind{significance level}', to use exclusively that model to make
predictions.
Steve thus persuaded me that
\begin{conclusionbox}
probability theory reaches parts that ad hoc methods cannot reach.
\end{conclusionbox}
% However, that is a topic for another lecture.
Let's look at a few more examples of simple inference problems.
\section{The bent coin}
\label{sec.bentcoin}
A \ind{bent coin}\index{inference problems!bent coin}
is tossed $F$ times; we observe a sequence $\bs$ of
heads and tails (which we'll denote by the symbols $\ta$ and $\tb$).
We wish to know the bias of the coin, and predict
the probability that the next toss will result in a head.
We first encountered this task in \exampleref{exa.bentcoin},
and we will encounter it again
in \chref{ch.four}, when we discuss adaptive data compression.
% the adaptive encoder for $a$s and $b$s.
It is also the original inference problem studied by
% Rev.\
{Thomas Bayes}
in his essay published in 1763.\index{Bayes, Rev.\ Thomas}
% cite{Bayes}
As in
% \chref{ch.prob.ent}
\exerciseref{ex.postpa}, we will
assume
% In chapter \chfour\ we assumed
a uniform prior distribution and
obtain a posterior distribution by multiplying by the likelihood. A
critic might object, `where did this prior come from?' I will not
claim that the uniform prior is in any way fundamental; indeed
we'll give examples of nonuniform priors later. The prior is
% It is simply
a subjective assumption. One of the themes of this book is:
%
% put this back somewhere?
%
% One way to justify the need for a prior is
% to assume, as in chapter \chfour,
% that our task is simply to make a code to encode the
% outcome $\bs$ as efficiently as possible. We have to compress the
% data from the source somehow, and any choice of a compression scheme
% must correspond to a prior distribution over coin biases. I see no
% way round this. The choice of code implies an assumed probability
% distribution over outcomes.
%\begin{quotation}
\begin{conclusionbox}
\noindent
you can't do inference -- or data compression -- without
making assumptions.
% You can't do data compression -- or inference -- without
% making assumptions.
\end{conclusionbox}
%\end{quotation}
%
% change notation? f_H?????????????????????????????????
%
%\subsubsection*{Likelihood function}
We give the name $\H_1$ to our assumptions. [We'll be introducing
an alternative set of assumptions in a moment.]
The probability, given $p_{\ta}$, that $F$ tosses
result in a sequence $\bs$
that contains $\{F_{\ta},F_{\tb}\}$ counts of the two outcomes
% $\{ a , b \}$
is
\beq
P( \bs \given p_{\ta} , F,\H_1 ) = p_{\ta}^{F_{\ta}} (1-p_{\ta})^{F_{\tb}} .
\label{eq.pa.likeb}
\eeq
[{For example, $P(\bs\eq {\tt{aaba}} \given p_{\ta},F \eq 4,\H_1)
= p_{\ta}p_{\ta}(1-p_{\ta})p_{\ta}.$}]
% This function of $p_{\ta}$ (\ref{eq.pa.likeb}) defines the likelihood function.
% Model 1
Our first model assumes a uniform prior distribution for $p_{\ta}$,
\beq
P(p_{\ta}\given\H_1) = 1 , \: \: \: \: \: \: p_{\ta} \in [0,1]
\label{eq.pa.priorb}
\eeq
and $p_{\tb} \equiv 1-p_{\ta}$.
\subsubsection{Inferring unknown parameters}
Given a string of length $F$ of which $F_{\ta}$ are $\ta$s and
$F_{\tb}$ are $\tb$s, we are interested in (a) inferring
what $p_{\ta}$ might be; (b) predicting whether the next character is an $\ta$
or a $\tb$. [Predictions\index{prediction} are always expressed as probabilities.
So `predicting whether the next character is an $\ta$'
is the same as computing the probability that the next character is an $\ta$.]
Assuming $\H_1$ to be true, the posterior probability of $p_{\ta}$, given a
string $\bs$ of length $F$ that has
counts $\{F_{\ta},F_{\tb}\}$, is, by \Bayes\ theorem,
\beqan
P( p_{\ta} \given \bs ,F,\H_1) &=&
\frac{ P( \bs \given p_{\ta} , F,\H_1 ) P(p_{\ta}\given\H_1) }{ P( \bs \given F,\H_1 ) } .
\label{eq.pa.post}
\label{eq.pa.post.again}
\eeqan
The factor $P( \bs \given p_{\ta} , F,\H_1 )$, which, as a function
of $p_{\ta}$, is known as the likelihood function,
was given in \eqref{eq.pa.likeb}; the prior
$P(p_{\ta}\given\H_1)$ was given in \eqref{eq.pa.priorb}.
Our inference of $p_{\ta}$ is thus:
% The posterior
\beqan
P( p_{\ta} \given \bs ,F,\H_1) &=&
\frac{ p_{\ta}^{F_{\ta}} (1-p_{\ta})^{F_{\tb}} }{ P( \bs \given F,\H_1 ) } .
\label{eq.pa.postb.again}
\eeqan
The normalizing constant is given by the beta integral
\beq
P( \bs \given F,\H_1 ) = \int_0^1 \d p_{\ta} \: p_{\ta}^{F_{\ta}} (1-p_{\ta})^{F_{\tb}} =
\frac{\Gamma(F_{\ta}+1)\Gamma(F_{\tb}+1)}{ \Gamma(F_{\ta}+F_{\tb}+2) }
= \frac{ F_{\ta}! F_{\tb}! }{ (F_{\ta} + F_{\tb} + 1)! } .
\label{eq.evidenceZ}
\eeq
% Our inference of $p_{\ta}$, assuming $\H_1$ to be true,
% is thus given by \eqref{eq.pa.postb.again}.
%%%%%%%%%%%%%
\exercissxA{2}{ex.postpaII}{
Sketch the posterior probability $P( p_{\ta} \given \bs\eq {\tt aba} ,F\eq 3)$.
What is the most probable value of $p_{\ta}$ (\ie, the value that maximizes
the posterior probability density)? What is the mean value of $p_{\ta}$
under this distribution?
Answer the same questions for
the posterior probability $P( p_{\ta} \given \bs\eq {\tt bbb} ,F\eq 3)$.
}
\subsubsection{From inferences to predictions}
Our prediction about the next toss, the probability that the next toss is an $\ta$,
is obtained by integrating over $p_{\ta}$. This has the effect of
taking into account our uncertainty about $p_{\ta}$ when making predictions.
By the sum rule,
\beqan
P(\ta \given \bs ,F)& =& \int \d p_{\ta} \: P(\ta \given p_{\ta} ) P(p_{\ta} \given \bs,F ) .
\eeqan
The probability of an $\ta$ given $p_{\ta}$ is simply $p_{\ta}$,
so
\beqan
\lefteqn{ P(\ta \given \bs ,F)
= \int \d p_{\ta} \: p_{\ta} \frac{p_{\ta}^{F_{\ta}} (1-p_{\ta})^{F_{\tb}}}
{P( \bs \given F ) } }
\\
&=& \int \d p_{\ta} \: \frac{p_{\ta}^{F_{\ta}+1} (1-p_{\ta})^{F_{\tb}}}
{P( \bs \given F ) }
\\
&=& \left.
% \frac
{ \left[ \frac{ (F_{\ta}+1)! F_{\tb}! }{ (F_{\ta} + F_{\tb} + 2)! } \right] } \right/
{ \left[ \frac{ F_{\ta}! F_{\tb}! }{ (F_{\ta} + F_{\tb} + 1)! } \right] }
\:\: = \:\: \frac{ F_{\ta}+1 }{ F_{\ta} + F_{\tb} + 2 } ,
\label{eq.laplacederived}
\eeqan
which is known as {\dem{\ind{Laplace's rule}}}.
\section{The bent coin and model comparison}
\label{sec.bentcoin2}
Imagine that a scientist introduces another theory for our data.
He asserts that the source is not really a bent coin but is really a
perfectly formed die with one face painted heads (`$\ta$') and the other five
painted tails (`$\tb$'). Thus the parameter $p_{\ta}$, which in the original model,
$\H_1$, could take any value between 0 and 1, is according
to the new hypothesis, $\H_0$, not a free parameter at all; rather, it
is equal to
% p_{\ta} =
$1/6$. [This hypothesis is termed $\H_0$ so that the suffix of each model
indicates its number of free parameters.]
How can we compare these two models in the light of data?
We wish to
infer how probable
$\H_1$ is relative to $\H_0$.
% , so we can use \Bayes\ theorem again.
% Let us write down the first model's probabilities again.
% {\em Here we repeat some material from the arithmetic coding
% chapter, chapter \ref{ch4}.}
\subsubsection*{Model comparison as inference}
In order to perform model comparison, we write down
\Bayes\ theorem again, but this time with a different\index{Bayes' theorem}
argument on the left-hand side. We wish to know how probable
$\H_1$ is given the data. By \Bayes\ theorem,
\beq
P( \H_1 \given \bs ,F ) = \frac{ P( \bs \given F,\H_1 ) P( \H_1 ) }{ P( \bs \given F) } .
\eeq
Similarly, the posterior probability of $\H_0$ is
\beq
P( \H_0 \given \bs ,F ) = \frac{ P( \bs \given F,\H_0 ) P( \H_0 ) }{ P( \bs \given F) }.
\eeq
The normalizing constant in both cases is $P(\bs\given F)$, which is the total
probability of getting the observed data.
% regardless of which model is true.
If $\H_1$ and $\H_0$ are the only models under
consideration, this probability is given by the sum rule:
\beq
P( \bs \given F) = P( \bs \given F,\H_1 ) P( \H_1 )
+ P( \bs \given F,\H_0 ) P( \H_0 ) .
\eeq
To evaluate the posterior probabilities of the hypotheses we
need to assign values to the prior probabilities $P( \H_1 )$
and $P( \H_0 )$; in this case, we might set these to 1/2 each. And
we need to evaluate the data-dependent terms
$P( \bs \given F,\H_1 )$ and $P( \bs \given F,\H_0 )$.
We can give names to these quantities.
The quantity $P( \bs \given F,\H_1 )$ is a measure of how much the data
favour $\H_1$, and we call it the {\dbf\ind{evidence}} for model $\H_1$.
We already encountered this quantity in equation (\ref{eq.pa.post.again})
where it appeared
as the normalizing constant of the first inference we made -- the
inference of $p_{\ta}$ given the data.
\medskip
\begin{conclusionbox}
%\begin{description}
%\item[How model comparison works:]
{\bf How model comparison works:}
The evidence for a model is usually\index{key points!model comparison}
the normalizing constant of an earlier Bayesian inference.
%\end{description}
\end{conclusionbox}
\medskip
We evaluated the normalizing constant for model $\H_1$ in
(\ref{eq.evidenceZ}).
The evidence for model $\H_0$ is very simple because this model
has no parameters to infer. Defining $p_0$ to be $1/6$, we have
\beq
P( \bs \given F,\H_0 ) = p_0^{F_{\ta}} (1-p_0)^{F_{\tb}} .
\eeq
Thus the posterior probability ratio of model $\H_1$ to model $\H_0$ is
\beqan
\frac{ P( \H_1 \given \bs ,F )}
{P( \H_0 \given \bs ,F )}
& =&
\frac{ P( \bs \given F,\H_1 ) P( \H_1 ) }
{ P( \bs \given F,\H_0 ) P( \H_0 ) }
\\
&=&
\left.
{ \frac{ F_{\ta}! F_{\tb}! }{ (F_{\ta} + F_{\tb} + 1)! } }
\right/
{ p_0^{F_{\ta}} (1-p_0)^{F_{\tb}} } .
% \frac{ \smallfrac{ F_{\ta}! F_{\tb}! }{ (F_{\ta} + F_{\tb} + 1)! } }{ p_0^{F_{\ta}} (1-p_0)^{F_{\tb}} } .
% SECOND EDN - sanjoy says use linefrac
\label{eq.compare.final}
\eeqan
Some values of this posterior probability ratio are illustrated in
table \ref{tab.mod.comp}. The first five lines illustrate that
some outcomes favour one model, and some favour the other.
No outcome is completely incompatible with either model.
\begin{table}
\figuremargin{%
\begin{center}
\begin{tabular}{cccl} \toprule
$F$ & Data $(F_{\ta},F_{\tb})$ & $\displaystyle \frac{ P( \H_1 \given \bs ,F )}
{P( \H_0 \given \bs ,F )}$ \\ \midrule
6 & $(5,1)$ & 222.2 & \\
6 & $(3,3)$ & 2.67 &\\
6 & $(2,4)$ & 0.71 & = 1/1.4 \\
6 & $(1,5)$ & 0.356 & = 1/2.8 \\
6 & $(0,6)$ & 0.427 & = 1/2.3 \\ \midrule
20 & $(10,10)$ & 96.5 & \\
20 & $(3,17)$ & 0.2 & = 1/5 \\
20 & $(0,20)$ & 1.83 & \\ \bottomrule
\end{tabular}
\end{center}
}{%
\caption{Outcome of model comparison between models $\H_1$ and $\H_0$
for the `bent coin'. Model $\H_0$ states that $p_{\ta}=1/6$, $p_{\tb}=5/6$.}
\label{tab.mod.comp}
}
\end{table}
With small amounts of data (six tosses, say) it is typically not the case that
one of the two models is overwhelmingly more probable than
the other. But with more data, the evidence against $\H_0$ given
by any data set with the ratio $F_{\ta} \colon F_{\tb}$ differing from $1 \colon 5$ mounts up.
%
% add figure showing some typical histories
%
You can't predict in advance how much data are needed to be pretty sure
which theory is true.\index{key points!how much data needed} It depends what $p_0$ is.
%
% THIS IS A VERY GENERAL
% message for machine learning.
% corrected Wed 28/11/01
The simpler model, $\H_0$, since it has no adjustable parameters,
is able to lose out by the biggest margin. The odds may be hundreds to one
against it. The more complex model can never lose out
by a large margin; there's no data set that is actually {\em unlikely\/}
given model $\H_1$.
\exercisaxB{2}{ex.evidencebounds}{
Show that after $F$ tosses have taken place, the
biggest value that the log evidence ratio
\beq
\log \frac{ P( \bs \given F,\H_1 ) }
{ P( \bs \given F,\H_0 ) }
\eeq
can have scales {\em linearly\/} with $F$ if
$\H_1$ is more probable, but
the log evidence in favour of $\H_0$ can grow
at most as $\log F$.
}
\exercissxB{3}{ex.evidenceest}{
Putting your sampling theory hat on, assuming $F_{\ta}$ has not yet been measured,
compute a plausible range that
% the mean and variance -- or some sort of most probable value, and indication of spread -- of the
the log evidence ratio might lie in, as a function of $F$ and
the true value of $p_{\ta}$,
and sketch it
as a function of $F$ for $p_{\ta}=p_0=1/6$, $p_{\ta}=0.25$,
and $p_{\ta}=1/2$.
[Hint: sketch the log evidence as a function
of the random variable $F_{\ta}$ and work out the mean
and standard deviation of $F_{\ta}$.]
% [Hint: Taylor-expand the log evidence as a function
% of $F_{\ta}$.]
}
%
% This page comes out rotated bizarrely by 90 degrees in pdf
%
\subsection{Typical behaviour of the evidence}
% see figs/sixtoone
% and bin/sixtoone.p
\Figref{fig.evidencetyp} shows the log evidence ratio
as a function of the number of
tosses, $F$, in a number of simulated experiments.
In the left-hand experiments, $\H_0$ was true.
In the right-hand ones, $\H_1$ was true, and the value of
$p_{\ta}$ was either 0.25 or 0.5.
% \newcommand{\sixtoone}[2]{% in newcommands1.tex
\begin{figure}
\figuremargin{%
\small%
\begin{center}
\begin{tabular}{cccc}
$\H_0$ is true &&
\multicolumn{2}{c}{$\H_1$ is true} \\ \cmidrule{1-1}\cmidrule{3-4}
\sixtoone{$p_{\ta}=1/6$}{h09}&&
\sixtoone{$p_{\ta}=0.25$}{h69}&
\sixtoone{$p_{\ta}=0.5$}{h29}\\
\sixtoone{}{h08}&&
\sixtoone{}{h68}&
\sixtoone{}{h28}\\
\sixtoone{}{h07}&&
\sixtoone{}{h67}&
\sixtoone{}{h27}\\
\end{tabular}
\end{center}
}{%
\caption[a]{Typical behaviour of the evidence in favour of $\H_1$ as
bent coin tosses accumulate\index{typicality!behaviour of evidence}\index{evidence!typical behaviour of}\index{model comparison!typical evidence}
under three different conditions. Horizontal axis is the number of
tosses, $F$. The vertical axis on the left is
$\ln \smallfrac{ P( \bs \given F,\H_1 ) }
{ P( \bs \given F,\H_0 ) }$;
the right-hand vertical axis shows the values of
$\smallfrac{ P( \bs \given F,\H_1 ) }
{ P( \bs \given F,\H_0 ) }$.
(See also \protect\figref{fig.evidenceMSD}, \pref{fig.evidenceMSD}.)
}
\label{fig.evidencetyp}
}%
\end{figure}
We will discuss model comparison more in a later chapter.
\section{An example of legal evidence}
\label{ex.blood.sol}% special label by hand
The following example
% (\exerciseonlyref{ex.blood})
illustrates that there is more
to Bayesian inference than the priors.
\begin{quote}
% Two people have left traces of their own blood at the scene of a
% crime. Their blood groups can be reliably identified from these
% traces and are found
% to be of type `O' (a common type in the local population, having
% frequency 60\%) and of type `AB' (a rare type, with frequency 1\%).
% A suspect is tested and found to have type `O' blood.
% A careless lawyer might claim that the fact that the suspect's
% blood type was found at the scene is positive evidence for the theory
% that he was present. But do these data
% $D=$ \{type `O' and `AB' blood were found at scene\} make it more
% probable that this suspect was one of the two people present at the
% crime?
Two people have left traces of their own blood at the scene of a
crime.
A suspect, Oliver, is tested and found to have type `O' blood.
The blood groups of the two traces
are found
to be of type `O' (a common type in the local population, having
frequency 60\%) and of type `AB' (a rare type, with frequency 1\%).
Do these data
(type `O' and `AB' blood were found at scene) give evidence in favour
of the proposition that Oliver was one of the two people present at the
crime?
\end{quote}
A careless \ind{lawyer} might claim that the fact that the suspect's
blood type was found at the scene is positive evidence for the theory
that he was present. But this is not so.
Denote the proposition `the suspect and one unknown person were
present' by $S$. The alternative, $\bar{S}$, states `two unknown people
from the population were present'.
The prior in this problem is the prior probability ratio between the
propositions $S$ and $\bar{S}$. This quantity is important to the final
verdict and would be based on all other available information
in the case. Our task here is just to evaluate the contribution made by the
data $D$, that is, the likelihood ratio, $P(D\given S,\H)/P(D\given \bar{S},\H)$.
In my view, a jury's task should generally be to multiply together carefully
evaluated
likelihood ratios from each independent piece of admissible evidence
with an equally carefully reasoned prior probability.
[This view is shared by many statisticians but learned British appeal judges\index{judge}
recently disagreed and actually overturned the verdict of a trial
because the \index{jury}{jurors} {\em had\/} been taught to use \Bayes\ theorem to
handle complicated \ind{DNA} evidence.]
%
The probability of the data given $S$ is the probability that one unknown person
drawn from the population has blood type AB:
\beq
P(D\given S,\H) = p_{\rm{AB}}
\eeq
(since given $S$, we already know that one trace will be of type O).
The probability of the data given $\bar{S}$ is the
probability that two unknown people drawn from the population have
types O and AB:
\beq
P(D\given \bar{S},\H) = 2 \, p_{\rm{O}} \, p_{\rm{AB}} .
\eeq
In these equations $\H$ denotes the assumptions that two people were
present and left blood there, and that the probability distribution
of the blood groups of unknown people in an explanation is the same
as the population frequencies.
% Our posterior probability ratio for
% $S$ relative to $\bar{S}$ is obtained by multiplying the probability
% ratio based on all other independent information by the ratio of
% these likelihoods. The most straightforward way to summarize the
% contribution of any piece of evidence is in terms of a likelihood
% ratio.
Dividing, we obtain the likelihood ratio:
\beq
\frac{P(D\given S,\H)}{P(D\given \bar{S},\H)} = \frac{1}{2 p_{\rm O}}
= \frac{1}{2 \times 0.6}
= 0.83 .
\eeq
Thus the data in fact provide weak evidence {\em against\/} the
supposition that Oliver was present.
This result may be found surprising, so let us examine it from
various points of view. First consider the case of another suspect,
Alberto,
who has type AB. Intuitively, the data do provide evidence in favour
of the theory $S'$ that this suspect was present, relative to the
null hypothesis $\bar{S}$. And indeed the likelihood ratio in this
case is:
\beq
\frac{P(D\given S',\H)}{P(D\given \bar{S},\H)} = \frac{1}{2\, p_{\rm{AB}}} = 50.
\eeq
Now let us change the situation slightly; imagine that 99\% of people
are of blood type O, and the rest are of type AB. Only these two
blood types exist in the population. The data at the
scene are the same as before. Consider again how these data influence
our beliefs about Oliver,
a suspect of type O, and Alberto, a suspect of type
AB. Intuitively, we still believe that the presence of the rare AB
blood provides positive evidence that \ind{Alberto} was
there. But does
% we still have the feeling that
the fact that type O
blood was detected at the scene favour the hypothesis that
Oliver was present? If this were the case, that would mean that
regardless of who the suspect is, the data make it more probable they
were present; everyone in the population would be
under greater suspicion, which would be absurd. The data may be {\em
compatible\/} with any suspect of either blood type being present, but
if they provide evidence {\em for\/} some theories, they must also
provide evidence {\em against\/} other theories.
Here is another way of thinking about this: imagine that instead of
two people's blood stains there are ten, and that in the entire local
population of one hundred, there are ninety type O suspects and ten
type AB suspects.
% Initially all 100 people are suspects.
Consider a particular type O suspect, \ind{Oliver}: without any other information,
and before the blood test results come in,
there is a one in 10 chance that he was at the scene, since
we know that 10 out of the 100 suspects were present. We now get the
results of blood tests, and find that {\em nine\/} of the ten stains are of
type AB, and {\em one\/} of the stains is of type O. Does this make it more
likely that Oliver was there? No,
% although he could have been,
there is now only a one in ninety chance that he was there, since we
know that only one person present was of type O.
Maybe the intuition is aided finally by writing down the formulae for
the general case where $n_{\rm{O}}$ blood stains of individuals of type O
are found, and $n_{\rm{AB}}$ of type $\rm{AB}$, a total of $N$ individuals in
all, and unknown people come from a large population with fractions
$p_{\rm{O}}, p_{\rm{AB}}$. (There may be other blood types too.)
The task is to evaluate the likelihood ratio for the
two hypotheses: $S$, `the type O suspect (Oliver)
and $N\!-\!1$ unknown others
left $N$ stains'; and $\bar{S}$, `$N$ unknowns left $N$ stains'. The
probability of the data under hypothesis $\bar{S}$ is just the
probability of getting $n_{\rm{O}}, n_{\rm{AB}}$ individuals of the two types
when $N$ individuals are drawn at random from the population:
\beq
P(n_{\rm{O}},n_{\rm{AB}}\given \bar{S}) =
\frac{ N! }{ n_{\rm{O}} ! \, n_{\rm{AB}}! } p_{\rm{O}}^{n_{\rm{O}}} p_{\rm{AB}}^{n_{\rm{AB}}} .
\eeq
In the case of hypothesis $S$, we need the distribution of
the $N\!-\!1$ other individuals:
\beq
P(n_{\rm{O}},n_{\rm{AB}}\given S) =
\frac{ (N-1)! }{ (n_{\rm{O}}-1)! \, n_{\rm{AB}}! } p_{\rm{O}}^{n_{\rm{O}}-1} p_{\rm{AB}}^{n_{\rm{AB}}} .
\eeq
The likelihood ratio is:
\beq
\frac{ P(n_{\rm{O}},n_{\rm{AB}}\given S) }{ P(n_{\rm{O}},n_{\rm{AB}}\given \bar{S}) }
= \frac{n_{\rm{O}}/N}{p_{\rm{O}}} .
\eeq
This is an instructive result. The likelihood ratio, \ie\ the
contribution of these data to the question of whether Oliver
was present, depends simply on a comparison of the frequency
of his blood type
% type O blood
in the observed data with the background frequency
% of type O blood
in the population. There is no dependence on the counts
of the other types found at the scene, or their frequencies in the
population. If there are more type O stains than the average number
expected under hypothesis $\bar{S}$, then the data give
evidence in favour of the presence of Oliver.
Conversely, if there are fewer type O stains than the expected number
under $\bar{S}$, then the data reduce the probability of the
hypothesis that he was there. In the special case $n_{\rm{O}}/N = p_{\rm{O}}$, the
data contribute no evidence either way, regardless of the fact that
the data are compatible with the hypothesis $S$.
\section{Exercises}
% \subsection*{The game show}
%\subsubsection*{The normal rules}
%\subsubsection*{The earthquake scenario}
\exercissxA{2}{ex.3doors}{
{\sf The \ind{three doors},\index{Monty Hall problem} normal rules.}
% "Let's Make A Deal," hosted by Monty Hall
On a \ind{game show},\index{doors, on game show}\index{game!three doors}
a contestant is told the rules as
follows:
\begin{quote}
There are three doors, labelled 1, 2, 3. A single
prize has been hidden behind one of
them. You get to select one door. Initially your chosen door will {\em not\/}
be opened. Instead, the gameshow host will open one of the other two doors,
and {\em he will do so in such a way as not to reveal the prize.}
For example, if you first
choose door 1, he will then open {one\/} of doors 2 and 3, and it
is guaranteed that he will choose which one to open so that
the prize will not be revealed.
At this point, you will be given a fresh choice of door:
you can either stick with your first choice,
or you can switch to the other
closed door. All the doors will then be opened and
you will receive whatever is behind your final
choice of door.
\end{quote}
Imagine that the contestant chooses door 1 first; then the gameshow host
opens door 3, revealing nothing behind the door, as promised.
Should the contestant (a) stick with door 1, or (b)
switch to door 2, or (c) does it make no difference?
}
\exercissxA{2}{ex.3doorsb}{
{\sf The three doors, earthquake scenario.}
Imagine that the game happens again
and just as the gameshow host is about to open one of the
doors a violent earthquake\index{earthquake, during game show}
rattles the building and one of the
three doors flies open. It happens to be door 3, and it
happens not to have the prize behind it. The contestant had initially
chosen door 1.
Repositioning his toup\'ee,
the host suggests, `OK, since you chose door 1 initially,
door 3 is a valid door for me to open, according to the
rules of the game; I'll let door 3 stay open. Let's carry on
as if nothing happened.'
Should the contestant stick with door 1, or switch to door 2, or
does it make no difference? Assume that the prize was placed randomly, that
the gameshow host does not know where it is, and that the door flew open
because its latch was broken by the earthquake.
[A similar alternative scenario is a gameshow whose {\em confused host\/}\index{confused gameshow host}
forgets the rules, and where the prize is, and opens one of
the unchosen doors at random. He opens door 3, and the prize is not revealed.
Should the contestant choose what's behind door 1 or door 2?
Does the optimal decision for
the contestant depend on the contestant's \ind{belief}s about
whether the gameshow host is confused or not?]\index{game show}\index{three doors}\index{doors, on game show}\index{prize, on game show}\index{Monty Hall problem}
}
\exercisaxB{2}{ex.girlboy}{
%\subsection
{\sf Another example in which the emphasis is not on priors.}
%\begin{quote}
You visit a family whose three children are all at the local school.
You don't know anything about the sexes of the children.
While walking clumsily round the home, you stumble through
one of the three unlabelled bedroom doors that you know
belong, one each, to the three children, and find that the bedroom
contains \ind{girlie stuff} in sufficient quantities to
convince you that the child who lives in that bedroom
is a girl.
Later, you sneak a look at a letter addressed to the parents,
which reads `From the Headmaster:
we are sending this letter to all parents who have male children at
the school to inform them about the following \ind{boyish matters}\ldots'.
These two sources of evidence establish that at least
one of the three children is
a girl, and that at least one of the children is a boy.
What are the probabilities that there are (a) two girls and one boy;
(b) two boys and one girl?
%\end{quote}
}
% Another example of legal evidence}
\exercissxB{2}{ex.simpsons}{
Mrs\ S is found stabbed in her family
garden.
% \index{Simpson, O.J., similar case to}
Mr\ S behaves strangely after her death and is considered as
a suspect. On investigation of police and social records
it is found that Mr\ S had beaten up his wife on at least
nine previous occasions. The prosecution advances this
data as evidence in favour of the hypothesis that Mr\ S is
guilty of the murder.
`Ah no,' says
% Mr.\ Merd-Kopf,
Mr\ S's highly paid lawyer,\index{lawyer}\index{wife-beater}\index{murder}
`{\em statistically}, only one in a thousand wife-beaters
actually goes on to murder his wife.\footnote{In the U.S.A., it
is estimated that
% http://www.umn.edu/mincava/papers/factoid.htm
2 million women are abused each year by their partners.
In 1994, $4739$ women were victims of homicide; of those,
% 28 \percent,
$1326$ women (28\%)
were slain by husbands and boyfriends.\\ (Sources:
{\tt http://www.umn.edu/mincava/papers/factoid.htm,\\
http://www.gunfree.inter.net/vpc/womenfs.htm})
% http://www.gunfree.inter.net/vpc/womenfs.htm
% In keeping
% with the fictitious nature of this story, the $1/100\,000$
% figure was made up by me.
}\label{footnote.murder} So the wife-beating
% , which is not denied by Mr\ S,
is not strong evidence at all. In fact,
given the wife-beating evidence alone, it's extremely unlikely
that he would be the murderer of his wife -- only a
$1/1000$ chance. You should therefore find him innocent.'
Is the lawyer
% Mr\ Merd-Kopf
right to imply that the history of wife-beating does
not point to Mr\ S's being the murderer? Or is the lawyer a slimy trickster? If
the latter, what is wrong with his argument?
[Having received an indignant letter from a lawyer about
the preceding paragraph, I'd like to
add an extra inference exercise at this point:
{\em Does my suggestion that Mr.\ S.'s lawyer
may have been a slimy trickster imply that
I believe {\em all} lawyers are slimy tricksters?} (Answer: No.)]
}
% Lewis Carroll's Pillow Problem
\exercisaxB{2}{ex.bagcounter}{ A bag contains one counter, known to be
either white or black. A white counter is put in, the bag is shaken,
and a counter is drawn out, which proves to be white. What is now the
chance of drawing a white counter?
[Notice that
the state of the bag, after the operations, is exactly identical to its state before.]
}
\exercissxB{2}{ex.phonetest}{% ????????????????? needs solution adding (was phonecheck!)
You move into a new house; the phone is connected, and
% you are unsure of your phone number --
you're pretty sure that
the \ind{phone number}\index{telephone number} is
% it's
{\tt 740511}, but not as sure as you would like to be.
%
As an experiment, you pick up the phone and dial {\tt 740511};
you obtain a `busy' signal.
Are you now more sure of your phone number? If so, how much?
}
%
\exercisaxB{1}{ex.othercoin}{
In a game, two coins are tossed. If either of the coins comes up
heads, you have won a prize. To claim the prize, you must point to
one of your coins that is a head
and say `look, that coin's a head, I've won'.
You watch Fred play the game. He tosses the two coins, and he
points to a coin and says `look, that coin's a head, I've won'.
What is the probability that the {\em other\/} coin is a head?
}
%\subsection*{Another quasi-legal story}
% \exercis{ex.}{
% During a radio chat show on the health consequences of
% secondary smoking, it is reported by an expert that
% twelve recent studies have investigated whether
% there was a link between secondary smoking and cancer.
% Of these, eleven studies failed to establish a link
% and one study found significant evidence of a causal
% link -- secondary smoking increasing the risk of getting
% cancer. The expert said that the net evidence from these
% twelve results was that there was significant evidence of a causal
% link.
%
% Shortly thereafter, a Mr.\ N.T.\ Social called in in support
% of smokers' ``rights'' to pollute public air. `If eleven
% of the studies didn't find a link, and only one found a link,
% then it's eleven to one that there isn't a link, isn't it?'
%
% `Well, you clearly don't understand statistics, do you?' responded
% the condescending host.
%
% Can you suggest a more helpful explanation of the expert's statement?
%}
% euro.tex
\exercissxB{2}{ex.eurotoss}{
A statistical statement appeared in
% \footnote{Quoted by Charlotte Denny and Sarah Dennis
{\em The Guardian} on Friday January 4, 2002:
\begin{quote}
When spun on edge 250
times, a Belgian one-euro
coin came up heads 140 times and tails 110.
`It looks very suspicious to me', said Barry Blight, a statistics lecturer
at the London School of Economics.
`If the coin were unbiased the
chance of getting a result as extreme as that would be less than 7\%'.
\end{quote}
But {\em do\/} these
data give evidence that the coin is biased rather than fair?
[Hint: see \eqref{eq.compare.final}.]
}
% \input{tex/bayes_occam.tex}
\dvips
\section{Solutions}% to Chapter \protect\ref{ch.bayes}'s exercises} %
\soln{ex.dieexponential}{
Let the data be $D$. Assuming equal prior probabilities,
\beqan
\frac{P(A \given D)}{P(B \given D)} = \frac{1}{2}\frac{3}{2}\frac{1}{1}\frac{3}{2}
\frac{1}{2}\frac{2}{2}\frac{1}{2} = \frac{9}{32}
\eeqan
and $P(A \given D) = 9/41.$
% (check me).
}
\soln{ex.dieexponentialb}{
The probability of the data given each hypothesis is:
\beq
P(D \given A) = \frac{3}{20}\frac{1}{20}\frac{2}{20}\frac{1}{20}
\frac{3}{20}\frac{1}{20} \frac{1}{20} =
\frac{18}{20^7} ;
\eeq
\beq
P(D \given B) = \frac{2}{20}\frac{2}{20}\frac{2}{20}\frac{2}{20}
\frac{2}{20}\frac{1}{20} \frac{2}{20}
= \frac{64}{20^7} ;
\eeq
\beq
P(D \given C) = \frac{1}{20}\frac{1}{20}\frac{1}{20}\frac{1}{20}
\frac{1}{20}\frac{1}{20} \frac{1}{20}
= \frac{1}{20^7}.
\eeq
So
\beq
% \hspace*{-0.1in}
P(A \given D) = \frac{18}{18+64+1} = \frac{18}{83} ; \hspace{0.3in}
P(B \given D) = \frac{64}{83} ;\hspace{0.3in}
P(C \given D) = \frac{1}{83} .
\eeq
}
\fakesection{Bent coin exercise solns}
\begin{figure}[htbp]
\figuremargin{%
\footnotesize
\begin{center}
\begin{tabular}{cc}
(a) \psfig{figure=figs/aba.ps,width=2in,angle=-90}&
(b) \psfig{figure=figs/bbb.ps,width=2in,angle=-90}\\
$P( p_{\tt{a}} \given \bs\eq {\tt{aba}} ,F\eq 3) \propto p_{\tt{a}}^2 (1-p_{\tt{a}})$
&
$P( p_{\tt{a}} \given \bs\eq {\tt{bbb}} ,F\eq 3) \propto (1-p_{\tt{a}})^3$ \\
\end{tabular}
\end{center}
}{%
\caption[a]{Posterior probability for the bias $p_a$ of a bent coin given
two different data sets.}
\label{fig.aba.bbb}
}%
\end{figure}
\soln{ex.postpaII}{% relabelled from postpa Sun 6/4/03 - beware incorrect refs likely
\ben
\item
$P( p_{\tt{a}} \given \bs\eq {\tt{aba}} ,F\eq 3) \propto p_{\tt{a}}^2 (1-p_{\tt{a}})$.
The most probable value of $p_{\tt{a}}$ (\ie, the value that maximizes
the posterior probability density) is $2/3$.
The mean value of $p_{\tt{a}}$ is $3/5$.
See \figref{fig.aba.bbb}a.
\item
$P( p_{\tt{a}} \given \bs\eq {\tt{bbb}} ,F\eq 3) \propto (1-p_{\tt{a}})^3$.
The most probable value of $p_{\tt{a}}$ (\ie, the value that maximizes
the posterior probability density) is $0$.
The mean value of $p_{\tt{a}}$ is $1/5$.
See \figref{fig.aba.bbb}b.
\een
}
%/home/mackay/_courses/itprnn/figs
%gnuplot> plot x**2*(1-x)
%gnuplot> set xrange [0:1]
%gnuplot> replot
%gnuplot> set nokey
%gnuplot> set size 0.4,0.4
%gnuplot> replot
%gnuplot> set noytics
%gnuplot> replot
%gnuplot> set yrange [0:0.4]
%gnuplot> replot
%gnuplot> set yrange [0:0.17]
%gnuplot> replot
%gnuplot> set term post
%Terminal type set to 'postscript'
%Options are 'landscape monochrome dashed "Helvetica" 14'
%gnuplot> set output "aba.ps"
%gnuplot> replot
%gnuplot> set term X
%Terminal type set to 'X11'
%gnuplot> set yrange [0:1]
%gnuplot> plot (1-x)**3
%gnuplot> set term post
%Terminal type set to 'postscript'
%Options are 'landscape monochrome dashed "Helvetica" 14'
%gnuplot> set output "bbb.ps"
%gnuplot> replot
\fakesection{evidence est}
\begin{figure}[htbp]
\figuremargin{%
\small%
\begin{center}
\begin{tabular}{cccc}
$\H_0$ is true &&
\multicolumn{2}{c}{$\H_1$ is true} \\ \cmidrule{1-1}\cmidrule{3-4}
\sixtoone{$p_a=1/6$}{h0MSD}&&
\sixtoone{$p_a=0.25$}{h6MSD}&
\sixtoone{$p_a=0.5$}{h2MSD}\\
\end{tabular}
\end{center}
}{%
\caption[a]{Range of plausible values of the log evidence in favour of $\H_1$ as
a function of $F$. The vertical axis on the left is
$\log \smallfrac{ P( \bs \given F,\H_1 ) }
{ P( \bs \given F,\H_0 ) }$;
the right-hand vertical axis shows the values of
$\smallfrac{ P( \bs \given F,\H_1 ) }
{ P( \bs \given F,\H_0 ) }$.
\index{typicality!behaviour of evidence}\index{evidence!typical behaviour of}\index{model comparison!typical behaviour of evidence}%
The solid line shows the log evidence if the random variable $F_a$
takes on its mean value, $F_a = p_aF$. The dotted lines show (approximately)
the log evidence if $F_a$ is at its 2.5th or 97.5th percentile.
(See also \protect\figref{fig.evidencetyp}, \pref{fig.evidencetyp}.)
}
\label{fig.evidenceMSD}
}%
\end{figure}
\soln{ex.evidenceest}{
The curves in \figref{fig.evidenceMSD} were found by finding the mean and standard deviation
of $F_a$, then setting $F_a$ to the mean $\pm$ two standard deviations
to get a 95\% plausible range for $F_a$, and computing the three
corresponding values of the log evidence ratio.
}%
\soln{ex.3doors}{
Let $\H_i$ denote the hypothesis that the prize is behind
door $i$.
We make the following assumptions: the three hypotheses
$\H_1$, $\H_2$ and $\H_3$ are equiprobable {\em a priori}, \ie,
\beq
P(\H_1) = P(\H_2) = P(\H_3) = \frac{1}{3} .
\eeq
The datum we receive, after choosing door 1,
is one of $D \eq 3$ and $D \eq 2$ (meaning door 3 or 2 is opened, respectively).
We assume that these two possible outcomes have the following probabilities.
If the prize is behind door 1 then the host has a free choice; in
this case we assume that the host selects at random between $D\eq 2$ and $D\eq 3$.
Otherwise the choice of the host is forced and the probabilities
are 0 and 1.
\beq
\begin{array}{|r@{\,}c@{\,}l|r@{\,}c@{\,}l|r@{\,}c@{\,}l|}
P( D\eq 2 \given \H_1) &=& \dfrac{1}{2} &
P( D\eq 2 \given \H_2) &=& 0 &
P( D\eq 2 \given \H_3) &=& {1} \\
P( D\eq 3 \given \H_1) &=& \dfrac{1}{2} &
P( D\eq 3 \given \H_2) &=& {1} &
P( D\eq 3 \given \H_3) &=& 0
\end{array}
\eeq
Now, using \Bayes\ theorem, we evaluate the posterior probabilities
of the hypotheses:
\beq
P( \H_i \given D\eq3 ) = \frac{P( D\eq3 \given \H_i) P(\H_i) }{P(D\eq3) }
\eeq
\beq
\begin{array}{|r@{\,}c@{\,}l|r@{\,}c@{\,}l|r@{\,}c@{\,}l|}
P(\H_1 \given D\eq 3) &=& \frac{ (1/2) (1/3) }{P(D\normaleq 3) } &
P(\H_2 \given D\eq 3) &=& \frac{ ({1}) (1/3) }{P(D\normaleq 3) } &
P(\H_3 \given D\eq 3) &=& \frac{ ({0}) (1/3) }{P(D\normaleq 3) }
\end{array}
\eeq
The denominator $P(D\eq 3)$ is $(1/2)$ because it is the normalizing
constant for this posterior distribution.
So
\beq
\begin{array}{|rcl|rcl|rcl|}
P( \H_1 \given D\eq3 ) &=& \dfrac{ 1}{3} &
P(\H_2 \given D\eq3) &=& \dfrac{ 2}{3} &
P(\H_3 \given D\eq3) &=& 0 .
\end{array}
\eeq
So the contestant should switch to door 2 in order to have
the biggest chance of getting the prize.
Many people find this outcome surprising. There are two
ways to make it more intuitive. One is to play the game\index{game!three doors}
thirty
times with a friend and keep track of the frequency with
which switching gets the prize. Alternatively,
you can perform a thought experiment in which the game is
played with a million doors. The rules are now that the contestant
chooses one door, then the game show host opens
999,998 doors in such a way as not to reveal the prize, leaving
the {\em contestant's\/}
selected door and {\em one other door\/}
closed. The contestant may
now stick or switch.
Imagine the contestant confronted by a million doors, of which
doors 1 and 234,598 have not been opened, door 1 having been
the contestant's initial guess. Where do you think the prize is?
}
%
\soln{ex.3doorsb}{
% earthquake rules.
If door 3 is opened by an earthquake, the inference comes out
differently -- even though visually the scene looks the same. The
nature of the data, and the probability of the data, are both now
different. The possible data outcomes are, firstly, that any number
of the doors might have opened. We could label the eight possible
outcomes $\bd = (0,0,0), (0,0,1), (0,1,0), (1,0,0), (0,1,1), \ldots,
(1,1,1)$. Secondly, it might be that the prize is visible after the
earthquake has opened one or more doors. So the data $D$ consists of
the value of $\bd$, and a statement of whether the prize was
revealed. It is hard to say what the probabilities of these outcomes
are, since they depend on our beliefs about the reliability
of the door latches and the properties of earthquakes,
but it is possible to extract the desired posterior probability
without naming the values of $P(\bd \given \H_i)$ for each $\bd$. All that
matters are the relative values of the quantities $P(D \given \H_1)$,
$P(D \given \H_2)$, $P(D \given \H_3)$, for the value of $D$ that actually occurred.
[This is the {\dem\ind{likelihood principle}}, which
we met in \sectionref{sec.lp}.]
% !!!!!!!!! add page ref?
The value of $D$ that actually occurred is
`$\bd \eq (0,0,1)$, and no prize visible'. First, it is clear that
$P(D \given \H_3)=0$, since the datum that no prize is visible is
incompatible with $\H_3$. Now, assuming that the contestant selected
door 1, how does the probability $P(D \given \H_1)$ compare with
$P(D \given \H_2)$? Assuming that earthquakes are not sensitive to
decisions of game show contestants,
these two quantities have to be equal, by symmetry. We don't know how likely it is
that door 3 falls off its hinges, but however likely it is, it's just
as likely to do so whether the prize is behind door 1 or door 2. So,
if $P(D \given \H_1)$ and $P(D \given \H_2)$ are equal, we obtain:
\beq
\begin{array}{|r@{\,\,=\,\,}l|r@{\,\,=\,\,}l|r@{\,\,=\,\,}l|}
P(\H_1 | D) & \smallfrac{ P(D | \H_1) (\smalldfrac{1}{3}) }{P(D) } &
P(\H_2 | D) & \smallfrac{ P(D | \H_2) (\smalldfrac{1}{3}) }{P(D) } &
P(\H_3 | D) & \smallfrac{ P(D | \H_3) (\smalldfrac{1}{3}) }{P(D) }
\\
& \dfrac{ 1}{2} &
& \dfrac{ 1}{2} &
& 0 .
\end{array}
\eeq
The two possible hypotheses are now equally likely.
If we assume that
the host knows where the prize is and might be acting
deceptively, then the answer might be further modified, because we
have to view the host's words as part of the data.
Confused? It's well worth making sure you
understand these two gameshow problems.
Don't worry, I slipped up on the second problem, the
first time I met it.
There is a general rule which helps immensely
when you have a confusing probability problem:\index{key points!how to solve probability problems}
\begin{conclusionbox}
Always write down the probability of everything.\\ {
\hfill {\em (Steve Gull)} \par
}
\end{conclusionbox}
From this joint probability, any desired inference can
be mechanically obtained (\figref{fig.everything}).
\amarginfig{b}{
\begin{center}
\newcommand{\tabwidth}{30}
\newcommand{\tabheight}{80}
\setlength{\unitlength}{1mm}{
\begin{picture}(43,92)(-13,0)
\put(15,90){\makebox(0,0){\small\sf{Where the prize is}}}
\put( 5,85){\makebox(0,0){\small{door}}}
\put(15,85){\makebox(0,0){\small{door}}}
\put(25,85){\makebox(0,0){\small{door}}}
\put( 5,82){\makebox(0,0){\small{1}}}
\put(15,82){\makebox(0,0){\small{2}}}
\put(25,82){\makebox(0,0){\small{3}}}
\put(-1, 5){\makebox(0,0)[r]{\footnotesize{1,2,3}}}
\put(-1,15){\makebox(0,0)[r]{\footnotesize{2,3}}}
\put(-1,25){\makebox(0,0)[r]{\footnotesize{1,3}}}
\put(-1,35){\makebox(0,0)[r]{\footnotesize{1,2}}}
\put(-1,45){\makebox(0,0)[r]{\footnotesize{3}}}
\put( 5,75){\makebox(0,0){\footnotesize{$\displaystyle\frac{p_{\rm none}}{3}$}}}
\put(15,75){\makebox(0,0){\footnotesize{$\displaystyle\frac{p_{\rm none}}{3}$}}}
\put(25,75){\makebox(0,0){\footnotesize{$\displaystyle\frac{p_{\rm none}}{3}$}}}
\put( 5,45){\makebox(0,0){\footnotesize{$\displaystyle\frac{p_{3}}{3}$}}}
\put(15,45){\makebox(0,0){\footnotesize{$\displaystyle\frac{p_{3}}{3}$}}}
\put(25,45){\makebox(0,0){\footnotesize{$\displaystyle\frac{p_{3}}{3}$}}}
\put( 5, 5){\makebox(0,0){\footnotesize{$\displaystyle\frac{p_{1,2,3}}{3}$}}}
\put(15, 5){\makebox(0,0){\footnotesize{$\displaystyle\frac{p_{1,2,3}}{3}$}}}
\put(25, 5){\makebox(0,0){\footnotesize{$\displaystyle\frac{p_{1,2,3}}{3}$}}}
\put(-1,55){\makebox(0,0)[r]{\footnotesize{2}}}
\put(-1,65){\makebox(0,0)[r]{\footnotesize{1}}}
\put(-1,75){\makebox(0,0)[r]{\footnotesize{none}}}
\put(-12,40){\makebox(0,0){\rotatebox{90}{\small\sf{Which doors opened by earthquake}}}}
\multiput(0,0)(0,10){9}{\line(1,0){\tabwidth}}
\multiput(0,0)(10,0){4}{\line(0,1){\tabheight}}
\end{picture}}
\end{center}
\caption[a]{The probability of everything, for the second three-door problem,
assuming an earthquake has just occurred.
Here, $p_3$ is the probability that door 3 alone is opened by an earthquake.}
\label{fig.everything}
}
}
\fakesection{simpsons}
\soln{ex.simpsons}{
The statistic quoted by the lawyer indicates the
% {prior\/}
probability
% \index{Simpson, O.J., similar case to}%
%\index{Simpson, O.J., allusion to}
\index{lawyer}\index{wife-beater}\index{murder}
that a randomly selected wife-beater will also murder his wife.
The probability that the husband was the murderer, {\em given
that the wife has been murdered}, is a completely different quantity.
To deduce the latter, we need to make further assumptions about
the probability that the wife is murdered by someone else.
If she lives in a neighbourhood with frequent random murders, then
this probability is large and the posterior probability that
the husband did it (in the absence of other evidence) may not
be very large. But in more peaceful regions, it may well be
that the most likely person to have murdered you, if you are found
murdered, is
one of your closest relatives.
%{\em Numbers here.}
Let's work out some illustrative numbers with the help
of the statistics on page \pageref{footnote.murder}.
Let $m\eq 1$ denote the proposition that a woman has been murdered;
$h\eq 1$, the proposition that the husband did it; and $b\eq 1$,
the proposition that he beat her in the year preceding the
murder. The statement `someone else did it'
is denoted by $h\eq 0$.
We need to define $P(h \given m\eq 1)$, $P(b \given h\eq 1,m\eq 1)$, and $P(b\eq 1 \given h\eq 0,m\eq 1)$
in order to compute the posterior probability $P(h\eq 1 \given b\eq 1,m\eq 1)$.
From the statistics, we can read out $P(h\eq 1 \given m\eq 1)=0.28$.
And if two million women out of 100 million are beaten,
then $P(b\eq 1 \given h\eq 0,m\eq 1)=0.02$. Finally, we need a
value for $P(b \given h\eq 1,m\eq 1)$: if a man murders his wife, how likely is
it that this is the first time he laid a finger on her? I
expect it's pretty unlikely; so maybe $P(b\eq 1 \given h\eq 1,m\eq 1)$ is 0.9
or larger.
By \Bayes\ theorem, then,
\beq
P(h\eq 1 \given b\eq 1,m\eq 1)
= \frac{ .9 \times .28 }{ .9 \times .28 + .02 \times .72 }
\simeq 95\% .
\eeq
One way to make obvious the sliminess of the lawyer on \pref{ex.simpsons}
is to construct arguments, with the same logical structure
as his, that
are clearly wrong. For example, the lawyer could say `Not only
was Mrs.\ S murdered, she was murdered between 4.02pm and
4.03pm. {\em Statistically}, only one in a {\em million\/} wife-beaters
actually goes on to murder his wife between 4.02pm and
4.03pm. So the wife-beating
% , which is not denied by Mr.\ S,
is not strong evidence at all. In fact,
given the wife-beating evidence alone, it's extremely unlikely
that he would murder his wife in this way -- only a
1/1,000,000 chance.'
}
% arrived here Sun 6/4/03
\soln{ex.phonetest}{% was phonecheck
There are two hypotheses.
$\H_0$: your number is {\tt 740511}; $\H_1$: it is another number.
The data, $D$, are `when I dialed {\tt 740511}, I got a busy signal'.
What is the probability of $D$, given each hypothesis?
If your number is {\tt 740511}, then we expect a busy signal with certainty:
\[
P(D \given \H_0) = 1 .
\]
On the other hand, if $\H_1$ is true, then the probability that the number dialled
returns a busy signal is smaller than 1, since various other outcomes
were also possible (a ringing tone, or a number-unobtainable signal,
for example). The value of this probability $P(D \given \H_1)$
will depend on the probability $\alpha$ that a random phone number
similar to your own phone number would be a valid phone number,
and on the probability $\beta$ that you get a busy signal when you dial
a valid phone number.
% 37 per col, 4 cols per page, 250 pages.
% 20 per col, 3 cols per page, 270 pages.
% 50,000. maybe another 50% ex-directory?
I estimate from the size of
my phone book that Cambridge has about $75\,000$ valid phone numbers, all of length six
digits. The probability that a random six-digit number is valid is
therefore about $75\,000/10^6 = 0.075$. If we exclude numbers beginning with 0, 1, and 9
from the random choice, the probability $\a$
is about $75\,000/700\,000 \simeq 0.1$.
If we assume that
telephone numbers are clustered then a misremembered number
might be more likely to be valid than a randomly chosen number; so
the probability, $\alpha$,
that our guessed number would be valid, assuming $\H_1$ is true,
might be bigger than 0.1. Anyway, $\alpha$ must be somewhere between 0.1 and 1.
We can carry forward this uncertainty in the probability
and see how much it matters at the end.
The probability $\beta$ that you get a busy signal when you dial
a valid phone number is equal to the fraction of phones you think are in use
or off-the-hook
when you make your tentative call.
This fraction varies from town to town and with the time of day.
In Cambridge, during the day, I would guess that about 1\% of phones
are in use. At 4am,
% four in the morning,
maybe 0.1\%, or fewer.
The probability $P(D \given \H_1)$ is the product of $\alpha$ and $\beta$,
that is, about $0.1 \times 0.01 = 10^{-3}$. According to
our estimates, there's about a one-in-a-thousand
chance of getting a busy signal when you dial a random number;
or one-in-a-hundred, if valid numbers are strongly clustered;
or one-in-$10^4$, if you dial in the wee hours.
How do the data affect your beliefs about your phone number?
The posterior probability ratio is the likelihood ratio
times the prior probability ratio:
\beq
\frac{ P(\H_0 \given D) }{ P(\H_1 \given D) }
= \frac{ P(D \given \H_0) }{ P(D \given \H_1) }
\frac{ P(\H_0) }{ P(\H_1) } .
\eeq
The likelihood ratio is about 100-to-1 or 1000-to-1, so the posterior
probability ratio is swung by a factor of 100 or 1000 in favour of $\H_0$.
If the prior probability of $\H_0$ was 0.5 then the posterior
probability is
\beq
P(\H_0 \given D) = \frac{1}{1 + \smallfrac{ P(\H_1 \given D) }{ P(\H_0 \given D) } }
\simeq 0.99 \: \mbox{or} \: 0.999 .
\eeq
}
\soln{ex.eurotoss}{
% see also
% http://www.dartmouth.edu/~chance/chance_news/recent_news/chance_news_11.02.html
% for lots of practical info on coin biases.
%%%%%%%%%%%%%%%%%%%%%%%%%%% included by _s8.tex
% First, could confirm his sampling theory
%Sampling theory: number of heads $\sim 125 \pm 8$
%$ \sqrt{62.5}$
%so two-tail probability is
% pr 2*(1-myerf(14.5/7.9)) ans = 0.066440
% if the data were 141 out of 250 then we get
% 2*(1-myerf(15.5/7.9)) ans = 0.049760
\index{euro}We compare the models $\H_0$ -- the coin is fair --
and $\H_1$ -- the \ind{coin} is biased, with
the prior on its bias set to the uniform
distribution $P(p|\H_1)=1$.
% ent, as defined in this chapter.
\amarginfig{t}{
\begin{center}
\mbox{\psfig{figure=gnu/euro.ps,width=1.62in,angle=-90}}
\end{center}
\caption[a]{The probability distribution of the
number of heads given the two hypotheses, that
the coin is fair, and that it is biased, with
the prior distribution of the bias being uniform.
The outcome ($D = 140$ heads) gives weak evidence
in favour of $\H_0$, the hypothesis that the coin is fair.}
\label{fig.euro}
}
[The use of a uniform prior seems reasonable to me, since I know
that some coins, such as American pennies,
have severe biases when spun on edge; so the situations $p=0.01$ or $p=0.1$
or $p=0.95$ would not surprise me.]
\begin{aside}
When I mention $\H_0$ -- the coin is fair -- a pedant would say, `how
absurd to even consider that the coin is fair -- any coin is surely
biased to some extent'. And of course I would agree. So will pedants
kindly understand $\H_0$ as meaning `the coin is fair to within
one part in a thousand, \ie, $p \in 0.5\pm 0.001$'.
\end{aside}
The likelihood ratio is:
% given in \eqref{eq.compare.final}.
\beq
% Bayesian approach: Model comparison:
\frac{ P( D|\H_1 )}
{P( D|\H_0 )}
= \frac{ \smallfrac{ 140! 110! }{ 251! } }{ 1/2^{250} } = 0.48 .
\eeq
Thus the data give scarcely any evidence
either way; in fact they
give weak evidence (two to one) in favour of $\H_0$!
% load 'gnu/euro.gnu'
`No, no', objects the believer in bias, `your silly uniform
prior doesn't represent {\em my\/} prior beliefs about
the bias of biased coins -- I was {\em expecting\/} only a small bias'.
To be as generous as possible to the $\H_1$,
let's see how well it could fare
if the prior were presciently set.
Let us allow a prior of the form
\beq
P(p|\H_1,\a) = \frac{1}{Z(\a)} p^{\a-1}(1-p)^{\a-1},
\:\:\:\: \mbox{where $Z(\a)=\Gamma(\alpha)^2/\Gamma(2 \alpha)$}
\eeq
(a Beta
% Dirichlet (or Beta)
distribution, with the original uniform prior reproduced
by setting $\a=1$). By tweaking $\alpha$,
the likelihood ratio for $\H_1$ over $\H_0$,
\beq
\frac{ P( D|\H_1,\a )}
{P( D|\H_0 )} =
\frac{\Gamma(140 \!+\! \alpha) \, \Gamma(110 \!+\! \alpha) \, \Gamma(2 \alpha) 2^{250}}
{ \Gamma(250 \!+\! 2 \alpha) \, \Gamma(\alpha)^2 },
\eeq
can
be increased a little. It
is shown for several values of $\a$ in \figref{fig.eurot}.%
%
% fig.eurot WAS here but has been moved away to avoid a crunch
% This figure belongs earlier.
\amarginfig{t}{
{\footnotesize
\begin{tabular}{r@{}l@{$\:\:\:$}r@{\hspace*{0.3in}}r@{}l}
\toprule
\multicolumn{2}{c}{$\alpha$}&
\multicolumn{3}{c}{$\displaystyle \frac{ P( D|\H_1,\a )}
{P( D|\H_0 )}$}\\
\midrule
&.37 & & &.25\\
1&.0 & & &.48\\
2&.7 & & &.82\\
7&.4 & &1&.3\\
20& & &1&.8\\
55& & &1&.9\\
148& & &1&.7\\
403& & &1&.3\\
1096& & &1&.1\\
% from euro.dat
\bottomrule
\end{tabular}
}
\caption[a]{Likelihood ratio for various choices of
the prior distribution's hyperparameter $\alpha$.
}
\label{fig.eurot}
}
%
Even the most favourable choice of $\alpha$ ($\a \simeq 50$)
can
yield a likelihood ratio of only two to one in favour of
$\H_1$.
In conclusion, the data are not `very suspicious'. They
can be construed as giving at most two-to-one evidence
in favour of one or other of the two hypotheses.
\begin{aside}
Are these wimpy likelihood ratios the fault
of over-restrictive
priors? Is there any way of producing
a `very suspicious' conclusion?
The prior that is best-matched to the data,
in terms of likelihood,
% and one that surely has to be viewed as unreasonable,
is the prior that sets $p$ to $f \equiv 140/250$ with probability
one. Let's call this model $\H_*$.
% , since it is a parameterless model like $\H_0$.
The likelihood ratio is $P(D|\H_*)/P(D|\H_0) = 2^{250} f^{140} (1-f)^{110}
=6.1$. So the strongest evidence that these data can possibly
muster against the hypothesis that there is no bias is six-to-one.
\end{aside}
% b.blight@lse.ac.uk
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% alternate answers for the case of 141 heads where
% the P value is 0.05 (0.04976)
%
%The outcomes of the computations for this case (141 from 250)
% are
% alpha , likelihood ratio
%
%.3678794412, .3166098681
%1., .6110726692
%2.718281828, 1.049115229
%7.389056099, 1.627382387
%20.08553692, 2.181864309
%54.59815003, 2.303276774
%148.4131591, 1.882663014
%403.4287935, 1.419011740
%1096.633158, 1.168433218
%2980.957987, 1.063851106
%8103.083928, 1.023737702
%22026.46579, 1.008765749
%
% and H_BF achieves 7.796
While we are noticing the absurdly misleading\index{sermon!sampling theory}\index{p-value}
answers that `sampling theory' statistics produces,
such as the \index{p-value}$p$-value of 7\% in the exercise we just solved,
let's stick the boot in.\label{sec.sampling5percent}
If we make a tiny change to the data set, increasing the
number of heads in 250 tosses from 140 to 141,
we find that the $p$-value goes below the mystical value of 0.05
(the $p$-value is 0.0497).
The sampling theory statistician would happily squeak `the probability
of getting a result as extreme as 141 heads is smaller than 0.05 --
we thus reject the null hypothesis at a significance level of 5\%'.
The correct answer
is shown for several values of $\a$ in \figref{fig.eurot141}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% alternate answers for the case of 141 heads where
% the P value is 0.05 (0.04976)
% Radford: Using R, I get that the true p-value (with genuine binomial
%probabilities) for 141 out of 250 is 0.04970679, close to your value.
%5
%The outcomes of the computations for this case (141 from 250)
% are
% alpha , likelihood ratio
%
%.3678794412, .3166098681
%1., .6110726692
%2.718281828, 1.049115229
%7.389056099, 1.627382387
%20.08553692, 2.181864309
%54.59815003, 2.303276774
%148.4131591, 1.882663014
%403.4287935, 1.419011740
%1096.633158, 1.168433218
%2980.957987, 1.063851106
%8103.083928, 1.023737702
%22026.46579, 1.008765749
%
% and H_BF achieves 7.796
The values worth highlighting from this table are, first,
the likelihood ratio when $\H_1$ uses the standard uniform prior,
which is 1:0.61 in favour of the {\em null hypothesis\/} $\H_0$.
Second, the most favourable choice of $\a$, from the
point of view of $\H_1$, can only
yield a likelihood ratio of about 2.3:1 in favour of
$\H_1$.\label{sec.pvalue05}
\amarginfig{c}{
{\footnotesize
\begin{tabular}{r@{}l@{$\:\:\:$}r@{\hspace*{0.3in}}r@{}l}
\toprule
\multicolumn{2}{c}{$\alpha$}&
\multicolumn{3}{c}{$\displaystyle \frac{ P( D'|\H_1,\a )}
{P( D'|\H_0 )}$ }\\
\midrule
&.37 & & &.32\\
1&.0 & & &.61\\
2&.7 & &1&.0\\
7&.4 & &1&.6\\
20& & &2&.2\\
55& & &2&.3\\
148& & &1&.9\\
403& & &1&.4\\
1096& & &1&.2\\
% from euro.dat
\bottomrule
\end{tabular}
}
\caption[a]{Likelihood ratio for various choices of
the prior distribution's \ind{hyperparameter} $\alpha$, when the data are
$D'=141$ heads in 250 trials.
}
\label{fig.eurot141}
}
%
Be warned! A $p$-value of 0.05 is often interpreted
% gives the impression to many
as implying
that the odds are stacked about twenty-to-one
{\em against\/} the null hypothesis. But the truth in this case
is that the evidence
either slightly {\em favours\/} the null hypothesis,
or disfavours it by at most 2.3 to one, depending on
the choice of prior.
% $p$-values
The $p$-values and `\ind{significance level}s' of
\ind{classical statistics}\index{sermon!classical statistics}
should be treated with {\em extreme caution}.\index{caution!sampling theory}
% This is the last we will see of them in this book.
Shun them!
Here ends the sermon.\index{sermon!sampling theory}
% Classical statistics and Microsoft Windows 95 --
% two of the greatest evils to come out of the twentieth century.
}
\dvipsb{solutions bayes}
% \input{tex/_l1b.tex}
%
% message passing was here
%
\renewcommand{\partfigure}{\poincare{8.0}}
\part{Data Compression}
\prechapter{About Chapter}
\fakesection{prerequisites for chapter 2}
%
In this chapter we
discuss how to measure the information content of the outcome
of a random experiment.
This chapter has some tough bits.
If you find the mathematical details hard,
% to follow,
skim through them and keep going -- you'll be able to enjoy Chapters
\ref{ch3} and \ref{ch4} without this chapter's tools.
% of typicality.
\amarginfignocaption{t}{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Cast of characters}
\footnotesize
\begin{tabular}{@{}lp{1.14in}}
\multicolumn{2}{c}{
{\sf Notation}
}\\
\midrule
$x \in \A$ & $x$ is a {\dem{member}\/} of the \ind{set} $\A$ \\
$\S \subset \A$ & $\S$ is a {\dem\ind{subset}\/} of the set $\A$ \\
$\S \subseteq \A$ & $\S$ is a {\ind{subset}} of, or equal to, the set $\A$ \\
% \union
$\V = \B \cup \A$
& $\V$ is the {\dem\ind{union}\/} of the sets $\B$ and $\A$ \\
$\V = \B \cap \A$
& $\V$ is the {\dem\ind{intersection}\/} of the sets $\B$ and $\A$ \\
$|\A|$ & number of elements in set $\A$\\
\bottomrule
\end{tabular} \medskip
% end marginstuff
}%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Before reading \chref{ch2}, you should have
read
% section \ref{ch1.secprob}
\chref{ch1.secprob}
and
worked on
% \exerciseref{ex.expectn}.
% It will also help if you have worked on
%
% do I need to ensure that {ex.Hadditive} occurs earlier?
%
\exerciseonlyrange{ex.expectn}{ex.Hineq} and \ref{ex.sumdice}
% \exerciseonlyrangeshort{ex.sumdice}{ex.RNGaussian}
\pagerange{ex.sumdice}{ex.invP},
% {ex.RNGaussian}.
% exercises \exnine-\exfourteen\ and \extwentyfive-\extwentyseven.
and \exerciseonlyref{ex.weigh} below.
The following
exercise is intended to
help you think about how to measure information content.
% Please work on this exercise now.
% weighing
% ITPRNN Problem 1
%
% weighing problem
%
\fakesection{the weighing problem}
\exercissxA{2}{ex.weigh}{
-- {\em Please work on this problem before reading \chref{ch.two}.}
\index{weighing problem}You are given 12 balls, all equal in weight except for
one that is either heavier or lighter. You are also given a two-pan
\ind{balance} to use.
% , which you are to use as few times as possible.
In each use of the balance you may put {any\/} number of the 12
balls on the left pan, and the same number on the right pan, and push
a button to initiate the weighing; there are three possible outcomes:
either the weights are equal, or the balls on the left are heavier,
or the balls on the left are lighter. Your task is to design a
strategy to determine which is the odd ball {\em and\/} whether it is
heavier or lighter than the others {\em in as few uses of the balance
as possible}.
% There will be a prize for the best answer.
While thinking about this problem,
you
% should
may find it helpful to
consider the following questions:
\ben
\item How can one measure {\dem\ind{information}}?
\item When you have identified the odd ball and whether it is heavy or
light, how much information have you gained?
\item Once you have designed a strategy, draw a tree showing,
for each of the possible outcomes
of a weighing, what weighing you perform next.
At each node in the tree, how much information have the outcomes
so far given you, and how much information remains to be
gained?
% What is the probability of each of the possible outcomes of the first
% weighing?
%\item
% What is the most information you can get from a single weighing?
% How much information do you get from a single weighing
% if the three outcomes are equally probable?
%\item What is the smallest number of weighings that might conceivably
%be sufficient always to identify the odd ball and whether it is heavy
%or light?
\item How much information is gained when you learn (i) the state of a
flipped coin; (ii) the states of two flipped coins;
(iii) the outcome when a four-sided die is rolled?
\item
How much information is gained on the first step of the weighing
problem if 6 balls are weighed against the other 6? How much is gained
if 4 are weighed against 4 on the first step, leaving out 4 balls?
% the other 4 aside?
\een
}
%
% How many possible outcomes of an e weighing process are there? To put it another way, imagine that you report the outcome by sending a postcard which says, for example, "ball number 5 is heavy", how many prepare a postcard
%
% how many outcomes are there?
% How many possible states of the world are y
% if you tell someone ball number x is heavy, how much info have you given
% them? how much information can be conveyed by $k$ uses of the balance?
%
%
% make clear that you can put any objects on the scales,
% don't have to weigh 6 vs 6.
% no cheating by gradually adding weights
%
% katriona's problem: 4 bits, randomly rotated every time you ask them
% to be flipped.
%
% hhhh llll gggg
% hhll lhgg lh
% if left is h then
% hh or l
% so do h vs h
%
% else gggg gggg ????
% -> ?? ?g
% -> hh l or ggg -> wegh last dude (1 bit)
% do h vs h
%
% if 13 and good avail, - hhhhh llll* gggg
% hhll lhgg hhl
%
\mysetcounter{page}{76}
\ENDprechapter
\chapter{The Source Coding Theorem}
\label{ch.two}\label{ch2}\label{chtwo}
% _l2.tex
% \part{Data Compression}
% \chapter{The Source Coding Theorem}
%
% I introduce the idea of a "name" (or label?) here, and should clarify
% (example 2.1)
%
% E = 13%, Q,Z = 0.1%
% TH = 3.7%
%
% New plan for this chapter:
% \section{Key concept}
% Rather than $H(\bp)$ being the measure of information content of
% an ensemble,
% I want the central idea of this chapter to be that
% $\log 1/P(\bx)$ is the information content of a particular
% outcome $\bx$. $H$ is then of interest because it is the average
% information content.
%
% An example to illustrate this is `hunt the professor'. Or crack
% the combination. Guess the PIN.
% An absent-minded professor wishes to remember an
% integer between 1 and 256, that is, eight bits of information.
% He takes 256 large numbered cardboard boxes, and climbs
% in the box whose number is the integer to be remembered.
% The only way to find him
% is to open the lid of a box. A single experiment involves
% opening a particular box. The outcome is either $x={\tt n}$ -- no
% professor -- or $x={\tt y}$ -- the professor is in there.
% The probabilities are
% \beq
% P(x\eq {\tt n}) = 255/256; P(x\eq {\tt y}) = 1/256.
% \eeq
% We open box $n$.
% If the professor is revealed, we have learned the integer,
% and thus recovered 8 bits of information. If he is not revealed,
% we have learned very little -- simply that the
% integer is not $n$. The information contents are:
% \beq
% h(x\eq 0) = \log_2( 256/255) = 0.0056 ; h(x\eq 1) = \log_2 256 = 8 .
% \eeq
% The average information content is
% \beq
% H(X) = 0.037 \bits .
% \eeq
% This example shows that in the event of an improbable outcome's occuring,
% a large amount of information really is conveyed.
%
% \section{Weighing problem}
% The weighing problem remains useful, let's keep it.
%
% \section{Source coding theorem}
% Relate `information content' $\log 1/P$ to message length
% in two steps. First, establish the AEP, that
% the outcome from an ensemble $X^N$
% is very likely to lie in a typical set having `information
% content' close to NH.
%
% Second, show that we can count the number of elements in the
% typical set, give them all names, and the number of
% names will be about $2^{NH}$.
%
% At what point should $H_{\delta}$ be introduced?
\section{How to measure the information content of a random variable?}
In the next few chapters, we'll be talking about probability
distributions and random variables. Most of the time
we can get by with sloppy notation, but occasionally, we will need
precise notation. Here is the
%definition and
notation that we established in \chapterref{ch.prob.ent}.\indexs{ensemble}
%
\sloppy
\begin{description}
\item[An ensemble] $X$ is a triple $(x,\A_X, \P_X)$,
where the {\dem outcome\/} $x$ is the value of a random variable,
% whose value $x$ can take on a
which takes on one of a
set of possible values,
% the alphabet
% {\em outcomes},
$\A_X = \{a_1,a_2,\ldots,a_i,\ldots, a_I\}$,
% \ie, possible values for a random variable $x$
% and a probability distribution over them,
having probabilities
$\P_X = \{p_1,p_2,\ldots, p_I\}$, with $P(x\eq a_i) = p_i$,
$p_i \geq 0$ and $\sum_{a_i \in \A_X} P(x \eq a_i) = 1$.
\end{description}
%\begin{description}
%\item[An ensemble] $X$ is a random variable $x$ taking on a value
% from a set of possible {\em outcomes},
% $$\A_X \eq \{a_1,\ldots,a_I\},$$
% having probabilities
% $$\P_X = \{p_1,\ldots, p_I\},$$ with $P(x\eq a_i) = p_i$,
% $p_i \geq 0$ and $\sum_{x \in \A_X} P(x) = 1$.
%\end{description}
% An ensemble is a set of possible values for a random variable
% and a probability distribution over them.
{How can we measure the information content of an outcome
$x = a_i$ from such an ensemble?}
In this chapter we examine the assertions
\ben
\item
that the
% It is claimed that the
{\dem{\ind{Shannon information content}}},\index{information content!Shannon}\index{information content!how to measure}
\beq
h(x\eq a_i) \equiv \log_2 \frac{1}{p_i},
\eeq
is a sensible measure of the information content of the outcome
$x = a_i$, and
\item
that
the {\dem{\ind{entropy}}} of the ensemble,
\beq
H(X) = \sum_i p_i \log_2 \frac{1}{p_i},
\eeq
is a sensible measure of the ensemble's average information content.
\een
\begin{figure}[htbp]
\figuremargin{%1
{\small%
\begin{center}
\mbox{
\mbox{
\hspace{-9mm}
\mbox{\psfig{figure=figs/h.ps,%
width=42mm,angle=-90}}$p$
\hspace{-35mm}
\makebox[0in][l]{\raisebox{\hpheight}{$h(p)= \log_2 \displaystyle \frac{1}{p}$ }}
\hspace{35mm}
}
\hspace{0.9mm}
\begin{tabular}[b]{ccc}\toprule
$p$ & $h(p)$ & $H_2(p)$ \\ \midrule
0.001 & 10.0 & 0.011 \\ % 9.96578 & 0.0114078
0.01\phantom{0} & \phantom{1}6.6 & 0.081 \\
0.1\phantom{01} & \phantom{1}3.3 & 0.47\phantom{1} \\
0.2\phantom{01} & \phantom{1}2.3 & 0.72\phantom{1} \\
0.5\phantom{01} & \phantom{1}1.0 & 1.0\phantom{01} \\ \bottomrule
\end{tabular}
\mbox{
% to put H at left: \hspace{1.2mm}
\hspace{6.2mm}
\raisebox{\hpheight}{$H_2(p)$}
% to put H at left: \hspace{-7.5mm}
\hspace{-20mm}
\mbox{\psfig{figure=figs/H2.ps,%
width=42mm,angle=-90}}$p$
}
% see also H2x.tex
\end{center}
}% end small
}{%
\caption[a]{The \ind{Shannon information content} $h(p) = \log_2 \frac{1}{p}$ and
the binary entropy function $H_2(p)=H(p,1\!-\!p)=p \log_2 \frac{1}{p}
+ (1-p)\log_2 \frac{1}{(1-p)}$ as a function of $p$.}
\label{fig.h2}
}%
\end{figure}
% gnuplot
% load 'figs/l2.gnu'
\noindent
\Figref{fig.h2} shows the Shannon information content
of an outcome with probability $p$, as a function of $p$.
The less probable an outcome is, the greater its
Shannon information content.
\Figref{fig.h2} also shows
% $h(p) = \log_2 \frac{1}{p}$,
the binary entropy function,
\beq
H_2(p)=H(p,1\!-\!p)=p \log_2 \frac{1}{p}
+ (1-p)\log_2 \frac{1}{(1-p)} ,
\eeq
which is the entropy of the ensemble $X$ whose alphabet and probability
distribution are
$\A_X = \{ a , b \}, \P_X = \{ p , (1-p) \}$.
%
\subsection{Information content of independent random variables}
Why should $\log 1/p_i$ have anything to do with the
information content? Why not some other function of $p_i$?
We'll explore this question in detail shortly,
but first, notice a nice property of this particular function
$h(x)=\log 1/p(x)$.
Imagine learning the value of two {\em independent\/} random
variables, $x$ and $y$.
The definition of independence is that the probability
distribution is separable into a {\em product}:
\beq
P(x,y) = P(x) P(y) .
\eeq
Intuitively, we might want any measure of
the `amount of information gained' to have the property of
{\em additivity} --
that is,
for independent random variables $x$ and $y$,
the information gained when we learn $x$ and $y$ should
equal the sum of the information gained if $x$ alone were learned
and the information gained if $y$ alone were learned.
The Shannon information content of the outcome $x,y$ is
\beq
h(x,y) = \log \frac{1}{P(x,y)}
= \log \frac{1}{P(x)P(y)}
= \log \frac{1}{P(x)}
+ \log \frac{1}{P(y)}
\eeq
so it does indeed satisfy
\beq
h(x,y) = h(x) + h(y), \:\:\mbox{if $x$ and $y$ are independent.}
\eeq
\exercissxA{1}{ex.Hadditive}{
Show that, if $x$ and $y$ are independent,
the entropy of the outcome $x,y$
satisfies
\beq
H(X,Y) = H(X) + H(Y) .
\eeq
In words, entropy is additive for independent variables.
}
We now explore these ideas with some examples;
then, in section \ref{sec.aep} and in Chapters \ref{ch3}
and \ref{ch4}, we prove that
the Shannon information content and the entropy are
related to the number of bits needed to describe
the outcome of an experiment.
% \section{Thinking about information content}
% \subsection{Ensembles with maximum average information content}
% The first property of the entropy that we will
% consider is the property that you proved when you solved
% \exerciseref{ex.Hineq}: the entropy of an ensemble
% $X$ is biggest if all the outcomes
% have equal probability $p_i \eq 1/|X|$.
%
% If entropy measures the average information content
% of an ensemble, then this idea of equiprobable outcomes
% should have relevance for the design of efficient experiments.
\subsection{The weighing problem: designing informative experiments}
Have you solved the \ind{weighing problem}\index{puzzle!weighing 12 balls}
\exercisebref{ex.weigh}\
yet? Are you sure? Notice that in three uses of the balance --
which reads either `left heavier', `right heavier', or `balanced' --
the number
of conceivable outcomes is $3^3=27$, whereas the number of possible
states of the world is 24: the odd ball could be any of twelve balls,
and it could be heavy or light. So in principle, the problem might be
solvable in three weighings -- but not in two, since $3^2 < 24$.
If you know how you
{can} determine the odd weight {\em and\/} whether it is heavy or
light in {\em three\/} weighings, then you may read on.
If you haven't found a strategy that always gets there in three weighings,
I encourage you to think about \exerciseonlyref{ex.weigh} some more.
% {ex.weigh}
% \subsection{Information from experiments}
Why is your strategy optimal? What is it about your series of weighings
that allows useful information to be gained as quickly as possible?
\begin{figure}%[htbp]
\fullwidthfigureright{%
% included by l2.tex
%
% shows weighing trees, ternary
%
% decisions of what to weigh are shown in square boxes with 126 over 345 (l:r)
% state of valid hypotheses are listed in double boxes
% three arrows, up means left heavy, straight means right heavy, down is balance
% actually s and d boxes end up having the same defn.
%
\setlength{\unitlength}{0.56mm}% page width is 160mm % was 6mm
\begin{center}
\small
\begin{picture}(260,260)(-50,-130)
%
% initial state
%
% all 24 hypotheses
\mydbox{-50,-100}{15,200}{$1^+$\\$2^+$\\$3^+$\\$4^+$\\$5^+$\\$6^+$\\$7^+$\\
$8^+$\\$9^+$\\$10^+$\\$11^+$\\$12^+$\\$1^-$\\$2^-$\\$3^-$\\$4^-$\\
$5^-$\\$6^-$\\$7^-$\\$8^-$\\$9^-$\\$10^-$\\$11^-$\\$12^-$}
\mysbox{-30,-8}{25,16}{$\displaystyle\frac{1\,2\,3\,4}{5\,6\,7\,8}$}
\put(-30,10){\makebox(25,8){weigh}}
%
% 1st arrows
%
\mythreevector{0,0}{1}{3}{30}
%
% first three boxes of hypotheses % boxes of actions
% #1 is bottom left corner, so has to be offset by height of box
% #2 is dimensions of box
%
% each digit is about 10 high
%
\mydbox{40,55}{15,70}{$1^+$\\$2^+$\\$3^+$\\$4^+$\\$5^-$\\$6^-$\\$7^-$\\$8^-$}
\mysbox{65,82}{25,16}{$\displaystyle\frac{1\,2\,6}{3\,4\,5}$}
\put(65,100){\makebox(25,8){weigh}}
\mydbox{40,-35}{15,70}{$1^-$\\$2^-$\\$3^-$\\$4^-$\\$5^+$\\$6^+$\\$7^+$\\$8^+$}
\mysbox{65,-8}{25,16}{$\displaystyle\frac{1\,2\,6}{3\,4\,5}$}
\put(65,10){\makebox(25,8){weigh}}
\mydbox{40,-125}{15,70}{$9^+$\\$10^+$\\$11^+$\\$12^+$\\$9^-$\\$10^-$\\$11^-$\\$12^-$}
\mysbox{65,-98}{25,16}{$\displaystyle\frac{9\,10\,11}{1\,2\,3}$}
\put(65,-80){\makebox(25,8){weigh}}
%
% 2nd arrows
%
\mythreevector{95,90}{1}{2}{15}
\mythreevector{95,0}{1}{2}{15}
\mythreevector{95,-90}{1}{2}{15}
% nine intermediate states. top ones
\mydbox{115,113}{35,14}{$1^+2^+5^-$}
\mysbox{155,112}{25,16}{$\displaystyle\frac{1}{2}$}
\mydbox{115,83}{35,14}{$3^+4^+6^-$}
\mysbox{155,82}{25,16}{$\displaystyle\frac{3}{4}$}
\mydbox{115,53}{35,14}{$7^-8^-$}
\mysbox{155,52}{25,16}{$\displaystyle\frac{1}{7}$}
% nine intermediate states. mid ones
\mydbox{115,23}{35,14}{$6^+3^-4^-$}
\mysbox{155,22}{25,16}{$\displaystyle\frac{3}{4}$}
\mydbox{115,-7}{35,14}{$1^-2^-5^+$}
\mysbox{155,-8}{25,16}{$\displaystyle\frac{1}{2}$}
\mydbox{115,-37}{35,14}{$7^+8^+$}
\mysbox{155,-38}{25,16}{$\displaystyle\frac{7}{1}$}
% nine intermediate states. bot ones
\mydbox{115,-67}{35,14}{$9^+10^+11^+$}
\mysbox{155,-68}{25,16}{$\displaystyle\frac{9}{10}$}
\mydbox{115,-97}{35,14}{$9^-10^-11^-$}
\mysbox{155,-98}{25,16}{$\displaystyle\frac{9}{10}$}
\mydbox{115,-127}{35,14}{$12^+12^-$}
\mysbox{155,-128}{25,16}{$\displaystyle\frac{12}{1}$}
% 3rd arrows mainline
\mythreevector{185,60}{1}{1}{10}
\mythreevector{185,0}{1}{1}{10}
\mythreevector{185,-60}{1}{1}{10}
% other branch lines
\mythreevector{185,120}{1}{1}{10}
\mythreevector{185,90}{1}{1}{10}
\mythreevector{185,30}{1}{1}{10}
\mythreevector{185,-30}{1}{1}{10}
\mythreevector{185,-90}{1}{1}{10}
\mythreevector{185,-120}{1}{1}{10}
% final answers aligned at 200,x*10
\mydbox{200,126}{10,8}{$1^+$}
\mydbox{200,116}{10,8}{$2^+$}
\mydbox{200,106}{10,8}{$5^-$}
\mydbox{200,96}{10,8}{$3^+$}
\mydbox{200,86}{10,8}{$4^+$}
\mydbox{200,76}{10,8}{$6^-$}
\mydbox{200,66}{10,8}{$7^-$}
\mydbox{200,56}{10,8}{$8^-$}
\mydbox{200,46}{10,8}{$\star$}% ---------- impossible outcome
\mydbox{200,36}{10,8}{$4^-$}
\mydbox{200,26}{10,8}{$3^-$}
\mydbox{200,16}{10,8}{$6^+$}
\mydbox{200,6}{10,8}{$2^-$}
\mydbox{200,-4}{10,8}{$1^-$}% the middle, 0
\mydbox{200,-14}{10,8}{$5^+$}
\mydbox{200,-24}{10,8}{$7^+$}
\mydbox{200,-34}{10,8}{$8^+$}
\mydbox{200,-44}{10,8}{$\star$}
\mydbox{200,-54}{10,8}{$9^+$}
\mydbox{200,-64}{10,8}{$10^+$}
\mydbox{200,-74}{10,8}{$11^+$}
\mydbox{200,-84}{10,8}{$10^-$}
\mydbox{200,-94}{10,8}{$9^-$}
\mydbox{200,-104}{10,8}{$11^-$}
\mydbox{200,-114}{10,8}{$12^+$}
\mydbox{200,-124}{10,8}{$12^-$}
\mydbox{200,-134}{10,8}{$\star$}
\end{picture}
\end{center}
}{%
\caption[a]{An optimal solution to the weighing problem.
%
At each step there are two boxes: the left box shows which hypotheses are still
possible; the right box shows the balls involved in the next weighing.
The 24 hypotheses are written $1^+,
% 2^+,\ldots,1^-,
\ldots, 12^-$,
with, \eg, $1^+$ denoting that 1 is the odd ball and
it is heavy.
Weighings are written by listing the names of the balls on the
two pans, separated by a line; for example, in the first weighing,
% $\displaystyle\frac{1\,2\,3\,4}{5\,6\,7\,8}$ denotes that
balls 1,
2, 3, and 4 are put on the left-hand side and 5, 6, 7, and 8 on the
right.
In each triplet of arrows the upper arrow leads to the situation when
the left side is heavier, the middle arrow to the situation when the right side is heavier,
and the lower arrow to the situation when the outcome is balanced.
The three points labelled $\star$
% arrows without subsequent boxes at the right-hand side
correspond to impossible outcomes.
%The total number of outcomes
% of the weighing process is 24, which equals $3^3 - 3$, so we would expect
% this ternary tree of depth three to have three spare branches.
}
\label{fig.weighing}\label{ex.weigh.sol}
}%
\end{figure}
The answer is that at each step of an optimal
procedure, the three outcomes (`left heavier', `right heavier', and `balance')
are {\em as close as possible to equiprobable}.
An optimal solution is shown in \figref{fig.weighing}.
Suboptimal strategies, such as weighing balls 1--6 against 7--12
on the first step, do not achieve all outcomes with equal probability:
these two sets of balls can never balance, so the only possible
outcomes are `left heavy' and `right heavy'.
% Similarly, strategies
% that after an unbalanced initial result
% do not mix together balls that might be heavy with balls that
% might be light are incapable of giving one of the three outcomes.
Such a binary outcome rules out only half of the possible
hypotheses, so a strategy that uses such outcomes must sometimes
take longer to find the right answer.
% Some suboptimal strategies produce binary trees rather than ternary trees like
% the one in \figref{fig.weighing}, and binary trees
% are necessarily deeper than balanced ternary trees
% with the same number of leaves.
The insight that the outcomes should be as near as possible
to equiprobable makes
it easier to search for an optimal strategy. The first weighing
must divide the 24 possible hypotheses into three groups of eight. Then
the second weighing must be chosen so that there is a 3:3:2
split of the hypotheses.
Thus we might conclude:
\begin{conclusionbox}
{the outcome of a random experiment is guaranteed to be most informative
if the probability distribution over outcomes is uniform.}
\end{conclusionbox}
This conclusion agrees with
the property of the entropy that you proved when you solved
\exerciseref{ex.Hineq}: the entropy of an ensemble
$X$ is biggest if all the outcomes
have equal probability $p_i \eq 1/|\A_X|$.
% for anyone who wants to play it against a machine:
% http://y.20q.net:8095/btest
% http://www.smalltime.com/dictator.html
% http://www.guessmaster.com/
\subsection{Guessing games}
In the game of \ind{twenty questions},\index{game!twenty questions}
one player thinks of
an object, and the other player attempts to guess what the object is
by asking questions that have yes/no answers, for example,
`is it alive?', or `is it human?'
The aim is to identify the object with as few questions
as possible.
What is the best strategy for playing this game?
For simplicity, imagine that we are playing the rather dull
version of twenty questions called `sixty-three'.
% % two hundred and fifty five'.
% In this game, the permitted objects are the $2^6$ integers
% $\A_X = \{ 0 , 1 , 2 , \dots 63 \}$.
% One player selects an $x \in \A_X$, and we ask
% questions that have yes/no answers in order to identify $x$.
\exampl{example.sixtythree}{ {\sf The game `sixty-three'}.
What's the smallest number of yes/no questions needed\index{game!sixty-three}
to identify an integer $x$ between 0 and 63?\index{twenty questions}
}
Intuitively,
the best questions successively divide
the 64 possibilities into equal sized sets.
Six questions suffice.
One reasonable strategy asks the following questions:
%
% want a computer program environment here.
%
\begin{quote}
\begin{tabbing}
{\sf 1:} is $x \geq 32$? \\
{\sf 2:} is $x \mod 32 \geq 16$? \\
{\sf 3:} is $x \mod 16 \geq 8$? \\
{\sf 4:} is $x \mod 8 \geq 4$? \\
{\sf 5:} is $x \mod 4 \geq 2$? \\
{\sf 6:} is $x \mod 2 = 1$?
\end{tabbing}
\end{quote}
%
% I'd like to put this in a comment column on the right beside the 'code':
%
[The notation $x \mod 32$, pronounced `$x$ modulo 32', denotes the remainder
when $x$ is divided by 32; for example, $35 \mod 32 = 3$
and $32 \mod 32 = 0$.]
The answers to these questions, if translated
from $\{\mbox{yes},\mbox{no}\}$
to $\{{\tt{1}},{\tt{0}}\}$,
give the binary expansion of $x$, for example
$35 \Rightarrow {\tt{100011}}$.\ENDsolution\smallskip
What are the
Shannon information contents of the outcomes in this example?
If we assume that all values of $x$ are equally likely, then the
answers to the questions are independent and each has
% entropy $H_2(0.5) = 1 \ubit$. The
Shannon information content
% of each answer is
$\log_2 (1/0.5)
= 1 \ubit$; the total Shannon information gained
is always six bits. Furthermore, the number $x$ that we learn from
these questions is a six-bit binary number. Our questioning
strategy defines a way of encoding the random variable $x$
as a binary file.
So far, the Shannon information content makes sense:
it measures the length of a binary file that encodes
$x$.
%
However, we have not yet studied ensembles where the
outcomes have unequal probabilities. Does the
Shannon information content make sense there too?
\fakesection{Submarine figure}
%
\newcommand{\subgrid}{\multiput(0,0)(0,10){9}{\line(1,0){80}}\multiput(0,0)(10,0){9}{\line(0,1){80}}}
\newcommand{\sublabels}{
\put(-5,75){\makebox(0,0){\sf\tiny{A}}}
\put(-5,65){\makebox(0,0){\sf\tiny{B}}}
\put(-5,55){\makebox(0,0){\sf\tiny{C}}}
\put(-5,45){\makebox(0,0){\sf\tiny{D}}}
\put(-5,35){\makebox(0,0){\sf\tiny{E}}}
\put(-5,25){\makebox(0,0){\sf\tiny{F}}}
\put(-5,15){\makebox(0,0){\sf\tiny{G}}}
\put(-5, 5){\makebox(0,0){\sf\tiny{H}}}
%
\put(75,-5){\makebox(0,0){\tiny{8}}}
\put(65,-5){\makebox(0,0){\tiny{7}}}
\put(55,-5){\makebox(0,0){\tiny{6}}}
\put(45,-5){\makebox(0,0){\tiny{5}}}
\put(35,-5){\makebox(0,0){\tiny{4}}}
\put(25,-5){\makebox(0,0){\tiny{3}}}
\put(15,-5){\makebox(0,0){\tiny{2}}}
\put( 5,-5){\makebox(0,0){\tiny{1}}}
}
\newcommand{\misssixteen}{
\put(45,65){\makebox(0,0){$\times$}}
\put(45,45){\makebox(0,0){$\times$}}
\put(35,75){\makebox(0,0){$\times$}}
\put(35,65){\makebox(0,0){$\times$}}
\put(35,55){\makebox(0,0){$\times$}}
\put(35,45){\makebox(0,0){$\times$}}
\put(35,35){\makebox(0,0){$\times$}}
\put(35,25){\makebox(0,0){$\times$}}
\put(35,15){\makebox(0,0){$\times$}}
\put(35, 5){\makebox(0,0){$\times$}}
\put(25,75){\makebox(0,0){$\times$}}
\put(25,65){\makebox(0,0){$\times$}}
\put(25,55){\makebox(0,0){$\times$}}
\put(25,45){\makebox(0,0){$\times$}}
\put(25,35){\makebox(0,0){$\times$}}
\put(25,25){\makebox(0,0){$\times$}}
\put(25,15){\makebox(0,0){$\times$}}
}
\newcommand{\missthirtytwo}{
\put(75,75){\makebox(0,0){$\times$}}
\put(75,65){\makebox(0,0){$\times$}}
\put(75,55){\makebox(0,0){$\times$}}
\put(75,45){\makebox(0,0){$\times$}}
\put(75,35){\makebox(0,0){$\times$}}
\put(75,25){\makebox(0,0){$\times$}}
\put(75,15){\makebox(0,0){$\times$}}
\put(75, 5){\makebox(0,0){$\times$}}
\put(65,75){\makebox(0,0){$\times$}}
\put(65,65){\makebox(0,0){$\times$}}
\put(65,55){\makebox(0,0){$\times$}}
\put(65,45){\makebox(0,0){$\times$}}
\put(65,35){\makebox(0,0){$\times$}}
\put(65,25){\makebox(0,0){$\times$}}
\put(65,15){\makebox(0,0){$\times$}}
\put(65, 5){\makebox(0,0){$\times$}}
\put(55,75){\makebox(0,0){$\times$}}
\put(55,65){\makebox(0,0){$\times$}}
\put(55,55){\makebox(0,0){$\times$}}
\put(55,45){\makebox(0,0){$\times$}}
\put(55,35){\makebox(0,0){$\times$}}
\put(55,25){\makebox(0,0){$\times$}}
\put(55,15){\makebox(0,0){$\times$}}
\put(55, 5){\makebox(0,0){$\times$}}
\put(45,75){\makebox(0,0){$\times$}}
%%\put(45,65){\makebox(0,0){$\times$}}
\put(45,55){\makebox(0,0){$\times$}}
%% \put(45,45){\makebox(0,0){$\times$}}
\put(45,35){\makebox(0,0){$\times$}}
\put(45,25){\makebox(0,0){$\times$}}
\put(45,15){\makebox(0,0){$\times$}}
\put(45, 5){\makebox(0,0){$\times$}}
\put(5,65){\makebox(0,0){$\times$}}
}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%% submarine figure %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}
\figuredangle{%
\begin{center}
%\begin{tabular}{l@{\hspace{-1mm}}*{5}{@{\hspace{2pt}}c}} \toprule
\begin{tabular}{l@{\hspace{0mm}}*{5}{@{\hspace{8.5mm}}c}} \toprule
% moves made & 1 & 2 & 32 & 48 & 49 \\
&
%
% 1 miss
%
% this fig actually needs extra width on left, but there is nothing there.
\setlength{\unitlength}{0.26mm}
\begin{picture}(80,95)(0,-10)\subgrid\sublabels
\put(25,15){\makebox(0,0){$\times$}}
\put(25,15){\circle{15}}
\end{picture}
&
%
% 2 miss
%
\setlength{\unitlength}{0.26mm}
\begin{picture}(80,95)(0,-10)\subgrid
\put(25,15){\makebox(0,0){$\times$}}
\put(5,65){\makebox(0,0){$\times$}}
\put(5,65){\circle{15}}
\end{picture}
&
%
% 32 miss
%
\setlength{\unitlength}{0.26mm}
\begin{picture}(80,95)(0,-10)\subgrid
\put(25,15){\makebox(0,0){$\times$}}
\put(45,35){\circle{15}}
\missthirtytwo
\end{picture}
&
%
% 49 miss
%
\setlength{\unitlength}{0.26mm}
\begin{picture}(80,95)(0,-10)\subgrid
\put(25,15){\makebox(0,0){$\times$}}
\put(5,65){\makebox(0,0){$\times$}}
\missthirtytwo
\misssixteen
\put(25,25){\circle{15}}
\end{picture}
&
\setlength{\unitlength}{0.26mm}
\begin{picture}(80,95)(0,-10)\subgrid
\put(25,15){\makebox(0,0){$\times$}}
\put(5,65){\makebox(0,0){$\times$}}
\missthirtytwo
\misssixteen
%%%%%%%%%%%%%%%%%%%%%%% hit the submarine:
\put(25,5){\circle{15}}
\put(25,5){\makebox(0,0){\tiny\bf S}}
\end{picture}
\\
move \# & 1 & 2 & 32 & 48 & 49 \\
question
& G3
& B1
& E5
& F3
& H3 \\
outcome
& $x = {\tt n}$ % $(\times)$
& $x = {\tt n}$ %$(\times)$
& $x = {\tt n}$ %$(\times)$
& $x = {\tt n}$ %$(\times)$
& $x = {\tt y}$ %({\small\bf S})
\\[0.1in]
$P(x)$
& $\displaystyle\frac{63}{64}$
& $\displaystyle\frac{62}{63}$
& $\displaystyle\frac{32}{33}$
& $\displaystyle\frac{16}{17}$
& $\displaystyle\frac{1}{16}$
\\[0.15in]
$h(x)$
& 0.0227
& 0.0230
& 0.0443
% & 0.0430 -------- 0.9556 , just before 32 are pasted
& 0.0874
& 4.0
\\[0.05in]
Total info.
& 0.0227
& 0.0458
& 1.0
& 2.0
& 6.0
\\ \bottomrule
\end{tabular}
\end{center}
}{%
\caption[a]{A game of {\tt submarine}. The submarine is hit on the 49th attempt.}
\label{fig.sub}
}%
\end{figure}
\subsection{The game of {\ind{submarine}}: how many bits can one bit convey?}
In the game of {\ind{battleships}}, each player hides a fleet of
ships in a sea represented by a square grid. On each\index{game!submarine}
turn, one player
attempts to hit the other's ships by firing at one square
in the opponent's sea. The response to a selected square such
as `G3' is either `miss', `hit', or `hit and destroyed'.
In a
% rather
boring version of battleships called {\tt submarine},
each player hides just one submarine in one square of
an eight-by-eight grid.
\Figref{fig.sub} shows a few pictures of this game in progress:
the circle represents the square that is being fired at, and the
$\times$s show squares in which the outcome was a miss, $x={\tt{n}}$; the
submarine is hit (outcome $x={\tt{y}}$ shown by
the symbol $\bs$) on the 49th attempt.
Each shot made by a player defines an ensemble. The
two possible outcomes are $\{ {\tt{y}} ,{\tt{n}}\}$,
corresponding to a hit and a miss, and their probabilities
depend on the state of the board.
At the beginning, $P({\tt{y}}) = \linefrac{1}{64}$ and
$P({\tt{n}}) = \linefrac{63}{64}$.
At the second shot, if the first shot missed,
% enemy sub has not yet been hit,
$P({\tt{y}}) = \linefrac{1}{63}$ and $P({\tt{n}}) = \linefrac{62}{63}$.
At the third shot, if the first two shots missed,
% enemy submarine has not yet been hit,
$P({\tt{y}}) = \linefrac{1}{62}$ and $P({\tt{n}}) = \linefrac{61}{62}$.
% According to the Shannon information content, t
The Shannon information
gained from an outcome $x$ is $h(x) = \log (1/P(x))$.
% Let's investigate this assertion.
If we are lucky, and hit the submarine on the first shot, then
\beq
h(x) = h_{(1)}({\tt y}) = \log_2 64 = 6 \ubits .
\eeq
Now, it might seem a little strange that
one binary outcome can convey six bits.
% , but it does make sense. W
But we have learnt the hiding place,
% where the submarine was,
which
could have been any of 64 squares; so we have, by one lucky
binary question, indeed learnt six bits.
What if the first shot misses? The Shannon information that we gain from this outcome
is
\beq
h(x) = h_{(1)}({\tt n}) = \log_2 \frac{64}{63} = 0.0227 \ubits .
\eeq
Does this make sense? It is not so obvious. Let's keep going.
If our second shot also misses, the Shannon information
content of the second outcome is
\beq
h_{(2)}({\tt n}) = \log_2 \frac{63}{62} = 0.0230 \ubits .
\eeq
If we miss thirty-two times (firing at a new square each time),
the total Shannon information gained is
\beqan
%\hspace*{-0.2in}
\lefteqn{ \log_2 \frac{64}{63} + \log_2 \frac{63}{62} + \cdots +
\log_2 \frac{33}{32} } \nonumber \\
& \!\!\!=\!\!\! & 0.0227 + 0.0230 + \cdots + 0.0430 \:\:=\:\:
1.0 \ubits .
\eeqan
Why this round number? Well, what have we learnt? We now know
that the submarine is not in any of the 32 squares we fired at;
learning that fact is just like playing a game of \sixtythree\
(\pref{example.sixtythree}),
asking as our first question `is $x$ one of the
thirty-two numbers corresponding to these squares I fired at?',
and receiving the answer `no'. This answer rules out half of the
hypotheses, so it gives us one bit.
%It doesn't matter what the
% outcome might have been; all that matters is the probability
% of what actually happened.
After 48 unsuccessful shots, the information
gained is 2 bits: the unknown location has been narrowed down to
one quarter of the original hypothesis space.
What if we hit the submarine on the 49th shot, when there
were 16 squares left?
The Shannon information content of this outcome is
\beq
h_{(49)}({\tt y}) = \log_2 16 = 4.0 \ubits .
\eeq
The total Shannon information content of all the outcomes is
\beqan
\lefteqn{ \log_2 \frac{64}{63} + \log_2 \frac{63}{62} + \cdots +
% \log_2 \frac{33}{32} + \cdots +
\log_2 \frac{17}{16} +
\log_2 \frac{16}{1} }
\nonumber \\
&=& 0.0227 + 0.0230 + \cdots
% + 0.0430 + \cdots
+ 0.0874 + 4.0 \:\: =\:\: 6.0 \ubits .
\label{eq.sum.me}
\eeqan
So once we know where the submarine is, the total Shannon information
content gained is 6 bits.
This result holds regardless of when
we hit the submarine. If we hit it when there are $n$ squares
left to choose from -- $n$ was 16 in
\eqref{eq.sum.me} -- then the total information gained
is:
\beqan
\lefteqn{ \log_2 \frac{64}{63} + \log_2 \frac{63}{62} + \cdots +
\log_2 \frac{n+1}{n} +
\log_2 \frac{n}{1} } \nonumber \\
&=& \log_2 \left[
\frac{64}{63} \times \frac{63}{62} \times \cdots
\times \frac{n+1}{n} \times \frac{n}{1} \right]
%\times 63 \times \cdots \times (n+1) \times n}
% {63 \times 62 \times \cdots \times n \times 1}
\:\:=\:\: \log_2 \frac{64}{1}\:\: =\:\: 6 \,\bits.
\eeqan
%
% add winglish here?
%
% follows in lecture 2, after submarine
%
% aim: introduce the language of Wenglish
% and demonstrate Shannon info content.
What have we learned from the examples so far?
I think the {\tt submarine} example makes quite a convincing
case for the claim that the Shannon information content
is a sensible measure of information content.
And the game of {\tt sixty-three} shows that
the Shannon information content can be intimately connected
to the size of a file that encodes the outcomes of
a random experiment, thus suggesting a possible connection to
data compression.
In case you're not convinced, let's look at one more example.
\subsection{The \Wenglish\ language}
\label{sec.wenglish}
% [this section under construction]}
{\dem{\ind{\Wenglish}}} is a language similar to \ind{English}.
\Wenglish\ sentences consist of words drawn at random from the
\Wenglish\ dictionary, which contains $2^{15}=32$,768 words, all of length 5
characters. Each word in the \Wenglish\ dictionary was constructed
% by the \Wenglish\ language committee, who created each of those $32\,768$ words
at random by picking five letters from the
probability distribution over {\tt a$\ldots$z} depicted
in \figref{fig.monogram}.
% Since all words are five characters long
%\begin{figure}
%\figuremargin{
\marginfig{\small
\begin{center}
\begin{tabular}{rc} \toprule
% & Word \\ \midrule
1 & {\tt{aaail}} \\
2 & {\tt{aaaiu}} \\
3 & {\tt{aaald}} \\
& $\vdots$ \\
129 & {\tt{abati}} \\
& $\vdots$ \\
$2047$ & {\tt{azpan}} \\
$2048$ & {\tt{aztdn}} \\
& $\vdots$ \\
& $\vdots$ \\
$16\,384$ & {\tt{odrcr}} \\
& $\vdots$ \\
& $\vdots$ \\
$32\,737$ & {\tt{zatnt}} \\
& $\vdots$ \\
$32\,768$ & {\tt{zxast}} \\ \bottomrule
\end{tabular}
\end{center}
%}{
\caption[a]{The \Wenglish\ dictionary.}
\label{fig.wenglish}
}
%\end{figure}
% 5366+1219+2602+2718+8377+1785+1280+3058+5903+70+800+3431+2319+5470+6526+1896+539+4660+5453+6767+3108+652+1388+765+1564+78
% 77794
Some entries from the dictionary are shown in
alphabetical order in \figref{fig.wenglish}.
Notice that the number of words in the \ind{dictionary}
(32,768)
is much smaller than the total number of possible words of length 5 letters,
$26^5 \simeq 12$,000,000.
Because the probability of the letter {{\tt{z}}} is about $1/1000$,
only 32 of the words in the dictionary begin with the letter {\tt z}.
In contrast, the probability of the letter {{\tt{a}}} is about $0.0625$,
and 2048 of the words begin with the letter {\tt a}. Of those 2048 words,
two start {\tt az}, and 128 start {\tt aa}.
Let's imagine that we are reading a \Wenglish\ document, and let's discuss
the Shannon \ind{information content} of the characters as we acquire them.
If we are given the text one word at a time, the Shannon information
content of each five-character word is $\log \mbox{32,768} = 15$ bits,
since \Wenglish\ uses all its words with equal probability. The
average information content per character is therefore 3 bits.
Now let's look at the information content if we read the document
one character at a time.
If, say, the first letter of a word is {\tt a}, the Shannon information
content is
$\log 1/ 0.0625 \simeq 4$ bits.
If the first letter is {\tt z}, the Shannon information content
is $\log 1/0.001 \simeq 10$ bits.
The information content is thus highly variable
at the first character. The total information
content of the 5 characters in a word, however,
is exactly 15 bits; so the letters that
follow an initial {\tt{z}} have lower average information content
per character than the letters that follow an initial {\tt{a}}.
A rare initial letter such as {\tt{z}} indeed conveys
more information about what the word is
than a common initial letter.
Similarly, in English, if rare characters occur at the start of
the word (\eg\ {\tt{xyl}\ldots}),
then often we can identify the whole word immediately; whereas
words that start with common characters (\eg\ {\tt{pro}\ldots}) require more characters
before we can identify them.
% Does this make sense? Well, in English,
% the first few characters of a word do very often fully identify the whole word.
%
% {\em MORE HERE........}
\section{Data compression}
\index{data compression}\index{source code}The
preceding examples justify the idea that the Shannon \ind{information
content} of an outcome is a natural measure of its
\ind{information content}. Improbable outcomes
do convey more information than probable outcomes.
We now discuss the information content
of a source by considering how many bits are needed to describe
the outcome of an experiment.
% , that is, by studying {data compression}.
If we can show that we can compress data from a particular source
into a file of $L$ bits per source symbol and recover the data reliably,
then we will say that the average information
content of that source is at most
% less than or equal to
$L$ bits per symbol.
%
% cut Sat 13/1/01
%
% We will show that, for any source, the information content of the source
% is intimately related to its entropy.
\subsection{Example: compression of text files}
A file is composed of a sequence of bytes. A byte is composed of 8
bits\marginpar{\small\raggedright{Here we use the word `bit' with its meaning, `a
symbol with two values', not to be confused with the
unit of information content.}}
and can have a decimal value between 0 and 255. A
typical text file is composed of the
ASCII character set (decimal values 0 to 127).
This character set uses only
seven of the eight bits in a byte.
\exercissxB{1}{ex.ascii}{
By how much could the size of a file be reduced given that
it is an ASCII file? How would you achieve this reduction?
}
Intuitively, it seems reasonable to assert that an ASCII file
contains $7/8$ as much information as an arbitrary file of the same
size, since we already know one out of every eight bits before we even
look at the file.
This is a
% very
simple example of redundancy.
Most sources of data have further redundancy: English text files
use the ASCII characters with non-equal frequency; certain pairs
of letters are more probable than others; and entire words
can be predicted given the context and a semantic understanding
of the text.
% this par is repeated in l4.
% compressibility.
\subsection{Some simple data compression methods that define
measures of information content}
%
% IDEA: connect back to opening
%
One way of measuring the information content of a random variable
is simply to count the number of {\em possible\/} outcomes,
$|\A_X|$. (The number of elements in a set $\A$ is denoted by $|\A|$.)
If we gave a binary name to each outcome, the length
of each name would be $\log_2 |\A_X|$ bits, if $|\A_X|$ happened
to be a power of 2.
We thus make the following definition.
\begin{description}%%%% was: [Perfect information content] Raw bit content
%%%%%%%%%%%%%%%%%%%%%%% see newcommands1.tex
\item[The \perfectic] of $X$ is
\beq
H_0(X) = \log_2 |\A_X| .
\eeq
\end{description}
$H_0(X)$ is a lower bound for
the number of binary questions that are always guaranteed to identify
an outcome from the ensemble $X$.
It is an additive quantity: the \perfectic\ of an ordered pair $x,y$,
having $|\A_X||\A_Y|$
possible outcomes,
satisfies
\beq
H_0(X,Y)= H_0(X) + H_0(Y).
\eeq
This measure of information content does not include any
probabilistic element, and the encoding rule it corresponds to
does not `compress' the source data, it simply maps each
outcome
% source character
to a constant-length binary string.
\exercissxA{2}{ex.compress.possible}{
Could there be a compressor that maps
an outcome $x$ to a binary code $c(x)$, and a decompressor
that maps $c$ back to $x$, such that {\em every
possible outcome\/} is compressed into a binary code
of length {\em shorter\/}
than $H_0(X)$ bits?
}
Even though a simple counting argument\index{compression!of {\em any\/} file}
shows that it is impossible to make a reversible
compression program that reduces the size of {\em all\/} files,
amateur compression enthusiasts frequently announce that they have invented
a program that can do this -- indeed that they can further compress
compressed files by putting them through their compressor several\index{compression!of already-compressed files}\index{myth!compression}
times. Stranger yet, patents have
been granted to these modern-day \ind{alchemists}. See
the {\tt{comp.compression}} frequently asked questions
% \verb+http://www.faqs.org/faqs/compression-faq/part1/+
for further reading.\footnote{\tt{http://sunsite.org.uk/public/usenet/news-faqs/comp.compression/}}
%\footnote{\verb+http://www.lib.ox.ac.uk/internet/news/faq/+}
% ............by_category.compression-faq.html+}
% http://www.faqs.org/faqs/compression-faq/part1/preamble.html
There are only two ways in which a `compressor' can actually
compress files:
\ben
\item
A {\dem lossy\/} compressor compresses some\index{compression!lossy}
files, but maps some files
% {\em distinct\/} files are mapped
to the
{\em same\/} encoding. We'll assume that
the user requires perfect recovery of the source
file, so the occurrence of one of these
confusable files leads to a failure (though in
applications such as \ind{image compression}, lossy compression is viewed as
satisfactory). We'll denote by
$\delta$
the probability that the
source string is one of the confusable files, so a
lossy compressor\index{error probability!in compression}
has a probability $\delta$ of
failure. If $\delta$ can be made very small then
a lossy compressor may be practically useful.
\item
A {\dem lossless} compressor maps all files
to different encodings; if it
% f a lossless compressor
shortens some files,\index{compression!lossless}
it necessarily {\em makes others longer}. We try to design the
compressor so that the probability that a
file is lengthened is very small, and the probability that
it is shortened is large.
\een
In this chapter we discuss a simple lossy compressor.
In subsequent chapters we discuss lossless compression
methods.
%
\section{Information content defined in terms of lossy
compression}
%
Whichever type of compressor we construct, we need somehow to
take into account the {\em probabilities\/} of the different outcomes.
Imagine comparing the information contents of
two text files -- one
in which all 128 ASCII characters are used with equal probability,
and one in which the characters are used with their frequencies
in English text.
%: $P(x={\tt e})=$,
% $P(x={\tt e})=$, $P(x={\tt e})=$,$P(x={\tt e})=$,$P(x={\tt e})=$, \ldots
% $P(x={\tt e})=$, \ldots.
% only the characters {\tt 0} and {\tt 1} are used.
Can we define a measure of information content that
distinguishes between these two files? Intuitively,
the latter file contains less information per character
because it is more predictable.
%And a file of {\tt 0}s
% and {\tt 1}s in which nearly all the characters are {\tt 0}s
% conveys even less information.
% Maybe introducing 0 and 1 is nto a good idea.
% At this point I start talking in terms of compression.
% How can we include a probabilistic element?
One simple way to use
our knowledge that some symbols have a smaller probability is
to imagine recoding the observations into a smaller alphabet -- thus losing
the ability to encode some of the more improbable
symbols -- and then measuring the \perfectic\ of the new alphabet.
% choice here - could either map multiple symbols onto
% one, so the compression is lossy,
% or could define no entry at all for some symbols, so compression
% fails.
% The general mapping situation is not ideal since I really want all
% the losers to be mapped to one symbol. Student might imagine mapping
% Z and z to Z, Y and y to Y.. and claim they are losing little info.
% But this messes up the defn of delta.
For example,
we might take a risk when compressing English text, guessing that the most
infrequent characters won't occur,
and make a reduced ASCII code that omits the characters
% for example,
% `\verb+!+', `\verb+@+', `\verb+#+',
% `\verb+$+', `\verb+%+', `\verb+^+', `\verb+*+', `\verb+~+',
% `\verb+<+', `\verb+>+', `\verb+/+', `\verb+\+', `\verb+_+',
% `\verb+{+', `\verb+}+', `\verb+[+', `\verb+]+',
% and `\verb+|+',
$\{$ \verb+!+, \verb+@+, \verb+#+,
% \verb+$+, $
\verb+%+, \verb+^+, \verb+*+, \verb+~+,
\verb+<+, \verb+>+, \verb+/+, \verb+\+, \verb+_+,
\verb+{+, \verb+}+, \verb+[+, \verb+]+, \verb+|+ $\}$,
thereby reducing the size of the alphabet
% the total number of characters
by seventeen.
%
% cut this dec 2000
% Thus we can give new
%%%% a (not necessarily unique)
% names to a {\em subset\/} of the possible outcomes and count how many names we
% use.
The larger the risk we are willing to take, the smaller
our final alphabet becomes.
% ] the number of names we need.
% We thus relax the exhaustive requirement of the definition of
%
% aside
%
% We could imagine doing this to the numbers coming out of the guessing
% game with which this chapter started, for example. It seems
% quite unlikely that the subject would have to guess 25, 26 or 27 times
% to get the next letter; these outcomes
%%`27' is
% are very improbable,
% and we might be willing to record the sequence of numbers using
% 24 symbols only, taking the gamble that in fact more guesses might
% be needed.
We introduce a parameter $\delta$ that describes the risk we
are taking when using this compression method: $\delta$ is
the probability that there will be no name for an outcome $x$.
\exampl{exHdelta}{
Let
\beq
\begin{array}{l*{14}{@{\,}c}}
& \A_X & = & \{ & {\tt a},& {\tt b},&{\tt c},&{\tt d},&{\tt e},&{\tt f},&{\tt g},&{\tt h} & \}, \\
\mbox{and }\:\:
& \P_X & = & \bigl\{ & \frac{1}{4} ,& \frac{1}{4} ,& \frac{1}{4} ,& \frac{3}{16} ,& \frac{1}{64} ,& \frac{1}{64} ,& \frac{1}{64} ,& \frac{1}{64} & \bigr\} .
\end{array}
\eeq
The \perfectic\ of this ensemble is 3 bits, corresponding to
8 binary names.
But notice that $P( x \in \{ {\tt a}, {\tt b}, {\tt c}, {\tt d} \} ) = 15/16$.
So if we are willing to run a risk of $\delta=1/16$ of not having a name
for $x$, then we can get by with four names --
half as many names as are needed if
every $x \in \A_X$ has a name.
Table \ref{fig.delta.examples} shows binary names that could be given
to the different outcomes in the cases $\delta = 0$ and $\delta = 1/16$.
When $\delta=0$ we need 3 bits to encode the outcome;
when $\delta=1/16$ we need only 2 bits.
%
%\begin{figure}[htbp]
%\figuremargin{%
\amargintab{b}{
\begin{center}
\begin{tabular}{cc}
\toprule
\multicolumn{2}{c}{$\delta = 0$}
\\
\midrule
$x$ & $c(x)$ \\ \midrule
{\tt a} & {\tt{000}} \\
{\tt b} & {\tt{001}} \\
{\tt c} & {\tt{010}} \\
{\tt d} & {\tt{011}} \\
{\tt e} & {\tt{100}} \\
{\tt f} & {\tt{101}} \\
{\tt g} & {\tt{110}} \\
{\tt h} & {\tt{111}} \\
\bottomrule
\end{tabular}
% \hspace{0.61in}
\hspace{0.1in}
\begin{tabular}{cc}
\toprule
\multicolumn{2}{c}{$\delta = 1/16$}
\\
\midrule
$x$ & $c(x)$ \\ \midrule
{\tt a} & {\tt{00}} \\
{\tt b} & {\tt{01}} \\
{\tt c} & {\tt{10}} \\
{\tt d} & {\tt{11}} \\
{\tt e} & $-$ \\
{\tt f} & $-$ \\
{\tt g} & $-$ \\
{\tt h} & $-$ \\
\bottomrule
\end{tabular}
\end{center}
%}{%
\caption[a]{Binary names for the outcomes,
for two failure probabilities $\delta$.}
\label{fig.delta.examples}
\label{tab.twosillycodes}
}%
%\end{figure}
}
%\noindent
Let us now formalize this idea.\index{source code}
%
To make a compression strategy with risk $\delta$,
% we consider all subsets $T$ of the alphabet $\A_X$ and
% seek out
we make the smallest possible subset
$S_{\delta}$ such that the
probability that $x$ is not in $S_{\delta}$ is less than or equal to
$\delta$, \ie,
$P(x \not\in S_{\delta} ) \leq \delta$. For each value of $\delta$ we can then
define a new measure of information content -- the log of the size
of this smallest subset $S_{\delta}$. [In ensembles in which
several elements have the same probability, there may be several
smallest subsets that contain different elements, but all that matters
is their sizes (which are equal), so we will not dwell on this ambiguity.]
% worry about this possibility.
\begin{description}
\item[The smallest $\delta$-sufficient subset] $S_{\delta}$ is the smallest
subset of $\A_X$ satisfying
\beq
P(x \in S_{\delta} ) \geq 1 - \delta.
\eeq
%\beq
% S_{\delta} = \argmin
%\eeq
\end{description}
The subset $S_{\delta}$ can be constructed by
ranking the elements of $\A_X$ in order of decreasing probability
and adding successive elements starting from the
most probable elements
% front of the list
until the total
probability is $\geq (1\!-\!\delta)$.
We can make a data compression code by assigning a binary name
to each element of the smallest sufficient subset.
% (\tabref{tab.twosillycodes}).
This compression
scheme motivates the following measure of information content:
\begin{description}
\item[The \essentialic] of $X$ is: %%%%% was ESSENTIAL information content
% consider risk-delta bit content?
\beq
H_{\delta}(X) = \log_2 |S_{\delta}| .
% = \log_2 \min \left\{ |S| : S\subseteq \A_X,
%% P(S)\geq 1-\delta \right\}.
% P(x \in S)\geq 1-\delta \right\}.
\eeq
\end{description}
Note that $H_0(X)$ is the special case of $H_{\delta}(X)$ with $\delta = 0$
(if $P(x) > 0$ for all $x \in \A_X$).
%
[{\sf Caution:} do not confuse $H_0(X)$ and $H_{\delta}(X)$
with the function $H_2(p)$ displayed in \figref{fig.h2}.]
%%%%%%%(Should I change notation to avoid confusion?)
%
\newcommand{\gapline}{\cline{1-4}\cline{6-9}}
\begin{figure}
\figuremargin{%
\begin{center}
\footnotesize%
\begin{tabular}{rc}
(a)&
\hspace*{-0.2in}\input{Hdelta/Sdelta/X.tex}\\
(b)&
\mbox{\makebox[0in][r]{\raisebox{1.3in}{$H_{\delta}(X)$}}\hspace{-5mm}%
\psfig{figure=Hdelta/byhand/X.ps,%
width=70mm,angle=-90}$\delta$}%
\\
\end{tabular}
\end{center}
}{%
\caption[a]{(a) The outcomes of $X$ (from \protect\exampleref{exHdelta}),
ranked by their probability.
(b) The
\essentialic\ $H_{\delta}(X)$. The labels on the graph
show the smallest sufficient set as a function of $\delta$.
Note $H_0(X) = 3$ bits and $H_{1/16}(X) = 2$ bits.
}
\label{fig.hd.1}
}
\end{figure}
%\noindent
{\Figref{fig.hd.1} shows $H_{\delta}(X)$ for the ensemble
of \exampleonlyref{exHdelta} as a function of $\delta$.
}
\subsection{Extended ensembles}
% The compression method we're studying in which a subset of
% outcomes are given binary names is not giving us a
% measure of information content for a single symbol.
%
% sanjoy wants a motivation here.
%
Is this compression method any more useful if we compress
{\em blocks\/} of symbols from a source?\index{source code!block code}\index{ensemble!extended}\index{extended ensemble}
%
We now turn to examples where the outcome $\bx = (x_1,x_2,\ldots, x_N)$ is a string of $N$
independent identically distributed random variables
from a single ensemble $X$.
We will denote by
% $\bX$ or
$X^N$ the ensemble $( X_1, X_2, \ldots, X_N )$.
% for which $\bx$ is the random variable.
Remember that entropy is additive for independent variables (\exerciseref{ex.Hadditive}),
% \footnote{There should have been an exercise on this by now.}
so
% $H(\bX) = N H(X)$.
$H(X^N) = N H(X)$.
\exampl{ex.Nfrom.1}{
% {\sf Example 2:}
Consider a string of $N$ flips of a bent coin,
$\bx = (x_1,x_2,\ldots, x_N)$, where $x_n \in
\{{\tt{0}},{\tt{1}}\}$, with probabilities $p_0 \eq 0.9,$ $p_1 \eq
0.1$. The most probable strings $\bx$ are those with most {\tt{0}}s. If
$r(\bx)$ is the number of {\tt{1}}s in $\bx$ then
\beq
% |p_0,p_1
P(\bx) = p_0^{N-r(\bx)} p_1^{r(\bx)} .
\eeq
To evaluate $H_{\delta}(X^N)$
we must find the smallest sufficient subset $S_{\delta}$.
This subset will contain
all $\bx$ with $r(\bx) = 0, 1, 2, \ldots,$ up to some $r_{\max}(\delta)-1$,
and some of the $\bx$ with $r(\bx) = r_{\max}(\delta)$.
% Working backwards, we can evaluate the cumulative probability
% $P(r(\bx) \leq r)$ and evaluate the size of the subset $T(r): \{ \bx:
% r(\bx) \leq r \}$.
%\beq
% |T(r)| = \sum_{r=0}^{r} \frac{N!}{(N-r)!r!}
%\label{l2.T}
%\eeq
%\beq
% P(r(\bx) \leq r) = \sum_{r=0}^{r} \frac{N!}{(N-r)!r!} p_0^{N-r} p_1^{r}
%\label{l2.Pr}
%\eeq
% We can then plot $\log |T(r)|$ versus $P(r(\bx) \leq r)$. This defines
% a graph of $H_{\delta}(\bX)$ against $\delta$.
Figures \ref{fig.hd.4} and \ref{fig.hd.10}
% Figure \ref{fig.hd.4}
show graphs of $H_{\delta}(X^N)$ against
$\delta$ for the cases $N=4$ and $N=10$. The steps are the values of
$\delta$ at which $|S_{\delta}|$ changes by~1, and the cusps where the slope
of the staircase changes are the points
where $r_{\max}$ changes by 1.
}
\exercissxC{2}{ex.cusps}{
What are the mathematical shapes of the curves between the cusps?
}
% , both with $p_1 =
% 0.1$. The points defined by equations (\ref{l2.T}) and (\ref{l2.Pr})
% are the cusps in the curve.
%
% I think this figure may be sick. CHECK IT.
%
\renewcommand{\gapline}{\cline{1-3}\cline{5-8}}
\begin{figure}
\figuremargin{%
%
% this table done by hand with help of (above hd.p command) /home/mackay/itp/Hdelta> more figs/4.tex
%
\begin{center}
\footnotesize%
\begin{tabular}{r@{\hspace*{-0.3in}}c}
(a)&
%%%%%%%% written by hand see also X.tex
%
% picture of Sdelta for X^4
%
\newcommand{\axislevel}{24}
\newcommand{\axislevelp}{29.5}
\newcommand{\axislevelm}{21}
\newcommand{\axislevelmm}{18}
\newcommand{\forestgap}{-0.7}
\newcommand{\forest}[3]{\multiput(#1)(\forestgap,0){#2}{\line(0,1){#3}}}
%
%
%
\setlength{\unitlength}{2.2pt}%
\begin{picture}(155,50)(-143,-20)% adjusted vertical height from 50 to 60 Sat 5/10/02. And put back again Sun 22/12/02 was (-143,-22) Sun 22/12/02
% - log P = 2.0 , 2.4 and 6.0
\forest{-6.1,0}{1}{16}% heights fictitious
\forest{-37.3,0}{4}{12.5}%
\forest{-68.5,0}{6}{9.4}% 69.5
\forest{-100.8,0}{4}{6.3}%
\forest{-132.9,0}{1}{4.2}%
% axis:
\put(-143,\axislevelm){\vector(1,0){151.0}}
%
% axis labels
\put(5,\axislevelp){\makebox(0,0)[b]{\small$\log_2 P(x)$}}
\put(0,\axislevel){\makebox(0,0)[b]{\small$0$}}
\put(-20,\axislevel){\makebox(0,0)[b]{\small$-2$}}
\put(-40,\axislevel){\makebox(0,0)[b]{\small$-4$}}
\put(-60,\axislevel){\makebox(0,0)[b]{\small$-6$}}
\put(-80,\axislevel){\makebox(0,0)[b]{\small$-8$}}
\put(-100,\axislevel){\makebox(0,0)[b]{\small$-10$}}
\put(-120,\axislevel){\makebox(0,0)[b]{\small$-12$}}
\put(-140,\axislevel){\makebox(0,0)[b]{\small$-14$}}
%
% this box is right size for the whole set
%\put(0,-2.5){\framebox(140,\axislevelm){}}
%\put(142,13){\makebox(0,0)[l]{\small$S_0$}}
% this box is round 3 clumps
\put(-83.5,-2.5){\framebox(83.5,\axislevelm){}}
\put(-84.5,13){\makebox(0,0)[r]{\small$S_{0.01}$}}
% a smaller box round 3 clumps
%\put(2.5,-1){\framebox(81,\axislevelmm){}}
%
\put(-53.5,-1){\framebox(51,\axislevelmm){}}
\put(-54.5,13){\makebox(0,0)[r]{\small$S_{0.1}$}}
%
% object labels
\put(-6.1,-12){\makebox(0,0)[t]{\footnotesize{\tt 0000}}}
\put(-37.7,-12){\makebox(0,0)[t]{\footnotesize${\tt 0010},{\tt 0001},\ldots$}}
\put(-69.5,-12){\makebox(0,0)[t]{\footnotesize${\tt 0110},{\tt 1010},\ldots$}}
\put(-101.2,-12){\makebox(0,0)[t]{\footnotesize${\tt 1101},{\tt 1011},\ldots$}}
\put(-132.9,-12){\makebox(0,0)[t]{\footnotesize{\tt 1111}}}
\multiput(-6.1,-10)(-31.6,0){5}{\vector(0,1){5}}
\end{picture}
%
%
%
%
(b)&
\makebox[0in][r]{\raisebox{1.3in}{$H_{\delta}(X^4)$}}\hspace{-5mm}%
\psfig{figure=Hdelta/figs/hd/4.ps,%
width=65mm,angle=-90}$\delta$%%
%
%
% useful for making table:
% hd.p mmin=4 mmax=4 mstep=6 scale_by_n=0 plot_sub_graphs=1 latex=1
%
\end{tabular}
\end{center}
}{%
%
% I think this figure may be sick. CHECK IT.
%
\caption[a]{(a) The sixteen outcomes of the ensemble $X^4$ with $p_1=0.1$, ranked by probability. (b) The
\essentialic\ $H_{\delta}(X^4)$. The upper
schematic diagram indicates the strings'
probabilities by the vertical lines' lengths (not to scale).}
\label{fig.hd.4}
}%
\end{figure}
%
%
%
\begin{figure}%[htbp]
\figuremargin{%
\begin{center}
\mbox{%%%%%%%%%%%%% (twocol) %}\\ \mbox{
\makebox[0in][r]{\raisebox{1.3in}{$H_{\delta}(X^{10})$}}\hspace{-5mm}%
\psfig{figure=Hdelta/figs/hd/10.ps,%
width=65mm,angle=-90}$\delta$}
% command, in Hdelta:
% hd.p mmin=4 mmax=10 mstep=6 scale_by_n=0 plot_sub_graphs=1 | gnuplot
\end{center}
}{%
\caption[a]{$H_{\delta}(X^N)$ for $N=10$ binary variables with $p_1=0.1$.}
\label{fig.hd.10}
}%
\end{figure}
For the examples shown in figures \ref{fig.hd.1}--\ref{fig.hd.10},
$H_{\delta}(X^N)$ depends strongly on the
value of $\delta$, so it might not seem a fundamental or useful
definition of information content.
But we will consider what happens as $N$, the number of independent variables
in $X^N$, increases. We will find the remarkable result that
$H_{\delta}(X^N)$ becomes almost independent of $\delta$ -- and for all
$\delta$ it is very close to $N H(X)$, where $H(X)$ is the
entropy of one of the random variables.
% sketch?
\begin{figure}
\figuremargin{%
\begin{center}
\mbox{\makebox[0in][r]{\raisebox{1.3in}{$\frac{1}{N}H_{\delta}(X^{N})$}}\hspace{-5mm}%
\psfig{figure=Hdelta/figs/hd/all.10.1010.ps,%
width=65mm,angle=-90}$\delta$}
\end{center}
}{%
\caption[a]{$\frac{1}{N} H_{\delta}(X^{N})$
for $N=10, 210, \dots,1010$ binary variables with $p_1=0.1$.}
\label{fig.hd.10.1010}
}
\end{figure}
\Figref{fig.hd.10.1010} illustrates this asymptotic tendency for
the binary ensemble of example \ref{ex.Nfrom.1}.
% discussed earlier with $N$ binary variables with $p_1 = 0.1$.
As $N$ increases, $\frac{1}{N} H_{\delta}(X^N)$ becomes an increasingly
flat function, except for tails close to $\delta=0$ and $1$.
% The limiting value of the plateau is $H(X) = 0.47$.
% We will explain and prove this result in the remainder of
% this chapter. Let's first note the implications of this result.
% The limiting value of the plateau, which for $N$ binary variables with $p_1 = 0.1$
% appears to be about 0.5, defines how much compression is possible:
% $N$ binary variables with $p_1 = 0.1$ can be compressed into
% about $N/2$ bits, with a probability of error $\delta$ which
% can be any value between 0 and 1.
% We will show that the plateau value to which $\frac{1}{N} H_{\delta}(X^N)$
% tends, for large $N$, is the entropy, $H(X)$.
%
% IDEA: Box this next sentence?
%
As long as we are allowed
a tiny probability of error $\delta$, compression down to
$NH$ bits is possible. Even if we are allowed a large probability of error,
we still can compress only down to $NH$ bits.
%
% IDEA: Box above?
%
This is the \ind{source coding theorem}.
% \subsection{The theorem}
\begin{ctheorem}
\label{thm.sct}
{\sf Shannon's source coding theorem.}
% HOW TO NAME THIS?????????????????
% this name is taken later
Let $X$ be an ensemble with entropy $H(X) = H$ bits. Given $\epsilon>0$
and $0<\delta<1$, there exists a positive integer $N_0$ such that for
$N>N_0$,
\beq
\left| \frac{1}{N} H_{\delta}(X^N) - H \right| < \epsilon.
\eeq
\end{ctheorem}
%
% sanjoy wants explan here
%
% The reason that increasing $N$ helps is that, if $N$ is large,
% the outcome $\bx$
\section{Typicality}
Why does increasing $N$ help?\indexs{typicality}
Let's examine long strings from $X^N$.
Table \ref{tab.typical.tcl} shows fifteen samples from $X^N$
for $N=100$ and $p_1=0.1$.
\begin{figure}
\figuremargin{%
\begin{center}
\begin{tabular}{lr} \toprule
$\bx$ &
% \multicolumn{1}{c}{$\log_2(P(\bx))$}
\hspace{-0.3in}{$\log_2(P(\bx))$}
% {\rule[-3mm]{0pt}{8mm}}%strut
\\ \midrule
% REQUIRE MONOSPACED FONT!!!
{\tinytt{%VERB
...1...................1.....1....1.1.......1........1...........1.....................1.......11...%END
}} & $-$50.1 \\
{\tinytt{%VERB
......................1.....1.....1.......1....1.........1.....................................1....%END
}} & $-$37.3 \\
{\tinytt{%VERB
........1....1..1...1....11..1.1.........11.........................1...1.1..1...1................1.%END
}} & $-$65.9 \\
{\tinytt{%VERB
1.1...1................1.......................11.1..1............................1.....1..1.11.....%END
}} & $-$56.4 \\
{\tinytt{%VERB
...11...........1...1.....1.1......1..........1....1...1.....1............1.........................%END
}} & $-$53.2 \\
{\tinytt{%VERB
..............1......1.........1.1.......1..........1............1...1......................1.......%END
}} & $-$43.7 \\
{\tinytt{%VERB
.....1........1.......1...1............1............1...........1......1..11........................%END
}} & $-$46.8 \\
{\tinytt{%VERB
.....1..1..1...............111...................1...............1.........1.1...1...1.............1%END
}} & $-$56.4 \\
{\tinytt{%VERB
.........1..........1.....1......1..........1....1..............................................1...%END
}} & $-$37.3 \\
{\tinytt{%VERB
......1........................1..............1.....1..1.1.1..1...................................1.%END
}} & $-$43.7 \\
{\tinytt{%VERB
1.......................1..........1...1...................1....1....1........1..11..1.1...1........%END
}} & $-$56.4 \\
{\tinytt{%VERB
...........11.1.........1................1......1.....................1.............................%END
}} & $-$37.3 \\
{\tinytt{%VERB
.1..........1...1.1.............1.......11...........1.1...1..............1.............11..........%END
}} & $-$56.4 \\
{\tinytt{%VERB
......1...1..1.....1..11.1.1.1...1.....................1............1.............1..1..............%END
}} & $-$59.5 \\
{\tinytt{%VERB
............11.1......1....1..1............................1.......1..............1.......1.........%END
}} & $-$46.8 \\ \midrule % [0.2in]
%
{\tinytt{%VERB
....................................................................................................%END
}} & $-$15.2 \\
{\tinytt{%VERB
1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111%END
}} & $-$332.1\\
%
\bottomrule
\end{tabular}
\end{center}
}{%
\caption[a]{The top 15 strings are samples from $X^{100}$,
where $p_1 = 0.1$ and $p_0 = 0.9$.
The bottom two are the most and least probable strings in this ensemble.
The final column shows the
% Compare the
log-probabilities of the random strings,
which may be compared with the entropy
% with
% the \aep: $H(X) = 0.469$, so
$H(X^{100}) = 46.9$ bits.}
\label{tab.typical.tcl}
}
\end{figure}
% 1000 Typical set size +/- 28.46 has log_2(P(x)) within +/- 90.22
% i.e. 1/N (logp) is within 0.090
% 100 Typical set size +/- 9 has log_2(P(x)) within +/- 28.53
% i.e. 1/N(logp) is within 0.285
% 200 Typical set size +/- 12.73 has log_2(P(x)) within +/- 40.35
%
% N=100 alternative (see hd.p for the commands)
%
\begin{figure}
\fullwidthfigureright{
%\figuremargin{%
\begin{center}
\begin{tabular}{r@{\hspace*{-0in}}c@{\hspace*{-0.1in}}c} \toprule
& $N=100$ & $N=1000$ \\ \midrule
\raisebox{0.71in}{\small$n(r) = {N \choose r}$}
& \mbox{\psfig{figure=Hdelta/figs/num/100.ps,%
width=50mm,angle=-90}}
& \mbox{\psfig{figure=Hdelta/figs/num/1000.ps,%
width=50mm,angle=-90}} \\
\raisebox{0.71in}{\small$P(\bx) = p_1^r (1-p_1)^{N-r}$}
& \mbox{\psfig{figure=Hdelta/figs/per/100.ps,%
width=50mm,angle=-90}}%
\makebox[0in][r]{\raisebox{0.4in}{%
\psfig{figure=Hdelta/figs/perdet/100.ps,%
width=30mm,angle=-90}}\hspace{0.2in}}
&
\\
\raisebox{0.71in}{\small$\log_2 P(\bx)$}
& \mbox{\psfig{figure=Hdelta/figs/logper/100.ps,%
width=50mm,angle=-90}}
& \mbox{\psfig{figure=Hdelta/figs/logper/1000.ps,%
width=50mm,angle=-90}} \\
\raisebox{0.71in}{\small$n(r)P(\bx)= {N \choose r} p_1^r (1-p_1)^{N-r}$}
& \mbox{\psfig{figure=Hdelta/figs/tot/100.ps,%
width=50mm,angle=-90}}
& \mbox{\psfig{figure=Hdelta/figs/tot/1000.ps,%
width=50mm,angle=-90}}
% \makebox[0in][l]{$r$}
\\
&
$r$ & $r$ \\ \bottomrule
\end{tabular}
\end{center}
}{%
\caption[a]{Anatomy of the typical set $T$.
For $p_1=0.1$
and $N=100$ and $N=1000$, these graphs show $n(r)$, the number of
strings containing $r$ {\tt{1}}s; the probability $P(\bx)$ of a single
string that contains $r$ {\tt{1}}s; the same probability on a
log scale; and the total probability
$n(r)P(\bx)$ of all strings that contain $r$ {\tt{1}}s.
The number $r$ is on the horizontal axis.
The plot of $\log_2 P(\bx)$ also shows by a dotted line the mean value of
$\log_2 P(\bx) = -N H_2(p_1)$ which equals $-46.9$
when $N=100$ and $-469$ when $N=1000$. The typical set includes
only the strings that have $\log_2 P(\bx)$ close to this value.
The range marked {\sf T} shows the set $T_{N \beta}$ (as defined
in \protect\sectionref{sec.ts})
for $N=100$ and $\beta = 0.29$ (left) and $N=1000$, $\beta = 0.09$ (right).
}
\label{fig.num.per.tot}
}%
\end{figure}
The probability of a string $\bx$ that contains $r$ {\tt{1}}s and
$N\!-\!r$ {\tt{0}}s is
\beq
P(\bx) = p_1^r (1-p_1)^{N-r} .
\eeq
The number of strings that contain $r$ {\tt{1}}s is
\beq
n(r) = {N \choose r} .
\eeq
So the number of {\tt{1}}s, $r$, has a binomial distribution:
\beq
P(r) = {N \choose r} p_1^r (1-p_1)^{N-r} .
\eeq
These functions are shown in \figref{fig.num.per.tot}.
The mean of $r$ is $N p_1$, and its standard deviation is
$\sqrt{N p_1 (1-p_1)}$ (\pref{sec.first.binomial}).
If $N$ is 100 then
\beq
r \sim N p_1 \pm \sqrt{N p_1 (1-p_1)} \simeq 10 \pm 3 .
\eeq
If $N=1000$ then
\beq
r \sim 100 \pm 10 .
\eeq
Notice that as $N$ gets bigger, the probability distribution
of $r$ becomes more concentrated, in the sense that
while the
range of possible values of $r$ grows
as $N$, the standard deviation of $r$
grows only as $\sqrt{N}$.
That $r$ is most likely to fall
in a small range of values implies
that the outcome $\bx$ is also most likely to
fall in a corresponding small subset of outcomes
that we will call the {{\dbf\inds{typical set}}}.
\subsection{Definition of the typical set}
\label{sec.ts}
% Let us generalize our discussion to an arbitrary ensemble $X$
% with alphabet $\A_X$
% and define typicality.
Let us define \ind{typicality}\index{typical set!for compression}
for an arbitrary ensemble $X$
with alphabet $\A_X$.
Our definition of a typical string will
involve the string's probability.
A long string
% message
of $N$ symbols will usually
contain
% with high probability
about $p_1N$ occurrences of the first symbol,
$p_2N$ occurrences of the second, etc. Hence the probability
of this string
% long message
is roughly
\beq
P(\bx)_{\rm typ}
= P(x_1)P(x_2)P(x_3) \ldots P(x_N)
\simeq p_1^{(p_1N)} p_2^{(p_2N)} \ldots p_I^{(p_IN)}
\eeq
% p_i^{p_iN}
so that
the information content of a typical string is
\beq
\log_2 \frac{1}{P(\bx)}
\simeq N \sum_i p_i \log_2 \frac{1}{p_i} \simeq N H .
\eeq
So the random variable $\log_2 \!\dfrac{1}{P(\bx)}$,
% So the random variable $\frac{1}{N} \log_2 \frac{1}{P(\bx)}$,
% which is the average information content per symbol, is
which is the information content of $\bx$, is
very likely to be close in value to $N H$.
We build our definition of typicality on this observation.
We define the typical elements of $\A_X^N$ to be
those elements that
have probability close to $2^{-NH}$. (Note that the typical set,
unlike the
% best subset for compression
smallest sufficient subset, does
{\em not\/} include the most probable elements of $\A_X^N$, but we
will show that these most probable elements
contribute negligible probability.)
We introduce a parameter $\beta$ that defines how close
the probability has to be to $2^{-NH}$ for
an element to be `typical'.
% $\beta$-
We call the set of typical elements the typical set,
% $T$, or, to be more precise,
$T_{N \beta}$:
% , where the parameter $\beta$
%% controls the breadth of the typical set by defining
% defines what we mean by a probability `close' to $2^{-NH}$:
\beq
T_{N\b} \equiv \left\{ \bx\in\A_X^N :
\left| \frac{1}{N} \log_2 \frac{1}{P(\bx)} - H \right| < \b
\right\} .
\label{eq.TNb}
\eeq
%
% check whether < has propagated to all necessary places
%
We will show that whatever value of $\beta$ we choose,
the typical set contains almost all the probability
as $N$ increases.
This important result is sometimes called the
{\dem `asymptotic equipartition' principle}.\index{asymptotic equipartition}
% \newpage
%\section{`Asymptotic Equipartition' and Source Coding}
\label{sec.aep}
% We will prove the following result:
\begin{description}
\item[`Asymptotic equipartition' principle\puncspace]
% (AEP).]
For an ensemble of $N$ independent identically distributed (\ind{i.i.d.})
random variables
$X^N \equiv ( X_1, X_2, \ldots, X_N )$, with $N$ sufficiently large,
the outcome $\bx = (x_1,x_2,\ldots, x_N)$ is almost certain to belong
to a subset of $\A_X^N$ having only $2^{N H(X)}$ members, each having
probability `close to' $2^{-N H(X)}$.
\end{description}
Notice that if $H(X) < H_0(X)$ then $2^{N H(X)}$ is a {\em tiny\/}
fraction of the number of possible outcomes $|\A_X^N|=|\A_X|^N=2^{N
H_0(X)}.$
\begin{aside}
The term \ind{equipartition} is chosen to describe the idea
that the members of the typical set have {\em roughly equal\/}
probability. [This should not be taken too literally, hence my
use of quotes around `asymptotic equipartition';
% in the phrase \aep;
see page \pageref{sec.aep.caveat}.]
A second meaning for equipartition, in thermal physics,
is the idea that each degree of freedom of a classical system
has {equal\/} average energy, $\half kT$. This second meaning
is not intended here.
\end{aside}
%
The \aep\ is equivalent to:
\begin{description}
\item[Shannon's source coding theorem (verbal statement)\puncspace]
$N$ i.i.d.\ random variables each
with entropy $H(X)$ can be compressed into more than $NH(X)$ bits with
negligible risk of information loss, as $N\rightarrow \infty$;
conversely if they are compressed into fewer than $NH(X)$ bits
it is virtually certain that information will be lost.
\end{description}
These two theorems are equivalent
because we can define a compression algorithm that gives a distinct
name of length $N H(X)$ bits to each $\bx$ in the typical set.
% probable subset.
% as follows:
% enumerate the $\bx$ belonging to
% the subset of $2^{N H(X)}$ equiprobable outcomes as 000\ldots000,
% 000\ldots001, etc.
\begin{figure}
\figuredangle{%
\begin{center}
%%%%%%%% written by hand see also X.tex
%
% picture of Sdelta for X^100
%
\newcommand{\axislevel}{27}
\newcommand{\axislevelp}{32.5}
\newcommand{\axislevelm}{24}
\newcommand{\axislevelmm}{21}
\newcommand{\forestgap}{-0.4}
\newcommand{\forestgab}{-0.6}
\newcommand{\forestgac}{-0.56}
\newcommand{\forestgad}{-0.52}
\newcommand{\forestgae}{-0.48}
\newcommand{\forestgaf}{-0.44}
% \newcommand{\forestgag}{0.48}
%\newcommand{\forestgap}{0.35} was .35 when I went up to 14.
\newcommand{\forest}[3]{\multiput(#1)(\forestgap,0){#2}{\line(0,1){#3}}}
\newcommand{\foresb}[4]{\multiput(#1)(#4,0){#2}{\line(0,1){#3}}}
%
% picture
%
%\setlength{\unitlength}{2.45pt}%
\setlength{\unitlength}{2.87pt}%
\begin{picture}(170,81)(-170,-42)
\forest{0,0}{1}{16.5}%
\foresb{-5,0}{2}{16}{\forestgab}
\foresb{-10,0}{3}{15.5}{\forestgab}
\foresb{-15,0}{4}{15}{\forestgac}
\foresb{-20,0}{5}{14.5}{\forestgad}
\foresb{-25,0}{6}{14}{\forestgae}
\foresb{-30,0}{7}{13.5}{\forestgaf}
\foresb{-35,0}{8}{13}{\forestgap}
\foresb{-40,0}{9}{12.5}{\forestgap}
\forest{-45,0}{10}{12}%
\forest{-50,0}{11}{11.5}%
\forest{-55,0}{12}{11}%
\forest{-60,0}{12}{10.5}%
\forest{-65,0}{12}{10}%
\forest{-70,0}{12}{9.5}%
\forest{-75,0}{12}{9}%
\forest{-80,0}{12}{8.5}%
\forest{-85,0}{12}{8}%
\forest{-90,0}{12}{7.5}%
\forest{-95,0}{12}{7}%
\forest{-100,0}{12}{6.5}%
\forest{-105,0}{12}{6}%
\forest{-110,0}{12}{5.5}%
\forest{-115,0}{11}{5}%
\forest{-120,0}{10}{4.5}%
\foresb{-125,0}{9}{4.2}{\forestgap}
\foresb{-130,0}{8}{3.9}{\forestgap}
\foresb{-135,0}{7}{3.6}{\forestgaf}
\foresb{-140,0}{6}{3.3}{\forestgae}
\foresb{-145,0}{5}{3.0}{\forestgad}
\foresb{-150,0}{4}{2.7}{\forestgac}
\foresb{-155,0}{3}{2.4}{\forestgab}
\foresb{-160,0}{2}{2.1}{\forestgab}
\forest{-165,0}{1}{1.8}%
%
% axis:
\put(-168,\axislevelm){\vector(1,0){171.0}}
%
% axis labels
\put(0,\axislevelp){\makebox(0,0)[br]{\small$\log_2 P(x)$}}
\put(-42.4,\axislevel){\makebox(0,0)[b]{\small$-NH(X)$}}
% tic mark (was at -40 until Tue 8/1/02)
\put(-42.4,\axislevelm){\line(0,1){2}}
% the S0 box
%\put(-3,-2.5){\framebox(172,\axislevelm){}}
%\put(142,16){\makebox(0,0)[l]{$S_0$}}
%
%
% typical set box
\put(-49.5,-1){\framebox(15,\axislevelmm){}}
\put(-51,16){\makebox(0,0)[r]{$T_{N\b}$}}
%
% object labels
\put(0,-40){\vector(0,1){35}}
\put(-15,-35){\vector(0,1){30}}
%\put(26,-30){\vector(0,1){25}}
\put(-36,-25){\vector(0,1){20}}
\put(-46,-20){\vector(0,1){15}}
%\put(56,-15){\vector(0,1){10}}
\put(-155,-10){\vector(0,1){5}}
\put( 0,-40){\makebox(0,0)[tr]{\footnotesize{{\tt 0000000000000}\ldots{\tt{00000000000}}}}}
\put(-15,-35){\makebox(0,0)[tr]{\footnotesize{{\tt 0001000000000}\ldots{\tt{00000000000}}}}}
%\put(26,-30){\makebox(0,0)[tl]{\footnotesize{{\tt 0000001000000}\ldots{\tt{00000010000}}}}}
\put(-36,-25){\makebox(0,0)[tr]{\footnotesize{{\tt 0100000001000}\ldots{\tt{00010000000}}}}}
\put(-46,-20){\makebox(0,0)[tr]{\footnotesize{{\tt 0000100000010}\ldots{\tt{00001000010}}}}}
%\put(56,-15){\makebox(0,0)[tl]{\footnotesize{{\tt 0100001000100}\ldots{\tt{00010100100}}}}}
\put(-155,-10){\makebox(0,0)[tl]{\footnotesize{{\tt 1111111111110}\ldots{\tt{11111110111}}}}}
\end{picture}
%
%
%
%
\end{center}
}{%
\caption[a]{Schematic diagram showing all strings
in the ensemble $X^{N}$
% with $p_0 = 0.9, p_1=0.1$
% of large length $N$
ranked by their probability, and
the typical set $T_{N\b}$.}
\label{fig.typical.set.explain}
}%
\end{figure}
\section{Proofs}
\label{sec.chtwoproof}
This section may be skipped if found tough going.
\subsection{The law of large numbers}
Our proof of the source coding theorem uses the
\ind{law of large numbers}.
\begin{description}
% \item[A random variable $u$] is any real function of $x$,
\item[Mean and variance] of a real random variable
%\footnote
are $\Exp[u] = \bar{u} = \sum_u P(u) u$ and $\var(u) =
\sigma^2_u = \Exp[(u-\bar{u})^2] = \sum_u P(u) (u - \bar{u})^2.$
\begin{aside}
Technical note:
strictly I am assuming here that $u$ is a function $u(x)$ of a
sample $x$ from a finite discrete ensemble $X$. Then the
summations $\sum_u P(u) f(u)$ should be written $\sum_x P(x)
f(u(x))$. This means that $P(u)$ is a finite sum of delta
functions. This restriction guarantees that the mean and
variance of $u$ do exist, which is not necessarily the case for general
$P(u)$.
\end{aside}
\item[Chebyshev's inequality 1\puncspace]
Let $t$ be a non-negative real random variable, and\index{Chebyshev inequality}
let $\a$ be a positive real number. Then\index{inequality}
\beq
P(t \geq \a) \:\leq\: \frac{\bar{t}}{\a}.
\label{eq.cheb.1}
\eeq
{\sf Proof:} $P(t \geq \a) = \sum_{t \geq \a} P(t)$.
We multiply each
term by $t/\a \geq 1$ and obtain:
$P(t \geq \a) \leq \sum_{t \geq \a} P(t) t/\a.$
We add the (non-negative) missing terms and obtain:
$P(t \geq \a) \leq \sum_{t} P(t) t/\a = \bar{t}/\a$. \hfill$\epfsymbol$\par
\item[Chebyshev's inequality 2\puncspace]
Let $x$ be a random variable, and let $\a$ be a positive real number.
Then
\beq
P\left( (x-\bar{x})^2 \geq \a \right) \:\leq\: \sigma^2_x / \a.
\eeq
{\sf Proof:} Take $t = (x-\bar{x})^2$ and apply the previous proposition. \hfill$\epfsymbol$\par
\item[Weak \ind{law of large numbers}\puncspace]
Take $x$ to be the average of $N$ independent random variables
$h_1, \ldots , h_N$, having common mean $\bar{h}$ and common variance
$\sigma^2_h$: $x = \frac{1}{N} \sum_{n=1}^N h_n$. Then
\beq
P( (x-\bar{h})^2 \geq \a ) \leq \sigma^2_h/\a N.
\eeq
{\sf Proof:} obtained by showing that $\bar{x}=\bar{h}$ and that
$\sigma^2_x = \sigma^2_h/ N$. \hfill$\epfsymbol$\par
\end{description}
We are interested in $x$ being very close to the mean ($\a$ very small).
No matter how large $\sigma^2_h$ is, and no matter how small the
required $\a$ is, and no matter how small the desired probability that
$(x-\bar{h})^2 \geq \a$, we can always achieve it by
taking $N$ large enough.
\subsection{Proof of theorem \protect\ref{thm.sct} (\pref{thm.sct})}
% the source coding theorem}
% or could say theorem 1
We apply the law of large numbers to the random variable $\frac{1}{N}
\log_2 \frac{1}{P(\bx)}$ defined for $\bx$ drawn from the ensemble $X^N$.
This random variable can be written as the average of $N$ information
contents
$h_n = \log_2 ( 1 / P(x_n))$, each of which is a random variable with
mean $H = H(X)$ and variance $\sigma^2 \equiv \var[ \log_2 ( 1 / P(x_n)) ]$.
(Each term $h_n$
is the Shannon information content of the $n$th
outcome.)
We again define the typical set with parameters $N$ and $\beta$ thus:
\beq
T_{N\b} = \left\{ \bx\in\A_X^N :
\left[ \frac{1}{N} \log_2 \frac{1}{P(\bx)} - H \right]^2 < \b^2
\right\} .
\label{eq.TNb.2}
\eeq
For all $\bx \in T_{N\b}$, the probability of $\bx$ satisfies
\beq
2^{-N(H+\b)} < P(\bx) < 2^{-N(H-\b)}.
\eeq
And by the law of large numbers,
\beq
P(\bx \in T_{N\b}) \geq 1 - \frac{\sigma^2}{\b^2 N} .
\eeq
We have thus proved the \aep. As $N$ increases, the probability
that $\bx$ falls in $T_{N\b}$ approaches 1, for any $\beta$.
How does this result relate to source coding?
% We will prove the \aep\ first; then w
We must relate $T_{N\b}$ to $H_{\delta}(X^N)$.
We will
show that for any given $\delta$ there is
a sufficiently big $N$ such that
$H_{\delta}(X^N) \simeq N H$.
\subsubsection{Part 1: $\frac{1}{N} H_{\delta}(X^N) < H +
\epsilon$.}
% of the source coding theorem.
%
% More words here reminding what H_delta is
%
The set $T_{N\b}$ is not the best subset for compression. So the
size of $T_{N\b}$ gives an upper bound on $H_{\delta}$.
We show how {\em small} $H_{\delta}(X^N)$ must be by calculating
% the largest cardinality that $T_{N\b}$ could have.
how big $T_{N\b}$ could possibly be.
We are
free to set $\beta$ to any convenient value.
The smallest possible
probability that a member of $T_{N\b}$ can have is $2^{-N(H+\b)}$, and
the total probability that $T_{N\b}$ contains can't be any bigger
than 1. So
\beq
|T_{N\b}| \, 2^{-N(H+\b)} < 1 ,
\eeq
that is, the size of the typical set is bounded by
% so we can bound
\beq
|T_{N\b}| < 2^{N(H+\b)} .
\eeq
If we set $\b = \epsilon$ and $N_0$ such that
$\frac{\sigma^2}{\epsilon^2 N} \leq \delta$, then $P(T_{N\b}) \geq
1 - \delta$,
and the set $T_{N\b}$ becomes a witness to the fact that
$H_{\delta}(X^N) \leq \log_2 | T_{N\b} | < N ( H + \epsilon)$.
%
\amarginfig{b}{
{\footnotesize
\setlength{\unitlength}{1.2mm}
\begin{picture}(40,40)(-5,0)
\put(5,5){\makebox(0,0)[bl]{\psfig{figure=figs/gallager/Hdeltaconcept.eps,width=36mm}}}
\put(5,35){\makebox(0,0){$\smallfrac{1}{N} H_{\delta}(X^N)$}}
\put(5,27){\makebox(0,0)[r]{$H_0(X)$}}
\put(5,4){\makebox(0,0)[t]{$0$}}
\put(30,4){\makebox(0,0)[t]{$1$}}
\put(35,4){\makebox(0,0)[t]{$\delta$}}
\put(33,11){\makebox(0,0)[l]{$H-\epsilon$}}
\put(33,15){\makebox(0,0)[l]{$H$}}
\put(33,19){\makebox(0,0)[l]{$H+\epsilon$}}
\end{picture}
}
\caption[a]{Schematic illustration of the two parts of the theorem.
Given any $\delta$ and $\epsilon$, we show that
for large enough $N$, $\frac{1}{N} H_{\delta}(X^N)$
lies (1) below the line
$H+\epsilon$ and (2) above the line $H-\epsilon$.}
\label{fig.Hd.schem}
}
\subsubsection{Part 2: $\frac{1}{N} H_{\delta}(X^N) >
H - \epsilon$.}
% of the source coding theorem.}
%
% needs work ,sanjoy says:
%
% (jan 99)_
%
Imagine that someone claims this second part is not so -- that,
for any $N$, the
smallest $\delta$-sufficient subset $S_{\delta}$ is smaller than the above
inequality would allow.
% They claim that
% $|S_{}| \leq 2^{N(H-\epsilon)}$ and $P(\bx \in S_{})
% \geq 1 - \delta$.
We can make use of our typical set to show that they must be mistaken.
Remember that we are free to set $\beta$ to any value we choose.
We will set $\beta = \epsilon/2$, so that our task is to
prove that a
% that an alternative {\em smaller\/}
subset $S'_{}$ having
$|S'_{}| \leq 2^{N(H-2\beta)}$ and achieving $P(\bx \in S'_{}) \geq 1 - \delta$
cannot exist (for $N$ greater than an $N_0$ that we will specify).
%(We attach the
% prime to $S$ to denote the fact that this is a conjectured smallest subset.)
So, let us consider the probability of falling in this rival smaller subset $S'_{}$.
The probability of the subset $S'_{}$ is\marginpar[t]{%
\begin{center}
\raisebox{-0.5in}[0in][0in]{
%%%%%%%% written by hand Sun 22/12/02
%
% Venn picture
%
%
\setlength{\unitlength}{0.321pt}%
{\begin{picture}(452,215)(-173,-132)%
% axis labels
\put(-100,39){\makebox(0,0)[r]{\small$T_{N\b}$}}
\put(100,39){\makebox(0,0)[l]{\small$S'$}}
\thinlines
\put(-33,-1){\circle{126}}
\thicklines
\put(33,-1){\circle{126}}
\thinlines
\put(18,-85){\vector(-1,4){18}}
\put(33,-90){\makebox(0,0)[t]{\small$ S'_{} \cap T_{N\b} $}}
\put(105,-51){\vector(-1,1){40}}
\put(112,-39){\makebox(0,0)[tl]{\small$ S'_{} \cap \overline{T_{N\b}} $}}
\end{picture}}
%
%
%
%
\end{center}}
\beq
P(\bx \in S'_{}) \,=\, P(\bx \in S' \! \cap \! T_{N\b}) +
P(\bx \in S'_{} \!\cap\! \overline{T_{N\b}}),
\eeq
where $\overline{T_{N\b}}$ denotes
the complement $\{ \bx \not \in T_{N\b}\}$.
The maximum value of the first term is found if
$S'_{} \cap T_{N\b} $ contains
$2^{N(H-2\beta)}$ outcomes all with the maximum probability,
$2^{-N(H-\beta)}$. The maximum value the second term can have is
$P( \bx \not \in T_{N\b})$. So:
\beq
P(\bx \in S'_{}) \, \leq \, 2^{N(H-2\beta)}
\, 2^{-N(H-\beta)}
+ \frac{\sigma^2}{\b^2 N}
= 2^{-N \b} + \frac{\sigma^2}{\b^2 N} .
\eeq
We can now set $\b = \epsilon/2$ and $N_0$ such that $P(\bx \in S'_{}) < 1-
\delta$, which shows that $S'$ cannot satisfy the definition of
a sufficient subset $S_{\delta}$.
Thus {\em any\/} subset $S'$ with size
$|S'| \leq 2^{N(H-\epsilon)}$ has probability less than $1-\delta$, so
by the definition of $H_\delta$, $H_{\delta}(X^N) > N ( H - \epsilon)$.
% this sentence used to be below at
% hereherehere
Thus for large enough $N$,
the function
$\frac{1}{N} H_{\delta}(X^N)$ is essentially a constant function of $\delta$,
for $0 < \delta < 1$,
as illustrated in figures \ref{fig.hd.10.1010}
and \ref{fig.Hd.schem}. \hfill $\Box$
\section{Comments}
The source coding theorem (\pref{thm.sct}) has two parts,
$\frac{1}{N} H_{\delta}(X^N) < H + \epsilon$,
and
$\frac{1}{N} H_{\delta}(X^N) >
H - \epsilon$.
% $H -\frac{1}{N} H_{\delta}(X^N)< \epsilon$.
Both results are interesting.
The first part tells us that even if the probability of
error $\delta$ is extremely small,
the
% average
number of bits per symbol
$\frac{1}{N} H_{\delta}(X^N)$ needed to specify a long $N$-symbol
string $\bx$ with vanishingly
small error probability does not
have to exceed $H+ \epsilon$ bits.
We need to have only a tiny tolerance for error, and the number of bits
required drops significantly from $H_0(X)$ to $(H + \epsilon)$.
What happens if we are yet more tolerant to compression errors? Part
2 tells us that even if $\delta$ is very close to 1, so that errors
are made most of the time, the average number of bits per symbol needed to
specify $\bx$ must still be at least $H - \epsilon$ bits. These two
extremes tell us that regardless of our specific allowance for error,
the number of bits per symbol needed to specify $\bx$ is
% boils down to
$H$ bits; no more and no less.
\medskip
% hereherehere
%In section 2.4.2 `$\epsilon$ can decrease with increasing $N$'. I'd prefer
%something like $N$ increases with decreasing $\epsilon$', since $N$
%depends on $\epsilon$ and not vice versa -- if I got it right.
% caution warning
\subsection{Caveat regarding `asymptotic equipartition'}
\label{sec.aep.caveat}
\index{caution!equipartition}I
put the words `asymptotic equipartition' in quotes because
it is important not to\index{asymptotic equipartition!why it is a misleading term}
% be misled into
think that the
elements of the typical set $T_{N\beta}$
really do have roughly the same
probability as each other. They are similar in probability only
in the sense that their values of $\log_2 \frac{1}{P(\bx)}$ are
within $2 N \beta$ of each other. Now, as $\beta$ is decreased,
how does $N$ have to increase, if we are to keep our bound on the
mass of the typical set,
$P(\bx \in T_{N\beta}) \geq 1 - \frac{\sigma^2}{\beta^2 N}$, constant?
% CHANGED 9802:
% Since $\beta$ can decrease
%scales
% with increasing
$N$ must grow as $1/ \beta^2$, so, if we write
$\beta$ in terms of
$N$ as $\alpha/\sqrt{N}$, for some constant $\alpha$, then
the most probable string in the typical set will be of order
$2^{\alpha \sqrt{N}}$ times greater than the least probable string in the
typical set. As $\beta$ decreases, $N$ increases,
and this ratio $2^{\alpha \sqrt{N}}$ grows exponentially.
Thus we have `equipartition' only in a weak sense!
% relative
\subsection{Why did we introduce the typical set?}
The best choice of subset for block compression is (by definition)
$S_{\delta}$, not a typical set. So why did we bother introducing
the typical set? The answer is, {\em we can count the typical set}.
We know that all its elements have `almost identical' probability ($2^{-NH}$),
and we know the whole set has probability almost 1, so the typical
set must have roughly $2^{NH}$ elements.
Without the help of the typical set (which is very similar
to $S_{\delta}$) it would have been
hard to count how many elements there are in $S_{\delta}$.
%\section{Summary and overview}
%\section{Where next}
% We have established that the entropy $H(X)$ measures
% the average information content of an ensemble.
%%
% In this chapter we discussed a lossy {block}-compression scheme that
% used large blocks of fixed size.
% In the next chapter we discuss variable length compression schemes that are
% practical for small block sizes and that are not lossy.
%%
%
\section{Exercises}
% weighing problems in here
% ITPRNN Problem 1a
%
\subsection*{Weighing problems}
%
\exercisaxB{1}{ex.weighexplain}{
While some people, when they first encounter
the
weighing problem with 12 balls and the three-outcome balance (\exerciseref{ex.weigh}),
think that weighing six balls against six balls is a good first weighing,
others say `no, weighing six against six conveys {\em no\/} information
at all'. Explain to the second group why they are both right and
wrong. Compute the information gained about {\em which is the
odd ball\/}, and the information gained about {\em which is the
odd ball and whether it is heavy or light}.
}
\exercisaxB{2}{ex.weighthirtynine}{
Solve the weighing problem for the case where there are 39 balls
of which one is known to be odd.
}
\exercisaxB{2}{ex.binaryweigh}{
You are given 16 balls, all of which are equal in weight except for
one that is either heavier or lighter. You are also given a bizarre
two-pan balance that can report only two outcomes: `the two sides balance'
or `the two sides do not balance'.
Design a
strategy to determine which is the odd ball {in as few uses of the balance
as possible}.
}
\exercisaxB{2}{ex.flourforty}{
You have a two-pan balance; your job is to weigh
out bags of flour with integer weights 1 to 40 pounds inclusive.
How many weights do you need? [You are allowed
to put weights on either pan. You're only allowed to
put one flour bag on the balance at a time.]
}
\exercissxC{4}{ex.twelve.generalize.weigh}{
\ben
\item% {ex.weigh}
Is it possible to solve \exerciseref{ex.weigh}
(the
weighing problem with 12 balls and the three-outcome balance)
using a sequence of three {\em fixed\/} weighings, such that the
balls chosen for the second weighing do not depend on the outcome of the first, and
the third weighing does not depend on the first or second?
\item
Find a solution to the general $N$-ball weighing problem in which exactly one of $N$
balls is odd.
Show that in $W$ weighings, an odd ball can be identified from among
$N = (3^W - 3 )/2$ balls.
%How large can $N$ be if you are allowed $W$ weighings?
% How are the weighings arranged in the case of the largest $N$?
\een
}
\exercisaxC{3}{ex.twelve.two.weigh}{
You are given 12 balls and the three-outcome balance
of \exerciseonlyref{ex.weigh}; this time, {\em two} of the balls are odd;
each odd ball may be heavy or light, and we don't know which.
We want to identify the odd balls and in which direction they are odd.
\ben
\item
{\em Estimate\/} how many weighings are required by the optimal strategy.
And what if there are three odd balls?
%\item
% How do your answers change if it is known in advance that
% the odd balls will all have the same bias (all heavy, or all light)?
\item
How do your answers change if it is known that all the regular balls
weigh 100\grams, that light balls weigh 99\grams, and heavy ones
weigh 110\grams?
\een
}
% end weighing
\subsection*{Source coding with a lossy compressor, with loss $\delta$}
\exercissxB{2}{ex.Hd46}{
% Let ${\cal P}_X = \{ 0.4,0.6 \}$. Sketch $\frac{1}{N} H_{\delta}(X^N)$
% as a function of $\delta$ for $N=1,2$ and 100.
Let ${\cal P}_X = \{ 0.2,0.8 \}$. Sketch $\frac{1}{N} H_{\delta}(X^N)$
as a function of $\delta$ for $N=1,2$ and 1000.
}
\exercisaxB{2}{ex.Hd55}{
Let ${\cal P}_Y = \{ 0.5,0.5 \}$. Sketch $\frac{1}{N} H_{\delta}(Y^N)$
as a function of $\delta$ for $N=1,2,3$ and 100.
}
\exercissxB{2}{ex.HdSB}{
(For \ind{physics} students.)
Discuss the
relationship
% similarities
between the proof of the \aep\ and the equivalence\index{entropy!Gibbs}\index{entropy!Boltzmann}
(for large systems) of the \ind{Boltzmann entropy} and the \ind{Gibbs entropy}.}
\subsection*{Distributions that don't obey the law of large numbers}
%
% Cauchy distbn here?
The \ind{law of large numbers}, which we used in this chapter,
shows that the mean of a set of $N$ i.i.d.\ random variables
has a probability distribution that becomes
% more concentrated
narrower, with width $\propto 1/\sqrt{N}$, as $N$ increases.
However, we have proved this property only for
discrete random variables, that is, for real numbers
taking on a {\em finite\/} set of possible values.
While many random variables
with continuous probability distributions also satisfy the
law of large numbers, there are important distributions that
do not. Some continuous distributions do not have
a mean or variance.
\exercissxB{3}{ex.cauchy}{
Sketch the \ind{Cauchy distribution}
\beq
P(x) = \frac{1}{Z} \frac{1}{x^2 + 1} , \:\:\:\: x \in (-\infty,\infty).
\eeq
What is its normalizing constant $Z$? Can you evaluate
its mean or variance?
Consider the sum $z=x_1 + x_2$, where $x_1$ and $x_2$ are independent
random variables from a Cauchy
distribution. What is $P(z)$? What is the probability
distribution of the mean of $x_1$ and $x_2$, $\bar{x}=(x_1+x_2)/2$?
What is the
probability
distribution of the mean of $N$ samples from this \ind{Cauchy distribution}?
}
%
\subsection{Other asymptotic properties}
% Levy flights too?
\exercisaxC{3}{ex.chernoff}{ {\sf\ind{Chernoff bound}.}
We derived the weak law of large numbers from Chebyshev's inequality\index{Chebyshev inequality}
(\ref{eq.cheb.1}) by letting the random variable $t$
in the inequality
$%\beq
P(t \geq \a) \:\leq\: \bar{t}/\a
%\label{eq.cheb.1a}
$
be a function, $t = (x-\bar{x})^2$,
of the random variable $x$ we were interested in.
Other useful inequalities can be obtained by using other
functions. The \ind{Chernoff bound}, which is useful\index{bound}
for bounding the \ind{tail}s of a distribution, is obtained by
letting $t = \exp( s x)$.
Show that
\beq
P( x \geq a ) \leq e^{-sa} g(s) , \:\:\:\mbox{ for any $s>0$ }
\eeq
and
\beq
P( x \leq a ) \leq e^{-sa} g(s) , \:\:\:\mbox{ for any $s<0$ }
\eeq
where $g(s)$ is the moment-generating function of $x$,
\beq
g(s) = \sum_x P(x) \, e^{sx} .
\eeq
%
% Hence show that if $z$ is a sum of $N$ random variables $x$,
%\beq
% P( z \geq a ) \leq
%\eeq
}
% end
%
\subsection*{Curious functions related to $p \log 1/p$}
% SOLN - BORDERLINE
\exercissxE{4}{ex.fxxxxx}{
This exercise has {no purpose at all}; it's included
for the enjoyment of those who like mathematical curiosities.
Sketch the function
\beq
f(x) = x^{x^{x^{x^{x^{\cdot^{\cdot^{\cdot}}}}}}}
% f(x) = x^{x^{x^{x^{x^{\ddots}}}}}
\eeq
for $x \geq 0$.
% To be explicit about the order in which the powers are evaluated,
% here's another definition of $f$:
%\beq
% f(x) = x^{\left(x^{\left(x^{\cdot^{\cdot^{\cdot}}}\right)}\right)}
%\eeq
{\sf Hint:}
Work out the inverse function to $f$ -- that is, the function $g(y)$
such that if $x=g(y)$ then $y=f(x)$ -- it's closely related to
$p \log 1/p$.
% {\sf Hints:}
%\ben
%\item Consider $f(\sqrt{2})$:
% you might be able to persuade yourself
% that $f(\sqrt{2})=2$. You might also be able
% to persuade yourself that $f(\sqrt{2})=4$. What's going on?
% [Yes, a two-valued function.]
%\item
% For a given $x$, if $f(x)=y$, then we have $y = x^{y}$, so
% $y$ is found at the intersection of the curves $u_1(y)=x^y$ and $u_2(y)=y$.
%\item
% Work out the inverse function to $f$ -- that is, the function $g(y)$
% such that if $x=g(y)$ then $y=f(x)$ -- hint: it's closely related to
% $p \log 1/p$.
%\een
}
\dvips
%\chapter{The Source Coding Theorem (old version of this Chapter)}
%\label{ch.two.old}
%\input{tex/_l2old.tex}
%\dvips
\section{Solutions}% to Chapter \protect\ref{ch.two}'s exercises}
\fakesection{_s2}
% chapter 2
% ex 39...
%
\soln{ex.Hadditive}{
Let $P(x,y)=P(x)P(y)$.
Then
\beqan
H(X,Y) &=& \sum_{xy} P(x)P(y) \log \frac{1}{P(x)P(y)} \\
& = & \sum_{xy} P(x)P(y) \log \frac{1}{P(x)}
+ \sum_{xy} P(x)P(y) \log \frac{1}{ P(y)} \\
&=& \sum_{x} P(x) \log \frac{1}{P(x)} +
\sum_{y} P(y) \log \frac{1}{ P(y)} \\
&=& H(X) + H(Y) .
\eeqan
}
%
\soln{ex.ascii}{
An ASCII file can be reduced in size by a factor of 7/8. This reduction
could be achieved by a block code that maps 8-byte blocks
into 7-byte blocks by copying the
% . The mapping would copy
56 information-carrying bits into
7 bytes, and ignoring the last bit of every character.
}
\soln{ex.compress.possible}{
% Theorem:
% No program can compress without loss *all* files of size >= N bits, for
% any given integer N >= 0.
%
%Proof:
% Assume that the program can compress without loss all files of size >= N
% bits. Compress with this program all the 2^N files which have exactly N
% bits. All compressed files have at most N-1 bits, so there are at most
% (2^N)-1 different compressed files [2^(N-1) files of size N-1, 2^(N-2) of
% size N-2, and so on, down to 1 file of size 0]. So at least two different
% input files must compress to the same output file. Hence the compression
% program cannot be lossless.
%
%The proof is called the "counting argument". It uses the so-called
The \ind{pigeon-hole principle}
states: you can't put 16 pigeons into 15 holes without using one of the
holes twice.
Similarly, you can't give $\A_X$ outcomes unique
binary names of some length $l$
shorter than $\log_2 |\A_X|$ bits, because there are only $2^l$
such binary names, and $l < \log_2 |\A_X|$ implies $2^l < |\A_X|$,
so at least two different inputs to the compressor would compress to
the same output file.
}
\soln{ex.cusps}{
Between the cusps, all the changes in
probability are equal, and the number of elements
in $T$ changes by one at each step. So $H_{\delta}$
varies logarithmically with $(-\delta)$.
% NEEDS WORK!
}
%
% Another solution from Conway:
% Label them
% F AM NOT LICKED
% then use these divisions
% MA DO LIKE
% ME TO FIND
% FAKE COIN
%
%\soln{ex.twelve.generalize.weigh}{
% Thu, 28 Jan 1999 19:19:30 -0500 (EST)
% From:
%
\begin{Sexercise}{ex.twelve.generalize.weigh}
This solution was found by Dyson and Lyness in 1946
and presented in the following elegant form by
{John Conway}\index{Conway, John H.} in 1999.
% \footnote{Posting to {\tt{geometry-puzzles@forum.swarthmore.edu}}
% Thu, 28 Jan 1999.
%}
%
Be warned: the symbols A, B, and C are used to name the
balls, to name the pans of the balance,
to name the outcomes, and to name
the possible states of the odd ball!
\ben%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% enumerate 1
\item
Label the 12 balls by the sequences
%
% verbatim not allowed in the argument of a command
%
{\small
\begin{verbatim}
AAB ABA ABB ABC BBC BCA BCB BCC CAA CAB CAC CCA
\end{verbatim}
}
and in the
{\small
\begin{verbatim}
1st AAB ABA ABB ABC BBC BCA BCB BCC
2nd weighings put AAB CAA CAB CAC in pan A, ABA ABB ABC BBC in pan B.
3rd ABA BCA CAA CCA AAB ABB BCB CAB
\end{verbatim}
}
Now in a given weighing, a pan will either end up in the
\bit
\item
{\tt C}anonical position ({\tt C}) that it assumes when the pans are balanced, or
\item
{\tt A}bove that position ({\tt A}), or
\item
{\tt B}elow it ({\tt B}),
\eit
so the three weighings determine for each pan a sequence of three of these letters.
If both sequences are {\tt CCC}, then there's no odd ball. Otherwise,
for {\em just one\/} of the two pans, the sequence is among the 12 above,
and names the odd ball, whose weight is {\tt A}bove or {\tt B}elow the proper
one according as the pan is {\tt A} or {\tt B}.
\item
In $W$ weighings the odd ball can be identified from
among
\beq
N = (3^W - 3 )/2
\eeq
balls in the same way, by labelling them with all
the non-constant sequences of $W$ letters from {\tt A}, {\tt B}, {\tt C} whose
first change is A-to-B or B-to-C or C-to-A, and at the
$w$th weighing putting those whose $w$th letter is {\tt A} in pan {\tt A}
and those whose $w$th letter is {\tt B} in pan {\tt B}.
\een
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%}
\end{Sexercise}
% {ex.twelve.two.weigh}{
% removed old solution to graveyard Tue 4/3/03
\soln{ex.Hd46}{% ex 42
% hd.p p=0.2 mmin=1 mmax=2 mstep=1 scale_by_n=1 plot_sub_graphs=1 | gnuplot
% hd.p p=0.2 mmin=2 mmax=2 mstep=1 scale_by_n=1 plot_sub_graphs=1 | gnuplot
% hd.p p=0.2 mmin=100 mmax=100 mstep=1 suppress_early_detail=1 scale_by_n=1 plot_sub_graphs=1 | gnuplot
% hd.p p=0.2 mmin=1000 mmax=1000 mstep=1 suppress_early_detail=1 scale_by_n=1 plot_sub_graphs=1 hd=figs/hd0.2 | gnuplot
%# gnuplot < gnu/Hd0.2.gnu
%#45:coll:/home/mackay/itp/Hdelta> gv figs/hd0.2/all.1.100.ps
The curves $\frac{1}{N} H_{\delta}(X^N)$
as a function of $\delta$ for $N=1,2$ and 1000 are shown in \figref{fig.hd.1.100}.
% and table \ref{tab.Hdelta.0.4}.
Note that $H_2(0.2) = 0.72$ bits.
\begin{figure}[htbp]
%\figuremargin{%
\figuredanglenudge{%
\begin{center}
\begin{tabular}[t]{rl}
\begin{tabular}[t]{l}\vspace{0in}\\% alignment hack
\mbox{\psfig{figure=Hdelta/figs/hd0.2/all.1.100.ps,%
width=60mm,angle=-90}}
\end{tabular}
%
\hspace{0in}
&
%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{tabular}[t]{r@{--}lcc} \toprule
\multicolumn{4}{c}{$N=1$} \\ \midrule
% delta 1/N Hdelta 2^{Hdelta}
\multicolumn{2}{c}{$\delta$} & $\frac{1}{N} H_{\delta}(\bX)$ & $2^{H_{\delta}(\bX)}$
% raise the roof!
% {\rule[-3mm]{0pt}{8mm}}
\\ \midrule
0 & 0.2 & 1 & 2 \\
0.2 & 1 & 0 & 1 \\ \bottomrule
\end{tabular}
\hspace{0.1in}
\begin{tabular}[t]{r@{--}lcc} \toprule% {r@{--}lcc}
\multicolumn{4}{c}{$N=2$} \\ \midrule
% delta 1/N Hdelta 2^{Hdelta}
\multicolumn{2}{c}{$\delta$} & $\frac{1}{N} H_{\delta}(\bX)$ & $2^{H_{\delta}(\bX)}$
% raise the roof!
% {\rule[-3mm]{0pt}{8mm}}
\\ \midrule
0 & 0.04 & 1 & 4 \\
0.04 & 0.2 & 0.79 & 3 \\ % was 0.792\,48
0.2 & 0.36 & 0.5 & 2 \\
0.36 & 1 & 0 & 1 \\ \bottomrule
\end{tabular}\\
\end{tabular}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{center}
}{%
\caption[a]{$\frac{1}{N} H_{\delta}(\bX)$ (vertical axis) against $\delta$ (horizontal),
for $N=1, 2, 100$ binary variables with $p_1=0.4$.}
\label{fig.hd.1.100}
\label{tab.Hdelta.0.4}
}{0.25in}
\end{figure}
%\begin{table}[htbp]
%\figuremargin{%
%\begin{center}
%\end{center}
%}{%
%\caption[a]{Values of $\frac{1}{N} H_{\delta}(\bX)$ against $\delta$.}
%% add 0.4 to this caption
%\label{tab.Hdelta.0.4}
%}
%\end{table}
%
}
\soln{ex.HdSB}{
The Gibbs entropy is $\kB \sum_i p_i \ln \frac{1}{p_i}$, where $i$
runs over all states of the system. This entropy is equivalent (apart from the factor of $\kB$)
to the Shannon entropy of the ensemble.
Whereas the Gibbs entropy can be
defined for any ensemble, the Boltzmann entropy is only
defined for {\dem microcanonical\/} ensembles, which
have a probability distribution that is uniform over a
set of accessible states.
The Boltzmann entropy is defined to be $S_{\rm B} = \kB \ln \Omega$
where $\Omega$ is the number of accessible states
of the microcanonical ensemble. This is equivalent
(apart from the factor of $\kB$) to the perfect information content
$H_0$ of that constrained
ensemble. The Gibbs entropy of a microcanonical
ensemble is trivially equal to the Boltzmann entropy.
We now consider a \ind{thermal distribution} (the
{\dem\ind{canonical}\/} ensemble),
where the probability of a state $\bx$ is
\beq
% P(\bx) =\frac{1}{Z} \exp( - \beta E(\bx) )?
P(\bx) =\frac{1}{Z} \exp\left( - \frac{ E(\bx) }{\kB T} \right) .
\eeq
With this canonical ensemble we can associate a
corresponding microcanonical ensemble,
% typically
% usually
an ensemble
with total energy fixed to the mean
energy of the canonical ensemble
(fixed to within some precision $\epsilon$).
% Recalling that under the
% thermal distribution (the canonical ensemble) we see that
Now, fixing the total energy to a precision $\epsilon$ is equivalent to
fixing the value of $\ln \dfrac{1}{P(\bx)}$ to within
% $\epsilon/\beta$.
$\epsilon \kB T$.
Our definition of the typical set
$T_{N \beta}$ was precisely that it consisted of all elements that
have a value of $\log P(\bx)$ very close to the mean value
of $\log P(\bx)$ under the canonical ensemble, $- N H(X)$.
Thus the microcanonical ensemble is equivalent to
a uniform distribution over
% constraining the state $\bx$ to be in
the typical set of the canonical ensemble.
Our proof of the \aep\ thus proves -- for the
case of a system whose energy is separable into a sum of independent
terms -- that the
Boltzmann entropy of the microcanonical ensemble
is very close (for large $N$) to the Gibbs entropy of
the canonical ensemble, if the energy of the microcanonical
ensemble is constrained to equal the mean energy of the
canonical ensemble.
}
\soln{ex.cauchy}{
The normalizing constant of the \ind{Cauchy distribution}\index{distribution!Cauchy}
\[
P(x) = \frac{1}{Z} \frac{1}{x^2 + 1}
\]
is
\beq
Z = \int^{\infty}_{-\infty} \d x \: \frac{1}{x^2 + 1}
= \left[ {\tan}^{-1} x \right]^{\infty}_{-\infty} = \frac{\pi}{2} - \frac{-\pi}{2} = \pi .
\eeq
The mean and variance of this distribution are both undefined. (The distribution
is symmetrical about zero, but this does not imply that its mean is zero. The mean
is the value of a divergent integral.)
% ; depending what limiting procedure we
% define to evaluate this integral we
The sum $z=x_1 + x_2$, where $x_1$ and $x_2$ both
have Cauchy distributions, has probability density given by the convolution
\beq
P(z) = \frac{1}{\pi^2} \int^{\infty}_{-\infty} \d x_1 \:
\frac{1}{x_1^2 + 1}
\frac{1}{(z-x_1)^2 + 1}
% P(x1,x2) delta [z=x1+x2] .. -> x2 = z-x1
,
\eeq
% Introducing $\Delta \equiv x_1-x_2$ this can be written more symmetrically
% as
% \beq
% P(z) = \frac{1}{\pi^2} \int^{\infty}_{-\infty} \d \Delta \:
% \eeq
which after a considerable labour using standard methods
%\footnote{Can anyone
% give me an elegant solution?}
gives
\beq
P(z) = \frac{1}{\pi^2} 2 \frac{\pi}{z^2+4} = \frac{2}{\pi} \frac{1}{z^2+2^2} ,
\label{eq.cauchysum}
\eeq
which we recognize as a Cauchy distribution with width parameter 2
(where the original distribution has width parameter 1).
This implies that the mean of the two points, $\bar{x} = (x_1+x_2)/2 = z/2$,
has a Cauchy distribution with width parameter 1. Generalizing, the mean
of $N$ samples from a Cauchy distribution is Cauchy-distributed
with the {\em same parameters\/} as the individual samples. The probability
distribution of the mean does {\em not\/} become narrower
as $1/\sqrt{N}$.
{\em The \ind{central-limit theorem} does not apply to the \ind{Cauchy distribution},
because it does not have a finite \ind{variance}.}
An alternative neat method for getting to \eqref{eq.cauchysum} makes
use of the \ind{Fourier transform}\index{generating function}
of the Cauchy distribution, which is
a \index{biexponential distribution}{biexponential} $e^{-|\omega|}$. Convolution in real space
corresponds to multiplication in Fourier space,
so the \ind{Fourier transform} of $z$ is simply $e^{-|2 \omega|}$.
Reversing the transform, we obtain \eqref{eq.cauchysum}.
}
%\begincuttable
\soln{ex.fxxxxx}{
\amarginfig{t}{
\begin{center}
\begin{tabular}{c}
\psfig{figure=gnu/fxxxxx50.ps,width=1.7in,angle=-90}\\
\psfig{figure=gnu/fxxxxx5.ps,width=1.7in,angle=-90}\\
\psfig{figure=gnu/fxxxxx.5.ps,width=1.7in,angle=-90}\\
\end{tabular}
\end{center}
%}{% gnu: load 'fxxxxx.gnu'
\caption[a]{
% The function
$\displaystyle
f(x) = x_{\:,}^{x^{x^{x^{x^{\cdot^{\cdot^{\cdot}}}}}}}
$ shown at three different scales.}
\label{fig.xxxxx}
}%
The function $f(x)$
%\beq
% f(x) = x^{x^{x^{x^{x^{\ddots}}}}}
%\eeq
has inverse function
% to $f$ is
\beq
g(y) = y^{1/y}.
\eeq
Note
\beq
\log g(y) = 1/y \log y .
\eeq
I obtained a tentative graph of $f(x)$ by plotting $g(y)$ with
$y$ along the vertical axis and $g(y)$ along the horizontal
axis. The resulting graph suggests that $f(x)$
is single valued for $x \in (0,1)$, and looks surprisingly well-behaved
and ordinary; for $x \in (1, e^{1/e})$, $f(x)$ is two-valued.
$f(\sqrt{2})$ is equal both to 2 and 4.
For $x > e^{1/e}$ (which is about 1.44), $f(x)$ is infinite.
% undefined.
However, it might be argued that this approach to sketching $f(x)$
is only partly valid, if we define $f$ as the limit of the
sequence of functions $x$,
$x^x$, $x^{x^x}, \ldots$;
this sequence does not
have a limit for
% , below
% pr (1.0/exp(1.0))**exp(1.0)
% 0.0659880358453126
$0 \leq x \leq (1/e)^e \simeq 0.07$
on account of a pitchfork \ind{bifurcation} at $x=(1/e)^e$;
and for $x \in (1,e^{1/e})$, the sequence's limit is single-valued --
the lower of the two values sketched in the figure.
% load 'fxxxxx.gnu2'
%
}
%\endcuttable
\dvipsb{solutions source coding}
\prechapter{About Chapter}
\fakesection{intro for chapter 3}
In the last chapter, we saw a proof of the fundamental status of the entropy
as a measure of average information content.
We defined a data compression scheme using
{\em fixed length block codes}, and
proved that as $N$ increases,
it is possible to encode $N$ i.i.d.\ variables
$\bx = (x_1,\ldots,x_N)$ into a block of $N(H(X)+\epsilon)$ bits
with vanishing probability of error, whereas if we attempt to
encode $X^N$ into $N(H(X)-\epsilon)$ bits, the probability of
error is virtually 1.
We thus verified the {\em possibility\/} of
data compression, but the block coding defined in the proof
did not give a practical algorithm.
In this chapter and the next,
we study practical data compression algorithms.
Whereas the last chapter's compression scheme
used large blocks of {\em fixed\/} size and was
{\em lossy}, in the next chapter we discuss
{\em variable-length\/} compression schemes that are
practical for small block sizes and that are {\em not lossy}.
Imagine a rubber glove filled with water. If we compress two
fingers of the glove, some other part of the glove has
to expand, because
the total volume of water is constant. (Water is essentially
incompressible.) Similarly, when we shorten
the codewords for some outcomes, there must be other
codewords that get longer, if the scheme is not lossy.
In this chapter we will discover the information-theoretic
equivalent of water volume.
% the constant volume of water in the glove.
%%
\medskip
\fakesection{prerequisites for chapter 3}
Before reading \chref{ch.three}, you should have worked on
\extwenty.
\medskip
We will use the\index{notation!intervals}
following notation for intervals:\medskip
% the statement
\begin{center}
\begin{tabular}{ll}
$x \in [1 ,2)$ & means that $x \geq 1$ and $x < 2$; \\
% the statement
$x \in (1 ,2]$ & means that $x > 1$ and $x \leq 2$.\\
\end{tabular}
\end{center}
% {All these definitions of source
% codes, Huffman codes, etc., can be generalized to codes over
% other $q$-ary alphabets, but little is lost by concentrating on
% the binary case.}
%\chapter{Data Compression II: Symbol Codes}
\mysetcounter{page}{102}
\ENDprechapter
\chapter{Symbol Codes}
\label{ch.three}
% %.tex
% \documentstyle[twoside,11pt,chapternotes,lsalike]{itchapter}
% \begin{document}
% \bibliographystyle{lsalike}
% \input{psfig.tex}
% \include{/home/mackay/tex/newcommands1}
% \include{/home/mackay/tex/newcommands2}
% \input{itprnnchapter.tex}
% \setcounter{chapter}{2}% set to previous value
% \setcounter{page}{34} % set to current value
% \setcounter{exercise_number}{45} % set to imminent value
% %
% \renewcommand{\bs}{{\bf s}}
% \newcommand{\eq}{\mbox{$=$}}
% \chapter{Data Compression II: Symbol Codes}
% % \section*{Source Coding: Lossless data compression with symbol codes}
% % Practical source coding
\label{ch3}
%\section{Symbol codes}
In this chapter, we discuss
{\dem variable-length symbol codes\/}\indexs{symbol code},\index{source code!symbol code}
% , variable-length},
which encode one source symbol at a time, instead of encoding huge strings of
$N$ source symbols. These codes are
{\dem lossless:}
unlike the last chapter's block codes, they are guaranteed to
compress and decompress without
any errors; but there is a chance that the codes may sometimes produce
encoded strings longer than the original source string.
The idea is that we can achieve compression, on average,
by assigning {\em shorter\/} encodings to the more
probable outcomes and {\em longer\/} encodings to the less probable.
The key issues are:
\begin{description}
\item[What are the implications if a symbol code is {\em lossless\/}?]
If some codewords are shortened, by how much do other codewords
have to be lengthened?
\item[Making compression practical\puncspace]
How can we ensure that a symbol code is easy to decode?
\item[Optimal symbol codes\puncspace]
How should we assign codelengths to achieve the best
compression, and what is the best achievable compression?
\end{description}
We again verify the
fundamental status of the Shannon \ind{information content}
and the entropy, proving:\index{source coding theorem}
%
%
\begin{description}
\item[Source coding theorem (symbol codes)\puncspace]
There exists a variable-length encoding $C$ of an ensemble
$X$ such that the average length of an encoded symbol,
$L(C,X)$, satisfies
$L(C,X) \in \left[ H(X) , H(X) + 1 \right)$.
The average length is equal to the entropy $H(X)$ only if the codelength
for each outcome is equal to its \ind{Shannon information content}.
\end{description}
%
We will also define a constructive procedure, the
\index{Huffman code}Huffman
coding algorithm, that produces optimal symbol codes.\index{symbol code!optimal}\index{source code!symbol code!optimal}
\begin{description}
\item[Notation for alphabets\puncspace] $\A^N$ denotes the set of
ordered $N$-tuples of elements from the set $\A$, \ie,
all strings of length $N$.
The symbol $\A^+$ will denote the set of all strings of finite
length composed of elements from the set $\A$.
\end{description}
\exampla{ $\{{\tt{0}},{\tt{1}}\}^3 = \{{\tt{0}}{\tt{0}}{\tt{0}},{\tt{0}}{\tt{0}}{\tt{1}},{\tt{0}}{\tt{1}}{\tt{0}},{\tt{0}}{\tt{1}}{\tt{1}},{\tt{1}}{\tt{0}}{\tt{0}},{\tt{1}}{\tt{0}}{\tt{1}},{\tt{1}}{\tt{1}}{\tt{0}},{\tt{1}}{\tt{1}}{\tt{1}}\}$. }
\exampla{
$\{{\tt{0}},{\tt{1}}\}^+ = \{ {\tt{0}} , {\tt{1}} , {\tt{0}}{\tt{0}} , {\tt{0}}{\tt{1}} , {\tt{1}}{\tt{0}} , {\tt{1}}{\tt{1}} , {\tt{0}}{\tt{0}}{\tt{0}} , {\tt{0}}{\tt{0}}{\tt{1}} , \ldots \}$.
}
% This notation is borrowed from the standard notation for expressions
% in computer science
\section{Symbol codes}
\label{sec.symbol.code.intro}
\begin{description}
\item[A (binary) symbol code]
$C$ for an ensemble $X$ is a mapping from the range of $x$,
$\A_X \eq \{a_1,\ldots, $ $a_I\}$, to $\{{\tt{0}},{\tt{1}}\}^+$.
% a set of finite length strings of symbols
% from an alphabet (NAME?).
$c(x)$ will denote the {\dem{codeword}\/}\indexs{symbol code!codeword}
corresponding to $x$,
and $l(x)$ will denote its length, with $l_i = l(a_i)$.
The {\dem \inds{extended code}\/} $C^+$
is a mapping from $\A_X^+$ to $\{{\tt{0}},{\tt{1}}\}^+$
obtained by concatenation, without punctuation, of the
corresponding codewords:\index{concatenation!in compression}
\beq
c^+(x_1 x_2 \ldots x_N) = c(x_1)c(x_2)\ldots c(x_N) .
\eeq
[The term `\ind{mapping}' here is a synonym for `function'.]
\end{description}
\exampla{
A symbol code for the ensemble
$X$ defined by
\beq
\begin{array}{*{4}{c}*{5}{@{\,}c}}
& \A_X & = & \{ & {\tt a}, & {\tt b}, & {\tt c}, & {\tt d} & \} , \\
& \P_X & = & \{ & \dhalf, & \dquarter, & \deighth, & \deighth & \},
\end{array}
\eeq
% : \A_X = \{{\tt{a}},{\tt{b}},{\tt{c}},{\tt{d}}\},$ $\P_X = \{ \dhalf,\dquarter,\deighth,\deighth \}$
is $C_0$, shown in the margin.
% = \{ {\tt{1}}{\tt{0}}{\tt{0}}{\tt{0}}, {\tt{0}}{\tt{1}}{\tt{0}}{\tt{0}}, {\tt{0}}{\tt{0}}{\tt{1}}{\tt{0}}, {\tt{0}}{\tt{0}}{\tt{0}}{\tt{1}}\}$.
\marginpar{
\begin{center}
$C_0$:
\begin{tabular}{clc} \toprule
$a_i$ & $c(a_i)$ & $l_i$
% {\rule[-3mm]{0pt}{8mm}}%strut
\\ \midrule
{\tt a} & {\tt 1000} & 4 \\
{\tt b} & {\tt 0100} & 4 \\
{\tt c} & {\tt 0010} & 4 \\
{\tt d} & {\tt 0001} & 4 \\
\bottomrule
\end{tabular}
\end{center}
}
Using the extended code, we may encode ${\tt{acdbac}}$
as
\beq
c^{+}({\tt{acdbac}}) =
{\tt{1000}}
{\tt{0010}}
{\tt{0001}}
{\tt{0100}}
{\tt{1000}}
{\tt{0010}} .
\eeq
}
There are basic requirements for a useful symbol code.
First, any encoded string must have a unique decoding.
Second, the symbol code must be easy to decode.
And third, the code should achieve as much compression as possible.
\subsection{Any encoded string must have a unique decoding}
\begin{description}
\item[A code $C(X)$ is uniquely decodeable] if, under the
extended code $C^+$, no two distinct
strings have the same encoding,
% every element of $\A_X^+$ maps into a different string,
\ie,
\beq
\forall \, \bx,\by \in \A_X^+, \:\: \bx \not = \by \:\: \Rightarrow \:\:
c^+(\bx) \not = c^+(\by).
\label{eq.UD}
\eeq
%cnp22@maths.cam.ac.uk:
% I'm missing the word `injectivity'. This would explain, why
% (3.2) is necessary for an inverse function.
%
% {\em I believe mathematicians would put it this way:
% a code is uniquely decodeable if the extended code is an injective
% mapping.}
\end{description}
The code $C_0$ defined above is an example of a uniquely decodeable
code.
\subsection{The symbol code must be easy to decode}
A symbol code
is easiest to decode if it is possible to identify the end of a
codeword as soon as it arrives, which means that no codeword can
be a {\dem{prefix}\/} of another codeword.
%
% {\em (Need a defn of a prefix here.)}
%\marginpar{\footnotesize
% [A word $c$
%% \in \A^{+}$
% is a {\dem prefix\/} of another word $d$
%% \in \A^{+}$
% if there exists a tail string $t$
%% \in \A^{*}
% such that the concatenation $ct$ is
% identical to $d$. For example, {\tt 1} is a prefix of {\tt 101},
% and so is {\tt 10}.]
%}
[A word $c$
% \in \A^{+}$
is a {\dem prefix\/} of another word $d$
% \in \A^{+}$
if there exists a tail string $t$
% \in \A^{*}
such that the concatenation $ct$ is
identical to $d$. For example, {\tt 1} is a prefix of {\tt 101},
and so is {\tt 10}.]
%
We will show later that we don't lose
any performance if we constrain our symbol code to be
a prefix code.
\begin{description}
\item[A symbol code is called a \inds{prefix code}]
if no codeword is a prefix of
any other codeword.
A prefix code is also known as an {\dem\ind{instantaneous}\/}
or {\dem\ind{self-punctuating}\/}
code, because an encoded string can be decoded
from left to right without looking ahead to subsequent
codewords. The end of a codeword is immediately recognizable.
A prefix code is uniquely decodeable.
\end{description}
\begin{aside}
{Prefix codes are also
% is more accurately called
known as `prefix-free codes' or `prefix condition codes'.}
\end{aside}
Prefix codes correspond to trees.
\exampla{
\amarginfignocaption{t}{\mbox{\small$C_1$ \psfig{figure=figs/C1.ps,angle=-90,width=1in}}}
The code $C_1 = \{ {\tt{0}} , {\tt{1}}{\tt{0}}{\tt{1}} \}$ is a prefix code because
${\tt{0}}$ is not a prefix of {\tt{1}}{\tt{0}}{\tt{1}}, nor is {\tt{1}}{\tt{0}}{\tt{1}} a prefix of {\tt{0}}.
}
\exampla{
Let $C_2 = \{ {\tt{1}} , {\tt{1}}{\tt{0}}{\tt{1}} \}$. This code is not a prefix code because
${\tt{1}}$ is a prefix of {\tt{1}}{\tt{0}}{\tt{1}}.
}
\exampla{
% \marginpar[t]{\mbox{\small\raisebox{0.4in}[0in][0in]{$C_3$} \psfig{figure=figs/C3.ps,angle=-90,width=1in}}}
The code $C_3 = \{
{\tt 0} ,
{\tt 10} ,
{\tt 110} ,
{\tt 111}
\}$
is a prefix code.
%
}
%%%%%%%%%%%%%%%
\exampla{
\amarginfignocaption{t}{\mbox{\small\raisebox{0.4in}[0in][0in]{$C_3$} \psfig{figure=figs/C3.ps,angle=-90,width=1in}}\\[0.21in]
\mbox{\small%
\raisebox{0.2in}[0in][0in]{$C_4$} \psfig{figure=figs/C4.ps,angle=-90,width=0.681in}%
}\\[0.125in]
\small\raggedright
Prefix codes can be represented on binary trees. {\dem Complete\/} prefix codes
correspond to binary trees with no unused branches. $C_1$ is an incomplete code.}
The code $C_4 = \{
{\tt 00} ,
{\tt 01} ,
{\tt 10} ,
{\tt 11}
\}$
is a prefix code.
%
}
%%%%%%%%%%%%%%%
\exercissxA{1}{ex.C1101}{
Is $C_2$ uniquely decodeable?
}
%
% example
%
% morse code with spaces stripped out. Is it a prefix code? Is it UD?
% (no,no)
%
\exampla{
% ref corrected 9802
Consider \exerciseref{ex.weigh} and \figref{fig.weighing} (\pref{fig.weighing}).
Any weighing strategy that identifies the odd ball and whether it
is heavy or light can be viewed as assigning a {\em ternary\/}
code to each of the 24 possible states.
This code is a prefix code.
}
\subsection{The code should achieve as much compression as possible}
\begin{description}
\item[The expected length $L(C,X)$] of a symbol code $C$ for ensemble $X$ is
\beq
L(C,X) = \sum_{x \in \A_X} P(x) \, l(x).
\eeq
We may also write this quantity as
\beq
L(C,X) = \sum_{i=1}^{I} p_i l_i
\eeq
where $I = |\A_X|$.
\end{description}
%
\exampla{
% {\sf Example 1:}
\marginpar[b]{
\begin{center}
$C_3$:\\[0.1in]
\begin{tabular}{cllcc} \toprule
$a_i$ & $c(a_i)$ & $p_i$ &
% \multicolumn{1}{c}{$\log_2 \frac{1}{p_i}$}
$h(p_i)$
& $l_i$
% {\rule[-3mm]{0pt}{8mm}}%strut
\\ \midrule
{\tt a} & {\tt 0} & \dhalf & 1.0 & 1 \\
{\tt b} & {\tt 10} & \dquarter & 2.0 & 2 \\
{\tt c} & {\tt 110} & \deighth & 3.0 & 3 \\
{\tt d} & {\tt 111} & \deighth & 3.0 & 3 \\
\bottomrule
\end{tabular}
\end{center}
}
Let
\beq
\begin{array}{*{4}{c}*{5}{@{\,}c}}
& \A_X & = & \{ & {\tt a}, & {\tt b}, & {\tt c}, & {\tt d} & \} , \\
\mbox{and} \:\:& \P_X & = & \{ & \dhalf, & \dquarter, & \deighth, & \deighth & \},
\end{array}
\eeq
and consider the code $C_3$.
% $c(a)\eq {\tt{0}}$, $ c(b)\eq {\tt{1}}{\tt{0}}$,
% $c(c)\eq {\tt{1}}{\tt{1}}{\tt{0}}$, $ c(d)\eq {\tt{1}}{\tt{1}}{\tt{1}}$.
%
The entropy of $X$ is 1.75 bits, and the expected length $L(C_3,X)$ of this
code is also 1.75 bits. The sequence of symbols $\bx\eq ({\tt acdbac})$ is
% 134213
encoded as $c^+(\bx)={\tt{0110111100110}}$.
% You can confirm that no other sequence of
% symbols $\bx$ has the same encoding.
% In fact,
$C_3$ is a {prefix code\/}
and is therefore \inds{uniquely decodeable}.
Notice that the codeword lengths satisfy $l_i \eq \log_2 (1/p_i)$, or
equivalently,
$p_i \eq 2^{-l_i}$.
}
%\medskip
%
%\noindent {\sf Example 2:}
\exampla{
Consider the fixed length code for the same ensemble
$X$, $C_4$.
% $ c(1)\eq {\tt{00}}$, $ c(2)\eq {\tt{01}}$, $ c(3)\eq {\tt{10}}$, $ c(4)\eq {\tt{11}}$.
%
% C4 by itself in a table, moved to graveyard
\marginpar[b]{
\begin{center}
\begin{tabular}{cll} \toprule
% $a_i$
&
$C_4$&
$C_5$
%&$C_6$
% \\
% $c(a_i)$ & $p_i$ &
% \multicolumn{1}{c}{$\log_2 \frac{1}{p_i}$}
% $h(p_i)$ & $l_i$
% {\rule[-3mm]{0pt}{8mm}}%strut
\\ \midrule
{\tt a} & {\tt 00} & {\tt 0} \\
{\tt b} & {\tt 01} & {\tt 1} \\
{\tt c} & {\tt 10} & {\tt 00} \\
{\tt d} & {\tt 11} & {\tt 11} \\
\bottomrule
\end{tabular}
\end{center}
}
The expected length $L(C_4,X)$ is 2 bits.
}
% edskip
%
% \noindent {\sf Example 3:}
\exampla{
Consider $C_5$.
%$ c(1)\eq {\tt{0}}$, $ c(2)\eq {\tt{1}}$, $ c(3)\eq {\tt{00}}$, $c(4)\eq {\tt{11}}$.
The expected
length $L(C_5,X)$ is 1.25 bits, which is less than $H(X)$.
But the code is not uniquely decodeable.
The sequence $\bx\eq ({\tt acdbac})$
% 134213)$
encodes as {\tt{000111000}}, which can also be
decoded as $({\tt cabdca})$.
}
% \medskip
%
% \noindent {\sf Example 4:}
\exampla{
Consider the code $C_6$.
\amargintabnocaption{c}{
\begin{center}
$C_6$:\\[0.1in]
\begin{tabular}{cllcc} \toprule
$a_i$ & $c(a_i)$ & $p_i$ &
% {$\log_2 \frac{1}{p_i}$}
$h(p_i)$
& $l_i$
% {\rule[-3mm]{0pt}{8mm}}%strut
\\ \midrule
{\tt a} & {\tt 0} & \dhalf & 1.0 & 1 \\
{\tt b} & {\tt 01} & \dquarter & 2.0 & 2 \\
{\tt c} & {\tt 011} & \deighth & 3.0 & 3 \\
{\tt d} & {\tt 111} & \deighth & 3.0 & 3 \\
\bottomrule
\end{tabular}
\end{center}
}
%$ c(1)\eq {\tt{0}}$, $ c(2)\eq {\tt{01}}$, $ c(3)\eq {\tt{011}}$, $c(4)\eq {\tt{111}}$.
The expected length $L(C_6,X)$ of this
code is 1.75 bits. The sequence of symbols $\bx\eq ({\tt acdbac})$ is
encoded as $c^+(\bx)={\tt{0011111010011}}$.
Is $C_6$ a {prefix code}?
It is not, because $c({\tt a}) = {\tt 0}$ is a prefix of both
$c({\tt b})$ and $c({\tt c})$.
Is $C_6$ {uniquely decodeable}? This is not so obvious. If you think that
it might {\em not\/} be {uniquely decodeable}, try to prove it
so by finding a pair of strings $\bx$ and $\by$ that have the same
encoding. [The definition of unique decodeability is given in \eqref{eq.UD}.]
$C_6$ certainly isn't {\em easy\/} to decode.
When we receive `{\tt{00}}', it is possible that $\bx$ could start `{\tt{aa}}',
`{\tt{ab}}' or `{\tt{ac}}'. Once we have received `{\tt{001111}}', the second symbol
is still ambiguous, as $\bx$ could be `{\tt{abd}}\ldots' or `{\tt{acd}}\ldots'.
But eventually a unique decoding crystallizes, once the next {\tt{0}} appears in the
encoded stream.
$C_6$ {\em is\/} in fact {uniquely decodeable}. Comparing with the prefix code $C_3$,
we see that the codewords of $C_6$ are the reverse of $C_3$'s.
That $C_3$ is uniquely decodeable proves that $C_6$ is too, since
any string from $C_6$ is identical to a string from $C_3$ read backwards.
}
% \medskip
% something I recall reading in cover was a contrary statement that said that
% with a nonprefix code it will take an arb long time to figure things out.
% maybe that was just a w.c. result.
% What is it that distinguishes a uniquely
\section{What limit is imposed by unique decodeability?}
We now ask, given a list of positive integers $\{ l_i
\}$, does there exist a uniquely decodeable\index{uniquely decodeable}\index{source code!uniquely decodeable} code with those
integers as its codeword lengths?
At this stage, we ignore the probabilities of the different
symbols; once we understand unique decodeability better, we'll
reintroduce the probabilities and discuss how to make
an {\dem optimal\/} uniquely decodeable symbol code.
In the examples above, we have observed that if we take a code
such as $\{{\tt{00}},{\tt{01}},{\tt{10}},{\tt{11}}\}$, and
shorten one of its codewords,
for example ${\tt{00}} \rightarrow {\tt{0}}$, then we can retain unique
decodeability only if we lengthen other codewords.
Thus there seems to be a constrained budget\index{symbol code!budget} that we can spend
on codewords, with shorter codewords being more expensive.
Let us explore the nature of this \ind{budget}.
If we build a code purely from codewords of length $l$ equal
to three, how many
codewords can we have and retain unique decodeability?
The answer is $2^l = 8$. Once we have chosen all eight
of these codewords, is there any way we could add to the code another
codeword of some {\em other\/} length and retain unique decodeability?
It would seem not.
What if we make a code that includes a length-one codeword, `{\tt{0}}',
with the other codewords being of length three? How many length-three
codewords can we have?
If we restrict attention to prefix codes, then
% it is clear that
we can have only four codewords of length three, namely
$\{ {\tt{100}},{\tt{101}},{\tt{110}},{\tt{111}} \}$. What about other codes? Is there any other
way of choosing codewords of length 3 that can give more codewords?
Intuitively, we think this unlikely.
A codeword of length $3$ appears to
have a cost that is $2^{2}$ times smaller than a codeword of length 1.
% "... cost ... times smaller ..."; I suspect some
% readers may have difficulty with this sentence.
Let's define a total budget of size 1,
which we can spend on codewords.
If we set the cost of a codeword whose length is $l$ to $2^{-l}$,
then we have a pricing system that fits the examples
discussed above. Codewords of length 3 cost $\deighth$ each;
codewords of length 1 cost $1/2$ each.
We can spend our budget on any codewords.
If we go over our budget then the code will certainly not be
uniquely decodeable. If, on the other hand,
\beq
\sum_i 2^{-l_i} \leq 1,
\label{eq.kraft}
\eeq
then the code may be uniquely decodeable. This inequality is
the \inds{Kraft inequality}.\label{sec.kraft}
\begin{description}
\item[\Kraft\ inequality\puncspace]
For any uniquely decodeable code $C(X)$ over the binary alphabet $\{0,1\}$,
the codeword lengths must satisfy:
\beq
\sum_{i=1}^I 2^{-l_i} \leq 1 ,
\eeq
where $I = |\A_X|$.
\end{description}
\begin{description}
\item[Completeness\puncspace]
If a uniquely
decodeable code satisfies the \Kraft\ inequality with equality
then it is called a {\dbf complete} code.
\end{description}
% It is less obvious that t
We want codes that are uniquely decodeable;
prefix codes are uniquely decodeable, and are easy to decode.
% ; and it is easy to assess whether a code is a prefix code.
% codes that are not prefix codes are less straightforward to decode than
% prefix codes.
So life would be simpler for us if we could restrict attention to prefix
codes.\index{prefix code}
Fortunately,
% we can prove that
for any source there {\em is\/}
an optimal symbol code that is also a prefix
code.
% We wi, and we will discuss an
% algorithm we can restrict attention to prefix
% codes.
% The following
% result is also true:
\begin{description}
\item[\Kraft\ inequality and prefix codes\puncspace]
Given a set of codeword lengths that satisfy
the Kraft inequality,
% this inequality,
there exists a uniquely decodeable prefix
code\index{source code!prefix code}\index{prefix code} with these
codeword lengths.
\end{description}
\begin{aside}
%\subsection*{The small print}
The Kraft inequality
% , which appears on page \pageref{sec.kraft},
might be more accurately referred to
as the Kraft--McMillan inequality:\index{Kraft, L.G.}\index{McMillan, B.}\nocite{mcmillan1956}
Kraft
% (1949)
proved that if the inequality is satisfied,
then a prefix code exists with the given lengths.
% McMillan
% (1956)
\citeasnoun{mcmillan1956}
proved the converse, that unique decodeability
implies that the inequality holds.
\end{aside}
\begin{prooflike}{Proof of the \Kraft\ inequality}
%
Define $S = \sum_i 2^{-l_i}$.
Consider the quantity
\beq
S^N = \left[ \sum_i 2^{-l_i} \right]^N
= \sum_{i_1=1}^{I} \sum_{i_2=1}^{I} \cdots \sum_{i_N=1}^{I}
2^{-\displaystyle \left(l_{i_1} + l_{i_2} + \cdots l_{i_N} \right) } .
\eeq
The quantity in the exponent, $\left(l_{i_1} + l_{i_2} + \cdots +
l_{i_N} \right)$, is the length of the encoding of the string $\bx =
a_{i_1} a_{i_2} \ldots a_{i_N}$. For every string $\bx$
of length $N$, there is one term in the above sum. Introduce an
array $A_l$ that counts how many strings $\bx$ have encoded length $l$.
Then, defining $l_{\min} = \min_i l_i$ and $l_{\max} = \max_i l_i$:
\beq
S^N = \sum_{l = N l_{\min} }^{N l_{\max}} 2^{-l} A_l .
\eeq
Now assume $C$ is
uniquely decodeable, so that for all $\bx \not = \by$,
$c^+(\bx) \not = c^+(\by)$. Concentrate on the $\bx$ that have encoded
length $l$. There are a total of $2^l$ distinct bit strings of length $l$,
so it must be the case that $A_l \leq 2^l$.
%
So
\beq
S^N = \sum_{l = N l_{\min} }^{N l_{\max}} 2^{-l} A_l \leq
\sum_{l = N l_{\min} }^{N l_{\max}} 1 \:\: \leq \:\: N l_{\max}.
\label{eq.kraft.climax}
\eeq
Thus $S^N \leq l_{\max} N$ for all $N$.
Now if $S$ were greater than 1, then as $N$ increases,
$S^N$ would be an exponentially growing function, and for large enough
$N$, an exponential always exceeds a polynomial such as $l_{\max} N$.
But our result $(S^N \leq l_{\max} N)$
% \ref{eq.kraft.climax}
is true for {\em any\/} $N$.
Therefore $S \leq 1$. \hfill
% Q.E.D.
%
% to have
% enabled me to understand it the first time round, it would have been
% sufficient to have said 'for the inequality to be true for all N,
% regardless of how large, S has to be <= 1.'
%
\end{prooflike}
\exercissxB{3}{ex.KIconverse}{
% (optional)
Prove
the result stated above,
that for any set of codeword lengths $\{ l_i \}$
satisfying the \Kraft\ inequality, there is a prefix code having those
lengths.
}
%
% Symbol Coding Budget
%
\begin{figure}
\figuremargin{%
\begin{center}
\mbox{\psfig{figure=figs/budget1.eps,height=3in}\ \psfig{figure=figs/budgetmax.eps,height=3in}}
\end{center}
}{%
\caption[a]{The symbol coding \ind{budget}.\index{source code!supermarket}\indexs{symbol code!budget}
The `cost' $2^{-l}$ of each codeword
(with length $l$)
is indicated by the size of the box it is written in. The total budget
available when making a uniquely decodeable code is 1.
You can think of this diagram as showing
a {\dem{codeword supermarket}\/}\index{supermarket for codewords},
with the codewords arranged in aisles by their length, and the cost of each codeword indicated by the
size of its box on the shelf.
If the cost of the codewords that you take exceeds the budget then your code
will not be uniquely decodeable.
}
\label{fig.budget1}
}%
\end{figure}
\begin{figure}
\figuredangle{%
\begin{center}
\mbox{
%\begin{tabular}{cc}
% $C_0$ & $C_3$ \\
%\psfig{figure=figs/budget0.eps,height=1.48in}&
%\psfig{figure=figs/budget3.eps,height=1.48in} \\[0.2in]
% $C_4$ & $C_6$ \\
%\psfig{figure=figs/budget4.eps,height=1.48in}&
%\psfig{figure=figs/budget6.eps,height=1.48in}\\
%\end{tabular}}
\begin{tabular}{cccc}
$C_0$ & $C_3$ & $C_4$ & $C_6$ \\
\psfig{figure=figs/budget0.eps,height=1.66in}&
\psfig{figure=figs/budget3.eps,height=1.66in}&
\psfig{figure=figs/budget4.eps,height=1.66in}&
\psfig{figure=figs/budget6.eps,height=1.66in}\\
\end{tabular}}
\end{center}
}{%
\caption[a]{Selections of codewords
% from the codeword supermarket
made by codes $C_0,C_3,C_4$ and $C_6$
from section \protect\ref{sec.symbol.code.intro}.}
\label{fig.budget0}
\label{fig.budget6}
}%
\end{figure}
A pictorial view of the \Kraft\ inequality may help you solve this exercise.
Imagine that we are choosing the codewords to make a symbol code.
We can draw the set of all candidate codewords
% that we might include in a code
in a supermarket that displays
the `cost' of the codeword by the area of a box (\figref{fig.budget1}).
The total budget available -- the `1' on the right-hand side of
the \Kraft\ inequality -- is shown at one side.
Some of the codes discussed in section \ref{sec.symbol.code.intro}
are illustrated in figure \ref{fig.budget0}. Notice that the codes that
are prefix codes, $C_0$, $C_3$,
and $C_4$, have the property that to the right of any selected
codeword, there are no other selected codewords --
because prefix codes correspond to trees.
% The {\em complete\/} prefix codes $C_0$, $C_3$,
% and $C_4$ have the property that
% the codewords abut
% Notice also that the
% `incomplete' code
% -\ref{fig.budget6}.
Notice that a {\em complete\/} prefix code
corresponds to a {\em complete\/} tree having no unused branches.
\medskip
We are now ready to put back the symbols' probabilities $\{ p_i \}$.
Given a set of symbol probabilities (the English language
probabilities of \figref{fig.monogram}, for example),
how do we make the best symbol code -- one with the smallest
possible expected length $L(C,X)$? And what is that smallest possible
expected length?
It's not
obvious how to assign the codeword lengths.
If we give short codewords to the more probable
symbols then the expected length might be reduced; on the other
hand, shortening some codewords necessarily causes others
to lengthen, by the Kraft inequality.
\section{What's the most compression that we can hope for?}
% there must be a compromise.
% of s
% Of the four codes displayed in figure \ref{fig.budget0},
% $C_3$ and $C_6$
We wish to minimize the expected length of a code,
\beqan
L(C,X) &=& \sum_i p_i l_i .
\eeqan
As you might have guessed, the entropy appears as the
% It is easy to show that there is a
lower bound on the expected length of a code.
\begin{description}
\item[Lower bound on expected length\puncspace] The expected length $L(C,X)$
of a uniquely decodeable code
is bounded below by $H(X)$.
\item[{\sf Proof.}]
% Introduce the optimum codelengths $l^*_i \equiv \log (1/p_i)$,
We define the {\dem\inds{implicit probabilities}\/}
$q_i \equiv 2^{-l_i}/z$,
where $z\eq \sum_{i'} 2^{-l_{i'}}$, so that $l_i \eq \log 1/q_i -
\log z$. We then use Gibbs' inequality,
$\sum_i p_i \log 1/q_i \geq \sum_i p_i \log 1/p_i$, with
equality if $q_i \eq p_i$, and the \Kraft\ inequality $z\leq 1$:
\beqan
L(C,X) &=& \sum_i p_i l_i =
\sum_i p_i \log 1/q_i - \log z
\label{eq.expected.length}
\\
& \geq & \sum_i p_i \log 1/p_i - \log z
\\
& \geq & H(X) .
\eeqan
The equality $L(C,X) \eq H(X)$ is achieved only if the \Kraft\
equality $z
% \sum_i 2^{-l_i}
\eq 1$ is satisfied, and if
the codelengths satisfy $l_i \eq \log (1/p_i)$. \hfill $\Box$
\end{description}
This is an important result so let's say it again:
\begin{description}
\item[Optimal source codelengths\puncspace]
The\index{source code!optimal lengths}
expected length is minimized and is equal to
$H(X)$ only if the codelengths
are equal to the {\dem Shannon information contents}:\index{Shannon information content}\index{information content}
\beq
l_i = \log_2 (1/p_i) .
\eeq
\item[Implicit probabilities defined by codelengths\puncspace]
Conversely, any choice of codelengths $\{l_i\}$ {\em implicitly\/}
defines a probability distribution $\{q_i\}$,
\beq
q_i \equiv 2^{-l_i}/z ,
\eeq
for which those codelengths would be the optimal codelengths.
If the code is complete then $z=1$ and the implicit probabilities
are given by $q_i = 2^{-l_i}$.
\end{description}
% This is one of the central themes of this course.
%
%
%
\section{How much can we compress?}
So, we can't compress below the entropy.
% using a symbol code.
How close can we expect to get to the entropy?
% if we are using a symbol code?
% \section{Existence of good symbol codes}
\begin{ctheorem}
{\sf Source coding theorem for symbol codes.}
For an ensemble $X$ there exists a prefix code $C$ with expected length
satisfying\indexs{extra bit}
\beq
H(X) \leq L(C,X) < H(X) + 1.
\label{eq.source.coding.symbol}
\eeq
\label{th.source.coding.symbol}
\end{ctheorem}
\begin{prooflike}{Proof} We set the codelengths to integers slightly
larger than the optimum lengths:
\beq
l_i = \lceil \log_2 (1/p_i) \rceil
\eeq
where $\lceil l^* \rceil$ denotes the smallest integer greater
than or equal to $l^*$.
[We are not asserting that the {\em optimal\/} code necessarily uses
these lengths, we are simply choosing these lengths
because we can use them to prove the theorem.]
We check that there {\em is\/} a
prefix code with these lengths by confirming that the
\Kraft\ inequality is satisfied.
\beq
\sum_i 2^{-l_i} = \sum_i 2^{-\lceil \log_2 (1/p_i) \rceil}
\leq \sum_i 2^{ -\log_2 (1/p_i) } = \sum_i p_i = 1 .
\eeq
Then we confirm
\beq
L(C,X) = \sum_i p_i \lceil \log (1/p_i) \rceil
< \sum_i p_i ( \log (1/p_i) + 1 ) = H(X) + 1.
\eeq
% corrected < to = , 9802
%
\end{prooflike}
\subsection{The cost of using the wrong codelengths}
If we use a code whose lengths are not equal to the optimal
codelengths, the average message length will be larger
than the entropy.
%when we use the `wrong' code.
If the true probabilities are $\{ p_i
\}$ and we use a complete code with lengths $l_i$,
% that satisfy the
% \Kraft\ equality (that is,
% the \Kraft\ inequality with equality),
we can view those lengths as defining
\ind{implicit probabilities} $q_i = 2^{-{l_i}}$.
% l_i \eq \log 1/q_i$ such
% that $\sum_i q_i \eq 1$, then
Continuing from \eqref{eq.expected.length},
the average length is
\beq
L(C,X) = H(X)+\sum_i p_i \log p_i/q_i,
\eeq
\ie, it exceeds the entropy by the \ind{relative entropy}
$D_{\rm KL}(\bp||\bq)$ (as defined on \pref{eq.KL}).
\section{Optimal source coding with symbol codes: Huffman coding}
Given a set of probabilities $\P$, how can we design an optimal
prefix code? For example,
what is the best symbol code for the English language ensemble
shown in \figref{fig.elfig}?
\marginfig{\begin{center}\input{tex/_paz.tex}\end{center}
\caption[a]{An ensemble in need of a symbol code.}\label{fig.elfig}}
When we say `optimal', let's assume our aim is to minimize the
expected length $L(C,X)$.
\subsection{How not to do it}
One might try
to roughly split the set $\A_X$ in two, and
continue bisecting the subsets so as to define a binary tree from the
root. This construction has the right spirit, as in the weighing problem,
% is how the {\em Shannon-Fano code\/} is constructed,\index{Shannon, Claude}\index{Fano}
but it is not
necessarily optimal; it achieves $L(C,X) \leq H(X) + 2$.
%
% find a reference for proof of this?
%
%{\em [Is Shannon-Fano
% the correct name? According to Goldie and Pinch this has a different
% meaning. Check.]}
\subsection{The Huffman coding algorithm}
We now present a beautifully simple algorithm for finding an optimal
prefix code.
\indexs{Huffman code}The trick is to
construct the code {\em backwards\/} starting from the tails of the
codewords; {\em we build the binary tree from its leaves}.
\begin{algorithm}[h]
\begin{framedalgorithmwithcaption}{\caption[a]{Huffman coding algorithm.}}
\ben
\item%[{\sf 1.}]
Take the two least probable symbols in the alphabet. These two symbols
will be given the longest codewords, which will have equal length,
and differ only in the last digit.
\item%[{\sf 2.}]
Combine these two symbols into a single symbol, and repeat.
\een
\end{framedalgorithmwithcaption}
\end{algorithm}
Since each step reduces the size of the alphabet by one,
this algorithm will have assigned strings to all the symbols
after $|\A_X|-1$ steps.
\exampla{
% {\sf Example:}
\begin{tabular}[t]{*{11}{@{\,}l}}
Let \hspace{0.1in} & $\A_X$ &=&$\{$& {\tt a},&{\tt b},&{\tt c},&{\tt d},&{\tt e} &$\}$ \\
and \hspace{0.1in} & $\P_X$ &=&$\{$& 0.25, &0.25, & 0.2, & 0.15, & 0.15 & $\}$.
\end{tabular}
\begin{center}
% \framebox{\psfig{figure=figs/huffman.ps,%
%angle=-90}}
\setlength{\unitlength}{0.015in}%was0125
\begin{picture}(200,95)(40,40)
\put( 60,105){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.25}}}
\put( 60,090){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.25}}}
\put( 60,075){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.2}}}
\put( 60,060){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.15}}}
\put( 60,045){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.15}}}
\put(100,105){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.25}}}
\put(100,090){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.25}}}
\put(100,075){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.2}}}
\put(100,060){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.3}}}
\put(140,105){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.25}}}
\put(140,090){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.45}}}
\put(140,060){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.3}}}
\put(180,105){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.55}}}
\put(180,090){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{0.45}}}
\put(220,105){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{1.0}}}
\put( 40,105){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt a}}}
\put( 40,090){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt b}}}
\put( 40,075){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt c}}}
\put( 40,060){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt d}}}
\put( 40,045){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt e}}}
\put( 85,067){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt 0}}}
\put( 85,045){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt 1}}}
\put(125,097){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt 0}}}
\put(125,075){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt 1}}}
\put(165,112){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt 0}}}
\put(165,065){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt 1}}}
\put(205,112){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt 0}}}
\put(205,090){\makebox(0,0)[lb]{\raisebox{0pt}[0pt][0pt]{\tt 1}}}
\thinlines
\put( 80,110){\line( 1, 0){ 15}}
\put( 80,095){\line( 1, 0){ 15}}
\put( 80,080){\line( 1, 0){ 15}}
\put( 80,065){\line( 1, 0){ 15}}
\put( 95,065){\line(-1,-1){ 15}}
\put(120,110){\line( 1, 0){ 15}}
\put(120,065){\line( 1, 0){ 15}}
\put(120,095){\line( 1, 0){ 15}}
\put(135,095){\line(-1,-1){ 15}}
\put(160,095){\line( 1, 0){ 15}}
\put(160,110){\line( 1, 0){ 15}}
\put(175,110){\line(-1,-3){ 15}}
\put(200,110){\line( 1, 0){ 15}}
\put(215,110){\line(-1,-1){ 15}}
\put( 40,125){\makebox(0,0)[bl]{\raisebox{0pt}[0pt][0pt]{$x$}}}
\put( 85,125){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{step 1}}}
\put(125,125){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{step 2}}}
\put(165,125){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{step 3}}}
\put(205,125){\makebox(0,0)[b]{\raisebox{0pt}[0pt][0pt]{step 4}}}
\end{picture}
\end{center}
The codewords are then obtained by concatenating the binary digits
in reverse order:
% Codewords
$C = \{ {\tt{00}}, {\tt{10}} , {\tt{11}}, {\tt{010}}, {\tt{011}} \}$.
\margintab{
\begin{center}
\begin{tabular}{clrrl} \toprule
$a_i$ & $p_i$ &
\multicolumn{1}{c}{$h(p_i)$%$\log_2 \frac{1}{p_i}$}
}
& $l_i$ & $c(a_i)$
%{\rule[-3mm]{0pt}{8mm}}%strut
\\ \midrule
{\tt a} & 0.25 & 2.0 & 2 & {\tt 00} \\
{\tt b} & 0.25 & 2.0 & 2 & {\tt 10} \\
{\tt c} & 0.2 & 2.3 & 2 & {\tt 11} \\
{\tt d} & 0.15 & 2.7 & 3 & {\tt 010} \\
{\tt e} & 0.15 & 2.7 & 3 & {\tt 011} \\ \bottomrule
\end{tabular}
\end{center}
\caption[a]{Code created by the Huffman algorithm.}
\label{tab.huffman}
}
The codelengths selected by the Huffman algorithm (column 4
of \tabref{tab.huffman}) are
in some cases longer and in some cases shorter than
the ideal codelengths, the Shannon information contents $\log_2 \dfrac{1}{p_i}$ (column 3).
The expected length of the code is $L=2.30$ bits, whereas the
entropy is $H=2.2855$ bits.\ENDsolution
}
If at any point there is more than one way of selecting the two least
probable symbols then the choice may be made in any manner -- the
expected length of the code will not depend on the choice.
\exercissxC{3}{ex.Huffmanconverse}{
% (Optional)
Prove\index{Huffman code!`optimality'}
that there is no better symbol code for a source than the
Huffman code.
}
%
\exampla{
We can make a Huffman code for the probability distribution
over the alphabet introduced in \figref{fig.monogram}.
The result is shown in \figref{fig.monogram.huffman}.
This code has an expected length of 4.15 bits; the entropy of
the ensemble is 4.11 bits.
% It is interesting to notice how
% some symbols, for example {\tt q}, receive codelengths that
% differ by more than 1 bit from
Observe the disparities between the assigned
codelengths and the ideal codelengths
$\log_2 \dfrac{1}{p_i}$.
}
%%%%%%%%%%%%%%%%%%%%%%%%% alphabet of english!
\begin{figure}
\figuremargin{%
\begin{center}
\mbox{\small
\begin{tabular}{clrrl} \toprule
$a_i$ & $p_i$ & \multicolumn{1}{c}{$\log_2 \frac{1}{p_i}$} & $l_i$ & $c(a_i)$
%{\rule[-3mm]{0pt}{8mm}}%strut
\\[0in] \midrule
{\tt a}& 0.0575 & 4.1 & 4 & {\tt 0000 } \\
{\tt b}& 0.0128 & 6.3 & 6 & {\tt 001000 } \\
{\tt c}& 0.0263 & 5.2 & 5 & {\tt 00101 } \\
{\tt d}& 0.0285 & 5.1 & 5 & {\tt 10000 } \\
{\tt e}& 0.0913 & 3.5 & 4 & {\tt 1100 } \\
{\tt f}& 0.0173 & 5.9 & 6 & {\tt 111000 } \\
{\tt g}& 0.0133 & 6.2 & 6 & {\tt 001001 } \\
{\tt h}& 0.0313 & 5.0 & 5 & {\tt 10001 } \\
{\tt i}& 0.0599 & 4.1 & 4 & {\tt 1001 } \\
{\tt j}& 0.0006 & 10.7 & 10 & {\tt 1101000000 } \\
{\tt k}& 0.0084 & 6.9 & 7 & {\tt 1010000 } \\
{\tt l}& 0.0335 & 4.9 & 5 & {\tt 11101 } \\
{\tt m}& 0.0235 & 5.4 & 6 & {\tt 110101 } \\
{\tt n}& 0.0596 & 4.1 & 4 & {\tt 0001 } \\
{\tt o}& 0.0689 & 3.9 & 4 & {\tt 1011 } \\
{\tt p}& 0.0192 & 5.7 & 6 & {\tt 111001 } \\
{\tt q}& 0.0008 & 10.3 & 9 & {\tt 110100001 } \\
{\tt r}& 0.0508 & 4.3 & 5 & {\tt 11011 } \\
{\tt s}& 0.0567 & 4.1 & 4 & {\tt 0011 } \\
{\tt t}& 0.0706 & 3.8 & 4 & {\tt 1111 } \\
{\tt u}& 0.0334 & 4.9 & 5 & {\tt 10101 } \\
{\tt v}& 0.0069 & 7.2 & 8 & {\tt 11010001 } \\
{\tt w}& 0.0119 & 6.4 & 7 & {\tt 1101001 } \\
{\tt x}& 0.0073 & 7.1 & 7 & {\tt 1010001 } \\
{\tt y}& 0.0164 & 5.9 & 6 & {\tt 101001 } \\
{\tt z}& 0.0007 & 10.4 & 10 & {\tt 1101000001 } \\
{--}& 0.1928 & 2.4 & 2 & {\tt 01 } \\ \bottomrule
%{\verb+-+}& 0.1928 & 2.4 & 2 & {\tt 01 } \\ \bottomrule
\end{tabular}
\hspace*{0.5in}\raisebox{-2in}{\psfig{figure=tex/sortedtree.eps,width=1.972in}}
}
\end{center}
}{%
\caption[a]{Huffman code for the English language ensemble (monogram statistics).}
% introduced in \protect\figref{fig.monogram}.}
\label{fig.monogram.huffman}
}%
\end{figure}
% see \cite[p. 97]{Cover&Thomas}
% \medskip
\subsection{Constructing a binary tree top-down is suboptimal}
In previous chapters we studied weighing problems
in which we built ternary or binary trees.
We noticed that balanced trees -- ones in which, at every step, the two
possible outcomes were as close as possible to equiprobable --
appeared to describe the most efficient experiments.
This gave an intuitive motivation for entropy as a measure of information
content.
It is not the case, however, that optimal codes can {\em always\/}
be constructed
by a greedy top-down method in which the alphabet
is successively divided into subsets that are as near as possible to equiprobable.
% /home/mackay/itp/huffman> huffman.p latex=1 < fiftywrong3
\exampla{
Find the optimal binary symbol code for the ensemble:
\beq
\begin{array}{*{3}{@{\,}c@{\,}}*{6}{c@{\,}}*{2}{@{\,}c}}
\A_X & = & \{ &
{\tt a}, &
{\tt b}, &
{\tt c}, &
{\tt d}, &
{\tt e}, &
{\tt f}, &
{\tt g} &
\} \\
\P_X & = & \{
& 0.01,
& 0.24,
& 0.05,
& 0.20,
& 0.47,
& 0.01,
& 0.02
& \} \\
\end{array} .
\eeq
Notice that a greedy top-down method can split this set into two
% equiprobable
subsets
$\{ {\tt a},{\tt b},{\tt c},{\tt d} \}$ and $\{{\tt e},{\tt f},{\tt g}\}$
which both have probability $1/2$,
and that $\{ {\tt a},{\tt b},{\tt c},{\tt d} \}$ can be divided
into
% equiprobable
subsets $\{ {\tt a},{\tt b} \}$ and $\{{\tt c},{\tt d}\}$,
which have probability $1/4$;
so a greedy top-down method gives the code shown
in the third column of \tabref{tab.greed},\margintab{
\begin{center}\small
\begin{tabular}{clll} \toprule
$a_i$ & $p_i$ & Greedy & Huffman \\[0in] \midrule
{\tt a} & .01 & {\tt 000} & {\tt 000000} \\
{\tt b} & .24 & {\tt 001} & {\tt 01} \\
{\tt c} & .05 & {\tt 010} & {\tt 0001} \\
{\tt d} & .20 & {\tt 011} & {\tt 001} \\
{\tt e} & .47 & {\tt 10} & {\tt 1} \\
{\tt f} & .01 & {\tt 110} & {\tt 000001} \\
{\tt g} & .02 & {\tt 111} & {\tt 00001} \\
\bottomrule
\end{tabular}
\end{center}
\caption[a]{A greedily-constructed code compared with the Huffman code.}
\label{tab.greed}
}
which has expected length 2.53.
The Huffman coding algorithm yields the code shown in the fourth
column,
%\begin{center}
%\begin{tabular}{clrrl} \toprule
%$a_i$ & $p_i$ & \multicolumn{1}{c}{$\log_2 \frac{1}{p_i}$} & $l_i$ & $c(a_i)$
%%{\rule[-3mm]{0pt}{8mm}}%strut
%\\[0in] \midrule
%{\tt a} & 0.01 & 6.6 & 6 & {\tt 000000} \\
%{\tt b} & 0.24 & 2.1 & 2 & {\tt 01} \\
%{\tt c} & 0.05 & 4.3 & 4 & {\tt 0001} \\
%{\tt d} & 0.20 & 2.3 & 3 & {\tt 001} \\
%{\tt e} & 0.47 & 1.1 & 1 & {\tt 1} \\
%{\tt f} & 0.01 & 6.6 & 6 & {\tt 000001} \\
%{\tt g} & 0.02 & 5.6 & 5 & {\tt 00001} \\
% \bottomrule
%\end{tabular}
%\end{center}
which has
expected length 1.97.\ENDsolution
% entropy 1.9323
%
}
%\subsection{Twenty questions}
% The Huffman algorithm defines the optimal way to
% play `twenty questions'.
%
% {\em [MORE HERE]}
\section{Disadvantages of the Huffman code}
\label{sec.huffman.probs}
The Huffman\index{Huffman code!disadvantages}\index{symbol code!disadvantages}
algorithm produces an
optimal symbol code for an ensemble, but this is not the end of the
story. Both the word `ensemble' and the phrase `symbol code'
need careful attention.
%\begin{description}
%\item[Changing ensemble.]
\subsection{Changing ensemble}
If we wish to communicate a sequence of outcomes from one
unchanging ensemble, then a Huffman code may be convenient.
But often the appropriate ensemble changes. If for
example we are compressing text, then the symbol frequencies
will vary with context: in English the letter {\tt{u}} is
much more probable after a {\tt{q}} than after an {\tt{e}} (\figref{fig.conbigrams}). And
furthermore, our knowledge of these context-dependent symbol
frequencies will also change as we learn
% accumulate statistics on
the statistical properties of the
text source.\index{adaptive models}
% So our probabilities should change
Huffman codes do not handle changing
ensemble probabilities with any elegance.
One brute-force approach would be to
recompute the Huffman code every time the probability over
symbols changes. Another attitude is to deny the option of
adaptation, and instead run through the entire file in
advance and compute a good probability distribution, which will
then remain fixed throughout transmission. The code itself must
also be communicated in this scenario. Such a technique is
not only cumbersome and restrictive, it is also suboptimal,
since the initial message specifying the code and the document
itself are partially redundant.
% -- knowing the algorithm that
% defines the code for a given document, one can deduce what the
% initial header has to be from the .
This technique therefore wastes bits.
% flag this:
% could discuss bits back here
%
\subsection{The extra bit}
%item[The extra bit.]
An equally serious problem with Huffman codes is the
innocuous-looking `\ind{extra bit}' relative to the ideal average
length of $H(X)$ -- a Huffman code achieves a length that
satisfies $H(X) \leq L(C,X) < H(X) + 1,$ as proved in theorem
\ref{th.source.coding.symbol}.
%\eqref{eq.source.coding.symbol}).
A
Huffman code thus incurs an overhead of between 0 and 1 bits per
symbol. If $H(X)$ were large, then this overhead would be an
unimportant fractional increase. But for many applications,
the entropy may be as low as one bit per symbol, or even smaller,
so the overhead
%`$+1$'
$L(C,X)- H(X)$ may dominate the encoded file length. Consider English
text: in some contexts, long strings of characters may be
highly predictable.
% , as we saw in the guessing game of chapter \chtwo.
% given a simple model of the language.
For
example, in the context `{\verb+strings_of_ch+}', one might
predict the next nine symbols to be `{\verb+aracters_+}' with
a probability of 0.99 each. A traditional Huffman code would
be obliged to use at least one bit per character, making a total cost
of nine bits where virtually no information is being
conveyed (0.13 bits in total, to be precise).
The entropy of English, given a good model, is about
one bit per character \cite{Shannon48}, so a Huffman code is likely to be highly
% nearly 100\%
inefficient.
A traditional patch-up of Huffman codes uses them to compress
{\dem blocks\/} of symbols, for example the `extended sources'
$X^N$ we discussed in \chref{ch.two}.
% \ref{ch2}
% rather than defining a code for single symbols.
The overhead per block is at most 1 bit so the
overhead per symbol
% goes down as
is at most $1/N$ bits. For
sufficiently large blocks, the problem of the extra bit may be
removed -- but only at the expenses of (a) losing the elegant
instantaneous decodeability of simple Huffman coding; and
(b) having
to compute the probabilities of all relevant strings and build
the associated Huffman tree. One will end up explicitly
computing the
probabilities and codes for a huge number of strings, most
of which will never actually occur. (See \exerciseref{ex.Huff99}.)
% A further problem is that it may not be appropriate to model
% successive symbols as coming independently from a single ensemble
% $X$. As we already asserted, any decent model for text will
% assign a probability over symbols that depends on the context.
% A changing probability distribution over symbols is
% not incompatible with the construction of Huffman codes for
% blocks of symbols. One could consider each possible sequence,
% computing the relevant probability distributions along the way
% to evaluate the probability of the entire sequence, then build
% a Huffman tree for the sequences. One could account for
% dependences between blocks as well, if one were willing to
% use a different Huffman code each time. But this modified
% encoder would be
% computationally expensive, since for large block sizes an
% exponentially large number of possible sequences would have
% to be considered along with their adaptive probabilities.
%% is context-dependent.
% \end{description}
% \medskip
\subsection{Beyond symbol codes}
%
Huffman codes, therefore, although widely trumpeted as
`optimal', have many defects for practical
purposes.\index{Huffman code!`optimality'}
They {\em are\/} optimal {\em symbol\/} codes, but for practical
purposes {\em we don't want a symbol code}.
The defects of Huffman codes are rectified by {\dem arithmetic
coding},\index{arithmetic coding} which dispenses with the
restriction that each symbol must translate into an integer
number of bits. Arithmetic coding is the main topic of the next
chapter.
% is not a symbol coding. This
% we will discuss next.
% In an arithmetic code, the probabilistic modelling is clearly
% separated from the encoding operation.
\section{Summary}
\begin{description}
\item[Kraft inequality\puncspace]
If a code is {\dbf uniquely decodeable} its lengths must satisfy
\beq
\sum_i 2^{-l_i } \leq 1 .
\eeq
For any lengths satisfying the Kraft inequality, there exists
a prefix code with those lengths.
\item[Optimal source codelengths for an ensemble] are equal to the
Shannon information contents\index{source code!optimal lengths}\index{source code!implicit probabilities}
\beq
l_i = \log_2 \frac{1}{p_i} ,
\eeq
and conversely, any choice of codelengths defines
{\dbf\ind{implicit probabilities}}
\beq
q_i = \frac{2^{-l_i}}{z} .
\eeq
\item[The \ind{relative entropy}] $D_{\rm KL}(\bp||\bq)$ measures
how many bits per symbol are wasted by using a
% mismatched
code whose implicit probabilities are $\bq$, when
the ensemble's true probability distribution is $\bp$.
\item[Source coding theorem for symbol codes\puncspace]
For an ensemble $X$, there exists a prefix code
whose expected length satisfies
\beq
H(X) \leq L(C,X) < H(X) + 1 .
\eeq
% The expected length is only equal to the entropy if the
\item[The Huffman coding algorithm] generates an optimal symbol code
iteratively. At each iteration, the two least probable symbols are combined.
\end{description}
\section{Exercises}
\exercisaxB{2}{ex.Cnud}{
Is the code $\{ {\tt 00}, {\tt 11}, {\tt 0101}, {\tt 111}, {\tt 1010},
{\tt 100100}, {\tt 0110} \}$
% $\{ 00,11,0101,111,1010,100100,0110 \}$
uniquely decodeable?
}
\exercisaxB{2}{ex.Ctern}{
Is the ternary code
$\{ {\tt 00},{\tt 012},{\tt 0110},{\tt 0112},{\tt 100},{\tt 201},{\tt 212},{\tt 22} \}$ uniquely decodeable?
}
\exercissxA{3}{ex.HuffX2X3}{
Make Huffman codes for $X^2$, $X^3$ and $X^4$ where ${\cal A}_X = \{ 0,1 \}$
and ${\cal P}_X = \{ 0.9,0.1 \}$. Compute their expected lengths and compare
them with the entropies $H(X^2)$, $H(X^3)$ and $H(X^4)$.
Repeat this exercise for $X^2$ and $X^4$ where ${\cal P}_X = \{ 0.6,0.4 \}$.
}
\exercissxA{2}{ex.Huffambig}{
Find a probability distribution $\{ p_1,p_2,p_3,p_4 \}$ such that
there are {\em two\/} optimal codes that assign different lengths $\{ l_i \}$
to the four symbols.
}
\exercisaxC{3}{ex.Huffambigb}{
(Continuation of \exerciseonlyref{ex.Huffambig}.)
Assume that the four probabilities $\{ p_1,p_2,p_3,p_4 \}$ are ordered
such that $p_1 \geq p_2 \geq p_3 \geq p_4 \geq 0$. Let
$\cal Q$ be the set of
all probability vectors $\bp$ such that
there are {\em two\/} optimal codes with different lengths.
Give a complete description of $\cal Q$.
Find three probability vectors $\bq^{(1)}$, $\bq^{(2)}$, $\bq^{(3)}$,
which are the \ind{convex hull} of $\cal Q$, \ie, such that
any $\bp \in \cal Q$ can be written as
\beq
\bp = \mu_1 \bq^{(1)} + \mu_2 \bq^{(2)} +\mu_3 \bq^{(3)} ,
\eeq
where $\{\mu_i\}$ are positive.
}
\exercisaxB{1}{ex.twenty.questions}{
Write a short essay discussing how to play
the game of {\sf{\ind{twenty questions}}} optimally.
[In twenty questions, one player thinks of an object,
and the other player has to guess the object using as few binary
questions as possible, preferably fewer than twenty.]
}
\exercisaxB{2}{ex.powertwogood}{
Show that, if each probability $p_i$ is equal to an integer power of 2
then there exists a source code whose expected length equals the entropy.
}
\exercissxB{2}{ex.make.huffman.suck}{
Make ensembles for which the difference between the entropy
and the expected length of the Huffman code is as big as possible.
}% 14. Gallager, R. G., "Variations on a Theme by Huffman",
% IEEE Trans. on Information Theory, Vol. IT-24, No. 6, Nov. 1978, pp. 668-674.
%
%\exercisxB{2}{ex.huffman.biggerhalf}{
% If one of the probabilities $p_m$ is greater than $1/2$, how
% big must the difference between the expected length and the entropy be?
% Sketch a graph the
%}
% from {tex/huffmanI.tex}
\exercissxB{2}{ex.huffman.uniform}
{
% from 02q.tex on rum
A source $X$ has an alphabet
of eleven characters $$\{ {\tt{a}} , {\tt{b}} , {\tt{c}} , {\tt{d}} , {\tt{e}} , {\tt{f}} , {\tt{g}} , {\tt{h}} , {\tt{i}} , {\tt{j}} , {\tt{k}} \},$$
all of which have equal probability, $1/11$.
% State the meaning of the ideal codelengths
Find an {optimal uniquely decodeable symbol code}
for this source.
How much greater is the expected length of this optimal code
than the entropy of $X$?
}
\exercisaxB{2}{ex.huffman.uniform2}{
Consider the optimal symbol code for an ensemble $X$ with alphabet size
$I$ from which all symbols have identical probability
$p = 1/I$. $I$ is not a power of 2.
Show that the fraction $f^+$ of the $I$ symbols that are assigned
codelengths equal to
\beq
l^+ \equiv \lceil \log_2 I \rceil
\eeq
satisfies
\beq
f^+ = 2 - \frac{2^{l^+}}{I}
\label{eq.HIf}
\eeq
and that the expected length of the optimal symbol code
is
\beq
L = l^+ -1 + f^+ .
\label{eq.HIL}
\eeq
By differentiating
the excess length
%\beq
$ \Delta L \equiv L - H(X)$
%\eeq
with respect to $I$, show that the excess
length is bounded by
\beq
\Delta L \leq 1 - \frac{ \ln ( \ln 2 )}{ \ln 2} -\frac{ 1 }{ \ln 2}
= 0.086 .
\eeq
}
\exercisaxA{2}{ex.Huff99}{
Consider a sparse binary source with ${\cal P}_X = \{ 0.99 , 0.01 \}$.
Discuss how Huffman codes could be used to compress this source
{\em efficiently}.\index{Huffman code}
Estimate how many codewords your proposed solutions require.
% The entropy - hint: could think about run length encoding?
%
}
\exercisaxB{2}{ex.poisonglass}{
% p.111 martin gardner mathematical carnival{Gardner:Carnival}
{\em Scientific American\/} carried the following puzzle\index{puzzle!poisoned glass} in 1975.
% roughly!
\begin{description}
\item[The poisoned glass\puncspace]% This should be \exercisetitlestyle ?
`Mathematicians are curious birds', the police commissioner said to
his wife. `You see, we had all those partly filled glasses lined up
in rows on a table in the hotel kitchen. Only one contained poison,
and we wanted to know which one before searching that glass for
fingerprints. Our lab could test the liquid in each glass, but the
tests take time and money, so we wanted to make as few of them as
possible by simultaneously testing mixtures of small samples from
groups of glasses. The university sent over a mathematics professor
to help us. He counted the glasses, smiled and said:
`$\,$``Pick any glass you want, Commissioner. We'll test it first.''
`$\,$``But won't that waste a test?'' I asked.
`$\,$``No,'' he said, ``it's part of the best procedure. We can test one glass
first. It doesn't matter which one.''$\,$'
`How many glasses were there to start with?' the commissioner's wife asked.
`I don't remember. Somewhere between 100 and 200.'
What was the exact number of glasses?
\end{description}% \cite{Gardner:Carnival}
Solve this puzzle and then explain why the professor was in fact
wrong and the commissioner was right. What is in fact the optimal procedure
for identifying the one poisoned glass? What is the expected waste
relative to this optimum if one followed the professor's strategy?
Explain the relationship to symbol coding.
}
% could get worked up over the all zero codeword, which corresponds to
% a possible non-detection; if this would require an extra test
% then presumably the story is a bit different, with some deliberate
% skewing of the tree to make it more likely that we get a positive
%result along the way.
\exercissxA{2}{ex.optimalcodep1}{% problem fixed Tue 12/12/00
Assume that a sequence of symbols
from the ensemble $X$ introduced at the beginning of this
chapter is compressed using the code $C_3$.
\amarginfignocaption{t}{
\begin{center}
$C_3$:\\[0.1in]
\begin{tabular}{cllcc} \toprule
$a_i$ & $c(a_i)$ & $p_i$ & \multicolumn{1}{c}{$h({p_i})$} & $l_i$
% {\rule[-3mm]{0pt}{8mm}}%strut
\\ \midrule
{\tt a} & {\tt 0} & \dhalf & 1.0 & 1 \\
{\tt b} & {\tt 10} & \dquarter & 2.0 & 2 \\
{\tt c} & {\tt 110} & \deighth & 3.0 & 3 \\
{\tt d} & {\tt 111} & \deighth & 3.0 & 3 \\
\bottomrule
\end{tabular}
\end{center}
}
Imagine picking one bit at random from
the binary encoded sequence $\bc = c(x_1)c(x_2)c(x_3)\ldots$ .
What is the probability that this bit is a 1?
}
\exercissxB{2}{ex.Huffmanqary}{
% (Optional)
How should the\index{Huffman code!general alphabet} binary
Huffman encoding scheme be modified to make optimal symbol codes
in an encoding alphabet with $q$ symbols? (Also known as `\ind{radix} $q$'.)
}
% answer, Hamming p.73:
% add enough states with probability zero to make the total
% number of states equal to $k(q-1)+1$, for some integer $k$.
% then repeatedly combine $q$ into 1
% \end{document}
%
% \item[A code $C(X)$ is {\em non-singular\/}] if every element of $\A_X$
% maps into a different string, \ie,
% \beq
% a_i \not = a_j \Rightarrow c(a_i) \not = c(a_j).
% \eeq
%
% \item[The extension $C^+$ of a code $C$] is a mapping from finite length
% strings of $\A_X$ to $\{0,1\}^+$
% % finite length strings of NAME?
% defined by the concatentation:
% \beq
% c(x_1 x_2 \ldots x_N) = c(x_1)c(x_2)\ldots c(x_N)
% \eeq
%
% \item[A code is uniquely decodeable] if its extension is non-singular.
%
\subsection*{Mixture codes}
It is a tempting idea to construct a `\ind{metacode}' from several symbol
codes that assign different-length codewords to the alternative
symbols, then switch from one
code to another, choosing whichever assigns the shortest codeword
to the current symbol.
Clearly we cannot do this for free.\index{bits back}
If one wishes to choose between two codes, then
it is necessary to lengthen the message in a way that
indicates which of the two codes is being used. If we indicate this
choice by
a single leading bit, it will be found that the resulting code
is suboptimal because it is incomplete (that is,
it fails the Kraft equality).
\exercissxA{3}{ex.mixsubopt}{
Prove that this metacode is incomplete,
and explain why this combined code is
suboptimal.
}
%
% need more on prefix property to make clear how strings are decodeable,
% self-punctuating.
\dvips
\section{Solutions}% to Chapter \protect\ref{ch3}'s exercises}
\fakesection{solns 3}
\soln{ex.C1101}{
Yes,
$C_2 = \{ {\tt{1}} , {\tt{1}}{\tt{0}}{\tt{1}} \}$
% $C_2 = \{ 1 , 101 \}$
is uniquely decodeable, even though
it is not a prefix code, because no two different strings
can map onto the same string; only the codeword $c(a_2)={\tt 101}$ contains
the symbol {\tt0}.
}
\soln{ex.KIconverse}{
We wish to prove that for any set of codeword lengths $\{ l_i \}$
satisfying the \Kraft\ inequality, there is a prefix code having those
lengths.
%
% Symbol Coding Budget -- cut this figure later, it is already in _l3
%
\begin{figure}[htbp]
\figuremargin{%
\begin{center}
\mbox{\psfig{figure=figs/budget1.eps,height=3in}\ \psfig{figure=figs/budgetmax.eps,height=3in}}
\end{center}
}{%
\caption[a]{The codeword supermarket and
the symbol coding budget. The `cost' $2^{-l}$ of each codeword
(with length $l$)
is indicated by the size of the box it is written in. The total budget
available when making a uniquely decodeable code is 1.}
\label{fig.budget1a}
}%
\end{figure}
This is readily proved by thinking of
the codewords illustrated in \figref{fig.budget1a}
as being in a `codeword supermarket', with size indicating
cost. We imagine purchasing\index{source code!supermarket}\index{supermarket for codewords}
codewords one at a time, starting from the shortest codewords (\ie, the biggest
purchases),
using the budget shown at the right of \figref{fig.budget1a}.
We start at one side of the codeword supermarket, say the top,
and purchase the first codeword of the required length. We advance down
the supermarket a distance $2^{-l}$, and purchase the next codeword
of the next required length, and so forth.
Because the codeword lengths are getting longer, and the corresponding
intervals are getting shorter, we can always buy
an adjacent codeword to the latest purchase, so there is no wasting
of the budget. Thus at the $I$th codeword we have advanced
a distance $\sum_{i=1}^{I} 2^{-l_i}$ down the supermarket;
if $\sum 2^{-l_i} \leq 1$, we will have purchased
all the codewords without running out of budget.
}
\soln{ex.Huffmanconverse}{
The proof that Huffman coding is optimal depends on
proving that the key step in the algorithm -- the decision to give
% combination of
the two symbols
with smallest probability equal encoded lengths
-- cannot lead to a larger expected length
than any other code. We can prove this by contradiction.
Assume that
the two symbols with smallest probability, called $a$ and $b$,
to which the Huffman algorithm would assign equal length
codewords,
do {\em not\/} have equal lengths in {\em any\/}
optimal symbol code.
The optimal symbol code is some
other rival code in which these two codewords
have unequal lengths $l_a$ and $l_b$ with $l_a < l_b$.
Without loss of
generality we can assume that this other code is a complete prefix code,
because any codelengths of a uniquely decodeable code
can be realized by a prefix code.
% We now consider transforming the other code into a new code
% in which we interchange \ldots
In this rival code, there must be some other symbol $c$ whose
probability $p_c$ is greater than $p_a$ and whose length
in the rival code is greater than or equal to $l_b$, because
the code for $b$ must have an adjacent codeword of equal or greater
length -- a complete prefix code never has a solo codeword
of the maximum length.
\begin{figure}%[htbp]
\figuremargin{%
\begin{tabular}{llllll} \toprule % \hline
symbol & \multicolumn{2}{c}{probability} & Huffman & Rival code's & Modified rival \\
& & & codewords & codewords & code \\ \midrule % [0.1in]\hline
$a$ & $p_a$ & \framebox[0.15in]{} & \framebox[1.50cm]{$c_{\rm H}(a)$} & \framebox[1.0cm]{$c_{\rm R}(a)$} & \framebox[1.6cm]{$c_{\rm R}(c)$}
\\[0.1in]
$b$ & $p_b$ & \framebox[0.1in]{} & \framebox[1.50cm]{$c_{\rm H}(b)$} & \framebox[1.5cm]{$c_{\rm R}(b)$} & \framebox[1.5cm]{$c_{\rm R}(b)$}
\\[0.1in]
$c$ & $p_c$ & \framebox[0.25in]{} & \framebox[0.95cm]{$c_{\rm H}(c)$} & \framebox[1.6cm]{$c_{\rm R}(c)$} & \framebox[1.0cm]{$c_{\rm R}(a)$}
\\ \bottomrule % [0.1in] \hline
\end{tabular}
}{%
\caption[a]{Proof that Huffman coding makes an optimal symbol code.
% The proof works by contradiction.
We assume that the rival code, which is said to be optimal, assigns {\em unequal\/} length
codewords to the two symbols with smallest probability, $a$ and $b$.
By interchanging codewords $a$ and $c$ of the rival code, where $c$ is a
symbol with rival codelength as long as $b$'s, we can make
a code better than the rival code. This shows that the rival code
was not optimal.
}
\label{fig.huffman.optimal}
}%
\end{figure}
Consider exchanging the codewords of $a$ and $c$ (\figref{fig.huffman.optimal}), so that
$a$ is encoded with the longer codeword that was $c$'s, and
$c$, which is more probable than $a$, gets the shorter codeword.
Clearly this reduces the expected length of the code.
The change in expected length is $(p_a-p_c)(l_c-l_a)$.
Thus we have contradicted the assumption that the rival code is optimal.
Therefore it is valid to give the two symbols
with smallest probability equal encoded lengths.
Huffman coding produces optimal symbol codes.\ENDsolution
}
%\soln{ex.Cnud}{
%\soln{ex.Ctern}{
\soln{ex.HuffX2X3}{
A Huffman code
for $X^2$ where ${\cal A}_X = \{ {\tt 0},{\tt 1} \}$
and ${\cal P}_X = \{ 0.9,0.1 \}$
is $\{{\tt 00},{\tt 01},{\tt 10},{\tt 11}\} \rightarrow
\{{\tt 1},{\tt 01},{\tt 000},{\tt 001}\}$.
This code has $L(C,X^2) = 1.29$, whereas the entropy $H(X^2)$ is 0.938.
A Huffman code for $X^3$ is
\[
\begin{array}{c}
\{{\tt 000},{\tt 100},{\tt 010},{\tt 001},{\tt 101},{\tt 011},{\tt 110},{\tt 111}\}
\rightarrow\\
\hspace*{1in} \{{\tt 1},{\tt 011},{\tt 010},{\tt 001},
{\tt 00000},{\tt 00001},{\tt 00010},{\tt
00011}\}.
\end{array}
\]
% corrected from 1.472 to 1.598
% 9802
This has expected length $L(C,X^3) = 1.598$ whereas the entropy $H(X^3)$
is 1.4069.
A Huffman code for $X^4$ maps the sixteen source strings to the
following codelengths:
\[
\begin{array}{c}
\{ {\tt 0000},{\tt 1000},{\tt 0100},{\tt 0010},{\tt 0001},{\tt 1100},{\tt 0110},{\tt 0011},{\tt 0101},
{\tt 1010},{\tt 1001},{\tt 1110},{\tt 1101}, \\
{\tt 1011},{\tt 0111},{\tt 1111} \}
\rightarrow \:\: \{ 1,3,3,3,4,6,7,7,7,7,7,9,9,9,10,10 \}.
% 10,10,9,9,9,7,7,7,7,7,6,4,3, 3,3,1\}.
\end{array}
\]
This has expected length $L(C,X^4) = 1.9702$ whereas the entropy $H(X^4)$
is 1.876.
%
% 0.6,0.4
When ${\cal P}_X = \{ 0.6,0.4 \}$, the Huffman code for $X^2$ has lengths
$\{ 2,2,2,2 \}$; the expected length is 2 bits, and the
entropy is 1.94 bits. A
Huffman code for $X^4$ is shown in \tabref{fig.X4huff2}.
% , has lengths
% $\{0000,1000,0100,0010,0001,1100,0110,0011,0101,1010,1001,1110,1101,1011,0111,1111\} \rightarrow$
% $\{3,3,4,4,4,4,4,4,4,4,4,4,5,5,5,5\}$.
The expected length is 3.92 bits, and the entropy is 3.88 bits.
% see tmp3 for soln using huffman.p
% $\{0000,1000,0100,0010,0001,1100,0110,0011,0101,1010,1001,1110,1101,1011,0111,1111\} \rightarrow \{5,5,5,5,4,4,4,4,4,4,4,4,4,4,3,3\}$.
}
% see tmp3 for use of huffman.p
%\begin{figure}
%\figuremargin{%
\margintab{\footnotesize
\begin{center}
\begin{tabular}{clrl} \toprule % \hline
$a_i$ & $p_i$ &
% \multicolumn{1}{c}{$h({p_i})$} &
$l_i$ & $c(a_i)$
% {\rule[-3mm]{0pt}{8mm}}%strut
% \\[0.1in] \hline
\\ \midrule
{\tt 0000} & 0.1296 & 3 & {\tt 000 }\\
{\tt 0001} & 0.0864 & 4 & {\tt 0100 }\\
{\tt 0010} & 0.0864 & 4 & {\tt 0110 }\\
{\tt 0100} & 0.0864 & 4 & {\tt 0111 }\\
{\tt 1000} & 0.0864 & 3 & {\tt 100 }\\
{\tt 1100} & 0.0576 & 4 & {\tt 1010 }\\
{\tt 1010} & 0.0576 & 4 & {\tt 1100 }\\
{\tt 1001} & 0.0576 & 4 & {\tt 1101 }\\
{\tt 0110} & 0.0576 & 4 & {\tt 1110 }\\
{\tt 0101} & 0.0576 & 4 & {\tt 1111 }\\
{\tt 0011} & 0.0576 & 4 & {\tt 0010 }\\
{\tt 1110} & 0.0384 & 5 & {\tt 00110 }\\
{\tt 1101} & 0.0384 & 5 & {\tt 01010 }\\
{\tt 1011} & 0.0384 & 5 & {\tt 01011 }\\
{\tt 0111} & 0.0384 & 4 & {\tt 1011 }\\
{\tt 1111} & 0.0256 & 5 & {\tt 00111 }\\ \bottomrule %\hline
%expected length 3.9248
%entropy 3.8838
\end{tabular}
\end{center}
%}{%
\caption[a]{Huffman code for $X^4$ when $p_0=0.6$. Column 3 shows the
assigned codelengths and column 4 the codewords. Some strings
whose probabilities are identical, \eg, the fourth and fifth,
receive different codelengths.}
\label{fig.X4huff2}
}%
%\end{figure}
\soln{ex.Huffambig}{
The set of probabilities $\{ p_1,p_2,p_3,p_4 \} =
\{ \dsixth,\dsixth,\dthird,\dthird\}$ gives rise to two different optimal
sets of codelengths, because at the second step of the Huffman
coding algorithm we can choose any of the three possible pairings.
We may either put them in a constant length code
$\{ {\tt00},{\tt01},{\tt10},{\tt11} \}$ or
the code $\{ {\tt000},{\tt001},{\tt01},{\tt1} \}$.
Both codes have expected length 2.
Another solution is $\{ p_1,p_2,p_3,p_4 \}$ $=$
$\{ \dfifth,\dfifth,\dfifth,\dtwofifth\}$.
% =$ $\{ 0.2 , 0.2 , 0.2 , 0.4 \} $.
And a third is $\{ p_1,p_2,p_3,p_4 \} =
\{ \dthird,\dthird,\dthird,0\}$.
}
\soln{ex.make.huffman.suck}{
Let $p_{\max}$ be the largest probability in $p_1,p_2,\ldots,p_I$.
The difference between the expected length
$L$ and the entropy $H$ can be no bigger than
$\max ( p_{\max} , 0.086 )$ \cite{Gallager78}.
%
See exercises \ref{ex.huffman.uniform}--\ref{ex.huffman.uniform2} to understand
where the curious 0.086 comes from.
}
\soln{ex.huffman.uniform}{
% removed to cutsolutions.tex
Length $-$ entropy = 0.086.
%length / entropy 1.0249
}
% \soln{ex.Huff99}{
% BORDERLINE
\soln{ex.optimalcodep1}{% problem fixed Tue 12/12/00
There are two ways to answer this problem correctly,
and one popular way to answer it incorrectly.
Let's give the incorrect answer first:
\begin{description}
\item[Erroneous answer\puncspace]
``We can pick a random bit by first picking a
random source symbol $x_i$ with probability $p_i$,
then picking a random bit from $c(x_i)$. If we define $f_i$
to be the fraction of the bits of $c(x_i)$ that are {\tt 1}s,
we find
\marginpar[b]{\small
\begin{center}
$C_3$:
\begin{tabular}{cllc} \toprule
$a_i$ & $c(a_i)$ & $p_i$ & $l_i$
\\ \midrule
{\tt a} & {\tt 0} & \dhalf & 1 \\
{\tt b} & {\tt 10} & \dquarter & 2 \\
{\tt c} & {\tt 110} & \deighth & 3 \\
{\tt d} & {\tt 111} & \deighth & 3 \\
\bottomrule
\end{tabular}
\end{center}
}
\beqan
\!\!\!\!\!\!\!\!\!\!
P(\mbox{bit is {\tt 1}}) &=& \sum_i p_i f_i
\label{eq.wrongp1}
\\ &=&
\dfrac{1}{2} \times 0 +
\dfrac{1}{4} \times \dfrac{1}{2} +
\dfrac{1}{8} \times \dfrac{2}{3} +
\dfrac{1}{8} \times 1
= \dthird \mbox{.''}
\eeqan
\end{description}
This answer is wrong because it falls for the \index{bus-stop paradox}{bus-stop fallacy},\index{paradox}
which was introduced in \exerciseref{ex.waitbus}: if buses arrive
at random, and we are interested in `the average time from one bus until
the next', we must distinguish two possible averages:
(a) the average time from a randomly chosen bus until the next;
(b) the average time between the bus you just missed and the next bus.
The second `average' is twice as big as the first because,
by waiting for a bus at a random time, you bias your selection of
a bus in favour of buses that follow a large gap. You're unlikely
to catch a bus that comes 10 seconds after a preceding bus!
Similarly, the symbols {\tt c} and {\tt d} get encoded into
longer-length binary strings than {\tt a}, so when we pick a bit
from the compressed string at random, we are more likely
to land in a bit belonging to a {\tt c} or a {\tt d}
than would be given by the probabilities $p_i$ in the
expectation (\ref{eq.wrongp1}). All the probabilities need to
be scaled up by $l_i$, and renormalized.
\begin{description}
\item[Correct answer in the same style\puncspace]
Every time symbol $x_i$ is encoded, $l_i$ bits
are added to the binary string, of which $f_i l_i$ are {\tt 1}s.
The expected number of {\tt 1}s added per symbol is
\beq
\sum_i p_i f_i l_i ;
\eeq
and the expected total number of bits added per symbol is
\beq
\sum_i p_i l_i .
\eeq
So the fraction of {\tt 1}s in the transmitted string is
\beqan
P(\mbox{bit is {\tt 1}}) &=& \frac{ \sum_i p_i f_i l_i }{ \sum_i p_i l_i }
\label{eq.rightp1}
\\ &=&
\frac{ \dfrac{1}{2} \times 0 +
\dfrac{1}{4} \times 1 +
\dfrac{1}{8} \times 2 +
\dfrac{1}{8} \times 3
}{ \dfrac{7}{4} }
= \frac{\dfrac{7}{8}}{\dfrac{7}{4}} = 1/2 .
\nonumber
\eeqan
\end{description}
For a general symbol code and a general ensemble,
the expectation (\ref{eq.rightp1}) is the correct answer.
But in this case, we can use a more powerful argument.
\begin{description}
\item[Information-theoretic answer\puncspace]
The encoded string $\bc$ is the output of
an optimal compressor that compresses samples from
$X$ down to an expected length of $H(X)$ bits. We can't expect to compress
this data any further. But if the probability $P(\mbox{bit is {\tt 1}})$
were not equal to $\dhalf$ then it {\em would\/} be possible to compress
the binary string further (using a block compression code, say).
Therefore $P(\mbox{bit is {\tt 1}})$
must be equal to $\dhalf$; indeed the probability of any sequence
of $l$ bits in the compressed stream taking on any particular
value must be $2^{-l}$. The output of a perfect compressor is always
perfectly random bits.
\begincuttable
To put it another way, if the probability $P(\mbox{bit is {\tt 1}})$
were not equal to $\dhalf$, then the information content per bit of
the compressed string would be at most $H_2( P(\mbox{{\tt 1}}) )$,
which would be less than 1;
but this contradicts the fact that we can recover the original data
from $\bc$, so the information content per bit of the
compressed string must be $H(X)/L(C,X)=1$.
\ENDcuttable
\end{description}
}
%
% this one is a new addition
%
\soln{ex.Huffmanqary}{ The \index{Huffman code!general alphabet}{general Huffman coding algorithm} for
an encoding alphabet with $q$ symbols
has one difference from the binary case.
The process of combining $q$ symbols into
1 symbol reduces the number of symbols by $q\!-\!1$.
So if we start with $A$ symbols, we'll only end up
with a complete $q$-ary tree if $A \mod (q\!-\!1)$ is equal
to 1.
Otherwise, we know that whatever prefix code we make, it
must be an incomplete tree with a number of missing
leaves equal, modulo $(q\!-\!1)$, to $A \mod (q\!-\!1) - 1$.
For example, if a ternary tree is built for eight symbols,
then there will unavoidably be one missing leaf in the tree.
The optimal $q$-ary code is made by putting these
extra leaves in the longest branch of the tree. This can be achieved
by adding the appropriate number of symbols to the original source
symbol set, all of these extra symbols having probability zero.
The total number of leaves is then equal to $r(q\!-\!1)+1$, for some
integer $r$.
The symbols are then repeatedly combined by taking
the $q$ symbols with smallest probability and replacing them
by a single symbol, as in the binary Huffman coding algorithm.}
\soln{ex.mixsubopt}{
%This is important but I haven't written it yet.
We wish to show that a greedy \ind{metacode}, which
picks the code which gives the shortest encoding, is
actually suboptimal, because it violates the Kraft
inequality.
% For generality, let's call the
% that the objects to be encoded,
% $x$, `symbols'.
We'll assume that each symbol $x$ is
assigned lengths $l_k(x)$ by each of the candidate codes $C_k$.
Let us assume there are $K$ alternative codes and that we can
encode which code is being used with a header of length $\log K$
bits.
Then the metacode assigns lengths $l'(x)$ that are given by
\beq
l'(x) = \log_2 K + \min_k l_k(x) .
\eeq
We compute the Kraft sum:
\beq
S = \sum_x 2^{- l'(x)}
= \frac{1}{K} \sum_x 2^{- \min_k l_k(x)} .
\eeq
Let's divide the set $\A_X$ into non-overlapping subsets $\{\A_k\}_{k=1}^{K}$
such that subset $\A_k$ contains all the symbols $x$
that the metacode sends via code $k$.
Then
\beq
S = \frac{1}{K} \sum_k \sum_{x \in \A_{k}} 2^{- l_k(x)} .
\eeq
Now if one sub-code $k$ satisfies the Kraft equality
$\sum_{x\in \A_X} 2^{- l_k(x)} \eq 1$, then
it must be the case that
\beq
\sum_{x \in \A_{k}} 2^{- l_k(x)} \leq 1 ,
\label{eq.from.kraft}
\eeq
with equality only if all the symbols $x$ are in $\A_k$, which would mean that we
are only using one of the $K$ codes.
So
\beq
S \leq \frac{1}{K} \sum_{k=1}^K 1 = 1 ,
\eeq
with equality only if \eqref{eq.from.kraft} is an equality for all codes $k$.
But it's impossible for all the symbols to be in {\em all\/} the
non-overlapping subsets $\{\A_k\}_{k=1}^{K}$, so
we can't have equality (\ref{eq.from.kraft}) holding
for {\em all\/} $k$.
So
%\beq
% S < 1 .
%\eeq
$S < 1$.
Another way of seeing that a mixture code is suboptimal is to consider
the binary tree that it defines. Think of the special case of two
codes. The first bit we send identifies which code we are using.
Now, in a complete code, any subsequent binary string is a valid
string. But once we know that we are using, say, code A, we know that
what follows can only be a codeword corresponding to a symbol $x$
whose encoding is shorter under code A than code B. So some strings
are invalid continuations, and the mixture code is incomplete
and suboptimal.
%%% MAYBE!!!!!!!!!!!!!!
For further discussion of this issue
and its relationship to probabilistic modelling
read about `\ind{bits back} coding' in \secref{sec.bitsback}
and in \citeasnoun{frey-98}.
}
% \dvipsb{solutions 3}
\prechapter{About Chapter}
\fakesection{prerequisites for chapter known as 4}
Before reading \chref{ch.four}, you should have read the previous chapter
and worked on
most of the exercises in it.
We'll also make use of some Bayesian modelling ideas
that arrived in the vicinity of \exerciseref{ex.postpa}.
% Arithmetic coding has been invented several times,
% by Elias, by Rissanen, and
% but is only slowly becoming well known
%
% {The description of Lempel--Ziv coding is based on that of Cover and Thomas (1991).}
%\chapter{Data Compression III: Stream Codes}
\mysetcounter{page}{126}
\ENDprechapter
\chapter{Stream Codes}
\label{ch.four}
\label{ch.ac}
% _l4.tex
\fakesection{Data Compression III: Stream Codes}
%
% still need to change notation for R(|)
%
\label{ch4}
In this chapter we discuss two data
compression schemes.\index{source code!stream codes|(}\index{stream codes|(}
%% that constitute the state of the art.
{\dem\indexs{arithmetic coding}Arithmetic coding}
is a beautiful method that goes
hand in hand with the philosophy that compression of data
from a source entails
probabilistic modelling of that source. As of 1999,
the best compression methods for text files use arithmetic coding,
and several state-of-the-art image compression systems
use it too.
{\dem\ind{Lempel--Ziv coding}} is a `\ind{universal}' method,
% in my opinion an ugly hack, but
designed under the philosophy that we would like a single compression
algorithm that will do a reasonable job for {\em any\/} source.
In fact, for many real
life sources, this
algorithm's universal properties hold only
in the limit of unfeasibly large amounts of data, but,
all the same, Lempel--Ziv compression is widely used
and often effective.
\section{The guessing game}
\label{sec.startofch4}
% \looseness=-1 this did not achieve what was advertised!
As a motivation for these\index{game!guessing}
two compression methods,
% let us
consider the redundancy in a typical
% imagine compressing a
\ind{English} text file. Such files have redundancy at several levels: for example,
they contain the ASCII characters with non-equal frequency; certain consecutive
pairs of letters are more probable than others; and entire words
can be predicted given the context and a semantic understanding
of the text.
To illustrate the redundancy of English, and a curious way in which
it could be compressed, we can imagine a \ind{guessing game}
in which an English speaker repeatedly
attempts to predict the next character
in a text file.
% \subsection{The guessing game}
\label{sec.guessing}
% Could discuss the compression of English text by guessing
For simplicity, let us assume that the allowed alphabet consists
of
the 26 upper case letters {\tt A,B,C,\ldots, Z} and a space `{\tt -}'.
The game involves asking the subject to guess the next character
repeatedly, the only feedback being whether the guess is correct
or not, until the character is correctly guessed.
After a correct guess, we note the number of guesses that
were made when the character was identified, and ask the subject
to guess the next character in the same way.
One sentence
% given by Shannon
gave the following result when a human was asked to guess a sentence.
% in a guessing game.
The numbers of guesses
are listed below each character.\index{reverse}\index{motorcycle}
% and the idea of having an identical twin. This introduces the idea
% of mapping to a different alphabet with nonuniform probability.
% The guessing game. From Shannon.
\smallskip
\begin{center}\hspace*{0.3in}
%\begin{tabular}{*{36}{c@{\,\,}}}
\begin{tabular}{*{36}{p{0.15in}@{}}}
\small\tt
T&\small\tt H&\small\tt E&\small\tt R&\small\tt E&\small\tt -&\small\tt I&\small\tt S&\small\tt -&\small\tt N&\small\tt O&\small\tt -&\small\tt R&\small\tt E&\small\tt V&\small\tt E&\small\tt R&\small\tt S&\small\tt E&\small\tt -&\small\tt O&\small\tt N&\small\tt -&\small\tt A&\small\tt -&\small\tt M&\small\tt O&\small\tt T&\small\tt O&\small\tt R&\small\tt C&\small\tt Y&\small\tt C&\small\tt L&\small\tt E&\small\tt -\\
\footnotesize
1&\footnotesize 1&\footnotesize 1&\footnotesize 5&\footnotesize 1&\footnotesize 1&\footnotesize 2&\footnotesize 1&\footnotesize 1&\footnotesize 2&\footnotesize 1&\footnotesize 1&\footnotesize \hspace{-0.05in}1\hspace{-0.25mm}5&\footnotesize 1&\footnotesize \hspace{-0.05in}1\hspace{-0.25mm}7&\footnotesize 1&\footnotesize 1&\footnotesize 1&\footnotesize 2&\footnotesize 1&\footnotesize 3&\footnotesize 2&\footnotesize 1&\footnotesize 2&\footnotesize 2&\footnotesize 7&\footnotesize 1&\footnotesize 1&\footnotesize 1&\footnotesize 1&\footnotesize 4&\footnotesize 1&\footnotesize 1&\footnotesize 1&\footnotesize 1&\footnotesize 1\\
\end{tabular}
\smallskip
\end{center}
% attempt to tighten this para:
\looseness=-1
Notice that in many cases, the next letter is guessed immediately, in one guess.
In other cases, particularly at the start of syllables,
more guesses are needed.
What do this game and these results offer us?
First, they demonstrate the redundancy of English from the point of
view of an English speaker.
Second, this game might be used in
a data compression scheme, as follows.
% encoding
The string of numbers `1, 1, 1, 5, 1, \ldots', listed above,
was obtained by presenting
the text to the subject. The maximum number of guesses that the
subject will make for a given letter is twenty-seven, so what the subject is
doing for us is performing a time-varying mapping of the twenty-seven letters
$\{ {\tt A,B,C,\ldots, Z,-}\}$ onto the twenty-seven numbers $\{1,2,3,\ldots,
27\}$, which we can view as symbols in a new alphabet. The total number of
symbols has not been reduced, but since he uses some of
these symbols much more frequently than others -- for example, 1 and
2 -- it should be easy to compress this new string of
symbols.
% ; we will discuss data compression
%% the details of how to do this
% properly shortly.
% decoding
How would the {\em uncompression\/} of the sequence of numbers
`1, 1, 1, 5, 1, \ldots' work? At uncompression time,
we do not have the original string `{\small\tt{THERE}}\ldots', we
have only the encoded sequence. Imagine that our subject has an
absolutely \ind{identical twin}\index{twin}
%({\em absolutely\/} identical)
who also
plays the guessing game\index{guessing game} with us, as if we
%, the experimenters,
knew the source text.
If we stop him whenever he has made a
number of guesses equal to the given number, then he will have just
guessed the correct letter, and we can then say `yes, that's right',
and move to the next character.
Alternatively, if the identical twin is not available, we could
design a compression system with the help of just one human as follows.
We choose a window length $L$, that is, a number of characters of context
to show the human. For every one of the $27^L$ possible
strings of length $L$, we ask them, `What would you predict is the next character?',
and `If that prediction were wrong, what would your next guesses be?'.
After tabulating their answers to these $26 \times 27^L$ questions,
we could use two copies of these enormous tables at the encoder and the
decoder in place of the two human twins.
Such a language model is called an $L$th order \ind{Markov model}.
These systems are clearly unrealistic for practical compression,
but they illustrate several principles that we will make use of now.
\section{Arithmetic codes}
\label{sec.ac}
% In lecture 2 we discussed fixed length block codes.
When we discussed variable-length symbol codes, and the optimal
Huffman algorithm for constructing them, we concluded by pointing
out two practical
and theoretical problems with Huffman codes (section \ref{sec.huffman.probs}).
%
% index decision: {arithmetic coding} not {arithmetic codes}
%
These defects are rectified by {\dem\index{arithmetic coding}{arithmetic codes}}, which
were invented by Elias\nocite{EliasACmentionedpages61to62},\index{Elias, Peter}
by \index{Rissanen, Jorma}{Rissanen} and by \index{Pasco, Richard}{Pasco},
and subsequently made practical by
% Witten, Neal, and Cleary.
\citeasnoun{arith_coding}.\index{Neal, Radford}
In an arithmetic code, the
probabilistic modelling is clearly separated from the encoding
operation.
The system is rather similar to the guessing game.\index{guessing game}
% that we considered in Chapter \chtwo.
The human predictor is replaced by a
{\dem\ind{probabilistic model}} of the source.
As each symbol is produced by the source, the probabilistic model
supplies a {\dem\ind{predictive distribution}}
over all possible values of the next
symbol, that is, a list of positive numbers $\{ p_i \}$ that sum to
one. If we choose to model the source as producing i.i.d.\ symbols with some
known distribution,
then the predictive distribution is the same every time; but arithmetic
coding can with equal ease handle complex adaptive models that produce
context-dependent
% time-varying
predictive distributions. The predictive model is usually
implemented in a computer program.
% a model which hypothesizes arbitrary
% context-dependences and non-stationarities, and which learns as it
% goes, so that predictive distributions in any given context gradually
% sharpen up.
% I will give an example later on,
% of an adaptive model producing appropriate probabilities
% but first let us discuss the arithmetic coding algorithm itself.
The encoder makes use of the model's predictions to create a
binary string. The decoder makes use of an identical twin of the
model (just as in the guessing \index{guessing game}game) to interpret the binary string.
Let the source alphabet be $\A_X = \{a_1 ,\ldots, a_I\}$, and let the
$I$th symbol $a_I$ have the special meaning `end of transmission'.
The source
spits out a sequence $x_1,x_2,\ldots,x_n,\ldots.$ The source does {\em not\/}
necessarily produce i.i.d.\ symbols.
We will assume that a computer program is provided to the encoder
that assigns a predictive
probability distribution over $a_i$ given the sequence that has occurred
thus far,
$P(x_n \eq a_i \given x_1,\ldots,x_{n-1})$.
% Nor will we assume that the source
% is correctly modeled by $P$. But if it is, then arithmetic coding achieves
% the Shannon rate.
%
% The encoder will send a binary transmission to the receiver.
%
The receiver has an identical program that produces the
same predictive
probability distribution $P(x_n \eq a_i \given x_1,\ldots,x_{n-1})$.
% and uses it to interpret the received message.
\medskip
\begin{figure}[htbp]
\figuremargin{%
\begin{center}
\setlength{\unitlength}{1mm}
\begin{picture}(50,40)(0,0)
\put(18,40){\makebox(0,0)[r]{0.00}}
\put(18,30){\makebox(0,0)[r]{0.25}}
\put(18,20){\makebox(0,0)[r]{0.50}}
\put(18,10){\makebox(0,0)[r]{0.75}}
\put(18, 0){\makebox(0,0)[r]{1.00}}
%
% major horizontals
%
\put(20,40){\line(1,0){37}}
\put(20,30){\line(1,0){13}}
\put(20,20){\line(1,0){28}}
\put(20,10){\line(1,0){13}}
\put(20, 0){\line(1,0){37}}
%
% biggest intervals
%
\put(45,30){\vector(0,1){9}}
\put(45,30){\vector(0,-1){9}}
\put(47,30){\makebox(0,0)[l]{{\tt{0}}}}
\put(45,10){\vector(0,1){9}}
\put(45,10){\vector(0,-1){9}}
\put(47,10){\makebox(0,0)[l]{{\tt{1}}}}
%
\put(35,25){\vector(0,1){4}}
\put(35,25){\vector(0,-1){4}}
\put(37,25){\makebox(0,0)[l]{{\tt{01}}}}
% some subdivs
\put(20,35){\line(1,0){7}}
\put(20,25){\line(1,0){7}}
\put(20,15){\line(1,0){7}}
\put(20, 5){\line(1,0){7}}
%
% 01101 = 13/32 = 16.25
% 01110 = 14/32 = 17.5
\put(20,23.75){\line(1,0){4}}
\put(20,22.50){\line(1,0){4}}
\put(62,23.125){\makebox(0,0)[l]{{\tt{01101}}}}
%
% interrupted pointer:
\put(60,23.125){\line(-1,0){14}}
\put(44,23.125){\line(-1,0){8}}
\put(34,23.125){\vector(-1,0){9.5}}
%
\end{picture}
\end{center}
}{%
\caption[a]{Binary strings define real intervals within the real line [0,1).
We first encountered a picture like this when we discussed the
\index{supermarket for codewords}\index{symbol code!supermarket}\index{source code!supermarket}{symbol-code supermarket} in \chref{ch3}.
}
\label{fig.arith.Rbinary}
}%
\end{figure}
\subsection{Concepts for understanding arithmetic coding}
\begin{aside}
%\item[Notation for intervals.]
{\sf Notation for intervals.} The interval $[0.01, 0.10)$ is all numbers
between $0.01$ and $0.10$, including $0.01\dot{0}\equiv0.01000\ldots$ but not $0.10\dot{0}\equiv0.10000\ldots.$
\end{aside}
A binary transmission defines an interval within
the real line from 0 to 1. For example, the string {\tt{01}} is
interpreted as a binary real number 0.01\ldots, which corresponds to
the interval $[0.01, 0.10)$ in binary, \ie, the interval
$[0.25,0.50)$ in base ten.
%
% why strange line breaks?
%
The longer string {\tt{01101}} corresponds to a smaller
interval $[0.01101,$ $0.01110)$. Because {\tt{01101}} has the first string,
{\tt{01}}, as a prefix, the new interval is a sub-interval
of the interval $[0.01, 0.10)$.
A one-megabyte binary file ($2^{23}$ bits) is thus viewed as specifying a number
between 0 and 1 to a precision of about two million
% $10^7$
decimal places -- {two million decimal digits, because
each byte translates into a little more than two decimal digits.}
% byte = 8 bits ~= 2 digits.
%
% one meg-byte = 2^3 * 2^20 = 2^23 binary places -> 2.5*10^7 or (2**23=8388608) .
% shall I tell you a bedtime number between 0 and 1 to 10^7 d.p. darling?
%
\medskip
Now, we can also
% Similarly, we can
divide the real line [0,1) into $I$ intervals of
lengths equal to the probabilities $P(x_1 \eq a_i)$, as shown
in \figref{fig.arith.R}.
% upsidedown
% p1 = 6 -- 34 mid: 37 w = 3-1
% p2 = 16 cum 22 -- 18 mid: 26 w = 8-1
% last = 6 cum -- 6 mid: 3 w = 3-1
\newcommand{\aonelevel}{34}
\newcommand{\atwolevel}{18}
\newcommand{\apenlevel}{6}% penultimate
\newcommand{\apenmid}{12}% put dots here
\newcommand{\aonemid}{37}
\newcommand{\aonew}{2}
\newcommand{\atwow}{7}
\newcommand{\atwomid}{26}
\newcommand{\aIw}{2}
\newcommand{\aImid}{3}
\begin{figure}[htbp]
\figuremargin{%
\begin{center}
\setlength{\unitlength}{1mm}
\begin{picture}(50,40)(0,0)
\put(18,40){\makebox(0,0)[r]{0.00}}
\put(18,\aonelevel){\makebox(0,0)[r]{$P(x_1\eq a_1)$}}
\put(18,\atwolevel){\makebox(0,0)[r]{$P(x_1\eq a_1)+P(x_1\eq a_2)$}}
\put(18,\apenlevel){\makebox(0,0)[r]{$P(x_1\eq a_1)+\ldots+P(x_1\eq a_{I\!-\!1})$}}
\put(18, 0){\makebox(0,0)[r]{1.0}}
%
% major horizontals
%
\put(20,40){\line(1,0){37}}
\put(20,\aonelevel){\line(1,0){20}}
\put(20,\atwolevel){\line(1,0){20}}
\put(20,\apenlevel){\line(1,0){20}}
\put(20, 0){\line(1,0){37}}
\put(30,\apenmid){\makebox(0,0)[l]{$\vdots$}}
%
% biggest intervals
%
\put(35,\aonemid){\vector(0,1){\aonew}}
\put(35,\aonemid){\vector(0,-1){\aonew}}
\put(37,\aonemid){\makebox(0,0)[l]{$a_1$}}% or $P(x_1\eq a_1)$}}
\put(35,\atwomid){\vector(0,1){\atwow}}
\put(35,\atwomid){\vector(0,-1){\atwow}}
\put(37,\atwomid){\makebox(0,0)[l]{$a_2$}}% or $P(x_1\eq a_2)$}}
\put(35,\aImid){\vector(0,1){\aIw}}
\put(35,\aImid){\vector(0,-1){\aIw}}
\put(37,\aImid){\makebox(0,0)[l]{$a_I$}}% or $P(x_1\eq a_I)$}}
\put(37,\apenmid){\makebox(0,0)[l]{$\vdots$}}
%
\put(20,23){\line(1,0){4}}% beg of a5
\put(20,20){\line(1,0){4}}% end a5
%
\put(62,21.5){\makebox(0,0)[l]{$a_2 a_5$}}
% interrupted pointer:
\put(60,21.5){\line(-1,0){24}}
\put(34,21.5){\vector(-1,0){9.5}}
%
% a2a1: 34 is the top
%
\put(20,30){\line(1,0){4}}% end of a1
\put(20,28){\line(1,0){4}}% end of a2
\put(20,25){\line(1,0){4}}% end of a3
%
\put(62,32){\makebox(0,0)[l]{$a_2 a_1$}}
% interrupted pointer:
\put(60,32){\line(-1,0){24}}
\put(34,32){\vector(-1,0){9.5}}
%
\end{picture}
\end{center}
}{%
\caption[a]{A probabilistic model defines real
intervals within the real line [0,1).}
\label{fig.arith.R}
}%
\end{figure}
We may then take each interval $a_i$ and subdivide it into intervals
denoted $a_ia_1,a_ia_2,\ldots, a_ia_I$, such that the length of
$a_ia_j$ is proportional to $P(x_2 \eq a_j \given x_1 \eq a_i)$. Indeed the
length of the interval $a_ia_j$ will be precisely the joint probability
\beq
P(x_1 \eq
a_i,x_2\eq a_j)=P(x_1\eq a_i)P(x_2 \eq a_j \given x_1 \eq a_i).
\eeq
Iterating this procedure, the interval $[0,1)$ can be divided
into a sequence of intervals corresponding to all possible finite
length strings $x_1x_2\ldots x_N$, such that the length of an
interval is equal to the probability of the string given our model.
% This iterative procedure
\subsection{Formulae describing arithmetic coding}
\begin{aside}
The process depicted in \figref{fig.arith.R} can be written
explicitly as follows.
The intervals are defined in terms of the lower and upper cumulative probabilities
\beqan
Q_{n}(a_i \given x_1,\ldots,x_{n-1})
& \equiv & \sum_{i'\eq 1 }^{i-1} P(x_n \eq a_{i'} \given x_1,\ldots,x_{n-1}) ,
\label{eq.arith.Q} \\
R_{n}(a_i \given x_1,\ldots,x_{n-1})
& \equiv & \sum_{i'\eq 1 }^{i} P(x_n \eq a_{i'} \given x_1,\ldots,x_{n-1}) .
\label{eq.arith.R}
\eeqan
%
As the $n$th symbol arrives, we subdivide the $n-1$th
interval at the points defined by $Q_n$ and $R_n$.
For example, starting with the first symbol,
the intervals `$a_1$', `$a_2$',
% `$a_3$',
and `$a_I$' are
% first interval,
% which we will call
\beq
a_1 \leftrightarrow [Q_{1}(a_1),R_{1}(a_1))= [0,P(x_1 \eq a_1)) ,
\eeq
\beq
a_2 \leftrightarrow [Q_{1}(a_2),R_{1}(a_2))=
\left[
P(x\eq a_1),P(x\eq a_1)+P(x\eq a_2) \right) ,
\eeq
%\beq
% a_3 \leftrightarrow [Q_{1}(a_3),R_{1}(a_3))=
% \left[
% P(x\eq a_1)+P(x\eq a_2) , P(x\eq a_1)+P(x\eq a_2) +P(x\eq a_3)\right),
%\eeq
and
\beq
a_I \leftrightarrow
\left[ Q_{1}(a_{I}) , R_{1}(a_{I}) \right)
= \left[ P(x_1\eq a_1)+\ldots+P(x_1\eq a_{I\!-\!1}) ,1.0 \right) .
\eeq
Algorithm \ref{alg.ac} describes the general procedure.
\end{aside}
\begin{algorithm}
\begin{framedalgorithmwithcaption}{
\caption[a]{Arithmetic coding.
Iterative procedure to find the interval $[u,v)$
for the string $x_1x_2\ldots x_N$.
}
\label{alg.ac}
}
%\algorithmmargin{%
\begin{center}
\begin{tabular}{l}
%\begin{description}% should be ALGORITHM
%\item[Iterative procedure to find the interval $[u,v)$
% corresponding to
% for the string $x_1x_2\ldots x_N$]
%
{\tt $u$ := 0.0} \\
{\tt $v$ := 1.0} \\
{\tt $p$ := $v-u$} \\
{\tt for $n$ = 1 to $N$ \verb+{+ } \\
\hspace*{0.5in} Compute the cumulative probabilities $Q_n$ and $R_n$
\protect(\ref{eq.arith.Q},\,\ref{eq.arith.R})
% $\{ R_{n}(a_i \given x_1,\ldots,x_{n-1}) \}_{i=1}^{I}$
%% $\{ R_{n,i \given x_1,\ldots,x_{n-1}} \}_{i=0}^{I}$
% using \eqref{eq.arith.R} \\
\\
\hspace*{0.5in} {\tt $v$ := $u + p R_{n}(x_n \given x_1,\ldots,x_{n-1}) $ } \\
\hspace*{0.5in} {\tt $u$ := $u + p Q_{n}(x_n \given x_1,\ldots,x_{n-1}) $ } \\
\hspace*{0.5in} {\tt $p$ := $v-u$} \\
% {\tt ) } \\
{\tt \verb+}+ } \\
\end{tabular}
\end{center}
%\end{description}
%}
\end{framedalgorithmwithcaption}
\end{algorithm}
To encode a string $x_1x_2\ldots x_N$,
we locate the interval corresponding to $x_1x_2\ldots x_N$, and
send a binary string whose interval lies within
that interval. This encoding can be performed
on the fly, as we now illustrate.
% \eof defined in itprnnchapter
\subsection{Example: compressing the tosses of a bent coin}
Imagine that we watch as a bent coin is tossed some number of times (\cf\
\exampleref{exa.bentcoin} and \secref{sec.bentcoin}
(\pref{sec.bentcoin})).
The two outcomes when the coin is tossed
are denoted $\tt a$ and $\tt b$. A third possibility is that the
experiment is halted, an event denoted by the `end of file' symbol, `$\eof$'.
Because the coin is bent, we expect that the probabilities of the outcomes $\tt a$
and $\tt b$ are not equal, though beforehand we don't know which
is the more probable outcome.
% Let $\A_X=\{a,b,\eof\}$, where
% $a$ and $\tb$ make up a binary alphabet with
% $\eof$ is an `end of file' symbol.
\subsubsection{Encoding\subsubpunc}
Let the source string be `$\tt bbba\eof$'. We pass along the string one symbol
at a time and use our model to compute the probability
distribution of the next symbol given the string thus far.
Let these probabilities be:
\[\begin{array}{l*{3}{r@{\eq}l}} \toprule
\mbox{Context } \\
\mbox{(sequence thus far) }
& \multicolumn{6}{c}{\mbox{Probability of next symbol}} \\[0.05in] \midrule
& P( \ta ) & 0.425 & P( \tb ) & 0.425 & P( \eof ) & 0.15 \\[0.05in]
\tb& P( \ta \given \tb ) & 0.28 & P( \tb \given \tb ) & 0.57 & P( \eof \given \tb ) & 0.15 \\[0.05in]
\tb\tb&P( \ta \given \tb\tb ) & 0.21 & P( \tb \given \tb\tb ) & 0.64 & P( \eof \given \tb\tb ) & 0.15 \\[0.05in]
\tb\tb\tb&P( \ta \given \tb\tb\tb ) & 0.17 & P( \tb \given \tb\tb\tb ) & 0.68 & P( \eof \given \tb\tb\tb ) & 0.15 \\[0.05in]
\tb\tb\tb\ta& P( \ta \given \tb\tb\tb\ta ) & 0.28 & P( \tb \given \tb\tb\tb\ta ) & 0.57 & P( \eof \given \tb\tb\tb\ta ) & 0.15 \\ \bottomrule
\end{array}
\]
\Figref{fig.ac} shows the corresponding intervals. The
interval $\tb$ is the middle 0.425 of $[0,1)$. The interval $\tb\tb$ is the
middle 0.567 of $\tb$, and so forth.
% in the following figure.
\begin{figure}[htbp]
\figuremargin{%
\begin{center}
% created by ac.p only_show_data=1 > ac/ac_data.tex %%%%%%% and edited by hand
\mbox{
\hspace{-0.1in}\small
\setlength{\unitlength}{4.8in}
%\setlength{\unitlength}{5.75in}
\begin{picture}(0.59130434782608698452,1)(-0.29565217391304349226,0)
\thinlines
% line 0.0000 from -0.5000 to 0.0000
\put( -0.2957, 1.0000){\line(1,0){ 0.2957}}
% a at -0.4500, 0.2125
\put( -0.2811, 0.7875){\makebox(0,0)[r]{\tt{a}}}
% line 0.4250 from -0.5000 to 0.0000
\put( -0.2957, 0.5750){\line(1,0){ 0.2957}}
% b at -0.4500, 0.6375
\put( -0.2811, 0.3625){\makebox(0,0)[r]{\tt{b}}}
% line 0.8500 from -0.5000 to 0.0000
\put( -0.2957, 0.1500){\line(1,0){ 0.2957}}
% \teof at -0.4500, 0.9250
\put( -0.2811, 0.0750){\makebox(0,0)[r]{\teof}}
% line 1.0000 from -0.5000 to 0.0000
\put( -0.2957, 0.0000){\line(1,0){ 0.2957}}
% ba at -0.3500, 0.4852
\put( -0.2220, 0.5148){\makebox(0,0)[r]{\tt{ba}}}
% line 0.5454 from -0.4500 to 0.0000
\put( -0.2661, 0.4546){\line(1,0){ 0.2661}}
% bb at -0.3500, 0.6658
\put( -0.2220, 0.3342){\makebox(0,0)[r]{\tt{bb}}}
% line 0.7862 from -0.4500 to 0.0000
\put( -0.2661, 0.2138){\line(1,0){ 0.2661}}
% b\teof at -0.3500, 0.8181
\put( -0.2220, 0.1819){\makebox(0,0)[r]{\tt{b\teof}}}
% bba at -0.2300, 0.5710
\put( -0.1510, 0.4290){\makebox(0,0)[r]{\tt{bba}}}
% line 0.5966 from -0.3500 to 0.0000
\put( -0.2070, 0.4034){\line(1,0){ 0.2070}}
% bbb at -0.2300, 0.6734
\put( -0.1510, 0.3266){\makebox(0,0)[r]{\tt{bbb}}}
% line 0.7501 from -0.3500 to 0.0000
\put( -0.2070, 0.2499){\line(1,0){ 0.2070}}
% bb\teof at -0.2300, 0.7682
\put( -0.1510, 0.2318){\makebox(0,0)[r]{\tt{bb\teof}}}
% bbba at -0.1000, 0.6096
\put( -0.0741, 0.3904){\makebox(0,0)[r]{\tt{bbba}}}
% line 0.6227 from -0.2300 to 0.0000
\put( -0.1360, 0.3773){\line(1,0){ 0.1360}}
% bbbb at -0.1000, 0.6749
\put( -0.0741, 0.3251){\makebox(0,0)[r]{\tt{bbbb}}}
% line 0.7271 from -0.2300 to 0.0000
\put( -0.1360, 0.2729){\line(1,0){ 0.1360}}
% bbb\teof at -0.1000, 0.7386
\put( -0.0741, 0.2614){\makebox(0,0)[r]{\tt{bbb\teof}}}
% line 0.6040 from -0.1000 to 0.0000
\put( -0.0591, 0.3960){\line(1,0){ 0.0591}}
% line 0.6188 from -0.1000 to 0.0000
\put( -0.0591, 0.3812){\line(1,0){ 0.0591}}
% line 0.0000 from 0.0100 to 0.5000
\put( 0.0059, 1.0000){\line(1,0){ 0.2897}}
% 0 at 0.0100, 0.2500
\put( 0.2811, 0.7500){\makebox(0,0)[l]{\tt0}}
% line 0.5000 from 0.0100 to 0.5000
\put( 0.0059, 0.5000){\line(1,0){ 0.2897}}
% 1 at 0.0100, 0.7500
\put( 0.2811, 0.2500){\makebox(0,0)[l]{\tt1}}
% line 1.0000 from 0.0100 to 0.5000
\put( 0.0059, 0.0000){\line(1,0){ 0.2897}}
% 00 at 0.0100, 0.1250
\put( 0.2397, 0.8750){\makebox(0,0)[l]{\tt00}}
% line 0.2500 from 0.0100 to 0.4500
\put( 0.0059, 0.7500){\line(1,0){ 0.2602}}
% 01 at 0.0100, 0.3750
\put( 0.2397, 0.6250){\makebox(0,0)[l]{\tt01}}
% 000 at 0.0100, 0.0625
\put( 0.1806, 0.9375){\makebox(0,0)[l]{\tt000}}
% line 0.1250 from 0.0100 to 0.3800
\put( 0.0059, 0.8750){\line(1,0){ 0.2188}}
% 001 at 0.0100, 0.1875
\put( 0.1806, 0.8125){\makebox(0,0)[l]{\tt001}}
% 0000 at 0.0100, 0.0312
% was at 0.1037, move 0.02 right -> 1207
\put( 0.1207, 0.9688){\makebox(0,0)[l]{\tt0000}}
% line 0.0625 from 0.0100 to 0.2800
\put( 0.0059, 0.9375){\line(1,0){ 0.1597}}
% 0001 at 0.0100, 0.0938
\put( 0.1207, 0.9062){\makebox(0,0)[l]{\tt0001}}
% 00000 at 0.0100, 0.0156
\put( 0.0387, 0.9844){\makebox(0,0)[l]{\tt00000}}
% line 0.0312 from 0.0100 to 0.1500
\put( 0.0059, 0.9688){\line(1,0){ 0.0828}}
% 00001 at 0.0100, 0.0469
\put( 0.0387, 0.9531){\makebox(0,0)[l]{\tt00001}}
% line 0.0156 from 0.0100 to 0.0400
\put( 0.0059, 0.9844){\line(1,0){ 0.0177}}
% line 0.0078 from 0.0100 to 0.0200
\put( 0.0059, 0.9922){\line(1,0){ 0.0059}}
% line 0.0234 from 0.0100 to 0.0200
\put( 0.0059, 0.9766){\line(1,0){ 0.0059}}
% line 0.0469 from 0.0100 to 0.0400
\put( 0.0059, 0.9531){\line(1,0){ 0.0177}}
% line 0.0391 from 0.0100 to 0.0200
\put( 0.0059, 0.9609){\line(1,0){ 0.0059}}
% line 0.0547 from 0.0100 to 0.0200
\put( 0.0059, 0.9453){\line(1,0){ 0.0059}}
% 00010 at 0.0100, 0.0781
\put( 0.0387, 0.9219){\makebox(0,0)[l]{\tt00010}}
% line 0.0938 from 0.0100 to 0.1500
\put( 0.0059, 0.9062){\line(1,0){ 0.0828}}
% 00011 at 0.0100, 0.1094
\put( 0.0387, 0.8906){\makebox(0,0)[l]{\tt00011}}
% line 0.0781 from 0.0100 to 0.0400
\put( 0.0059, 0.9219){\line(1,0){ 0.0177}}
% line 0.0703 from 0.0100 to 0.0200
\put( 0.0059, 0.9297){\line(1,0){ 0.0059}}
% line 0.0859 from 0.0100 to 0.0200
\put( 0.0059, 0.9141){\line(1,0){ 0.0059}}
% line 0.1094 from 0.0100 to 0.0400
\put( 0.0059, 0.8906){\line(1,0){ 0.0177}}
% line 0.1016 from 0.0100 to 0.0200
\put( 0.0059, 0.8984){\line(1,0){ 0.0059}}
% line 0.1172 from 0.0100 to 0.0200
\put( 0.0059, 0.8828){\line(1,0){ 0.0059}}
% 0010 at 0.0100, 0.1562
\put( 0.1207, 0.8438){\makebox(0,0)[l]{\tt0010}}
% line 0.1875 from 0.0100 to 0.2800
\put( 0.0059, 0.8125){\line(1,0){ 0.1597}}
% 0011 at 0.0100, 0.2188
\put( 0.1207, 0.7812){\makebox(0,0)[l]{\tt0011}}
% 00100 at 0.0100, 0.1406
\put( 0.0387, 0.8594){\makebox(0,0)[l]{\tt00100}}
% line 0.1562 from 0.0100 to 0.1500
\put( 0.0059, 0.8438){\line(1,0){ 0.0828}}
% 00101 at 0.0100, 0.1719
\put( 0.0387, 0.8281){\makebox(0,0)[l]{\tt00101}}
% line 0.1406 from 0.0100 to 0.0400
\put( 0.0059, 0.8594){\line(1,0){ 0.0177}}
% line 0.1328 from 0.0100 to 0.0200
\put( 0.0059, 0.8672){\line(1,0){ 0.0059}}
% line 0.1484 from 0.0100 to 0.0200
\put( 0.0059, 0.8516){\line(1,0){ 0.0059}}
% line 0.1719 from 0.0100 to 0.0400
\put( 0.0059, 0.8281){\line(1,0){ 0.0177}}
% line 0.1641 from 0.0100 to 0.0200
\put( 0.0059, 0.8359){\line(1,0){ 0.0059}}
% line 0.1797 from 0.0100 to 0.0200
\put( 0.0059, 0.8203){\line(1,0){ 0.0059}}
% 00110 at 0.0100, 0.2031
\put( 0.0387, 0.7969){\makebox(0,0)[l]{\tt00110}}
% line 0.2188 from 0.0100 to 0.1500
\put( 0.0059, 0.7812){\line(1,0){ 0.0828}}
% 00111 at 0.0100, 0.2344
\put( 0.0387, 0.7656){\makebox(0,0)[l]{\tt00111}}
% line 0.2031 from 0.0100 to 0.0400
\put( 0.0059, 0.7969){\line(1,0){ 0.0177}}
% line 0.1953 from 0.0100 to 0.0200
\put( 0.0059, 0.8047){\line(1,0){ 0.0059}}
% line 0.2109 from 0.0100 to 0.0200
\put( 0.0059, 0.7891){\line(1,0){ 0.0059}}
% line 0.2344 from 0.0100 to 0.0400
\put( 0.0059, 0.7656){\line(1,0){ 0.0177}}
% line 0.2266 from 0.0100 to 0.0200
\put( 0.0059, 0.7734){\line(1,0){ 0.0059}}
% line 0.2422 from 0.0100 to 0.0200
\put( 0.0059, 0.7578){\line(1,0){ 0.0059}}
% 010 at 0.0100, 0.3125
\put( 0.1806, 0.6875){\makebox(0,0)[l]{\tt010}}
% line 0.3750 from 0.0100 to 0.3800
\put( 0.0059, 0.6250){\line(1,0){ 0.2188}}
% 011 at 0.0100, 0.4375
\put( 0.1806, 0.5625){\makebox(0,0)[l]{\tt011}}
% 0100 at 0.0100, 0.2812
\put( 0.1207, 0.7188){\makebox(0,0)[l]{\tt0100}}
% line 0.3125 from 0.0100 to 0.2800
\put( 0.0059, 0.6875){\line(1,0){ 0.1597}}
% 0101 at 0.0100, 0.3438
\put( 0.1207, 0.6562){\makebox(0,0)[l]{\tt0101}}
% 01000 at 0.0100, 0.2656
\put( 0.0387, 0.7344){\makebox(0,0)[l]{\tt01000}}
% line 0.2812 from 0.0100 to 0.1500
\put( 0.0059, 0.7188){\line(1,0){ 0.0828}}
% 01001 at 0.0100, 0.2969
\put( 0.0387, 0.7031){\makebox(0,0)[l]{\tt01001}}
% line 0.2656 from 0.0100 to 0.0400
\put( 0.0059, 0.7344){\line(1,0){ 0.0177}}
% line 0.2578 from 0.0100 to 0.0200
\put( 0.0059, 0.7422){\line(1,0){ 0.0059}}
% line 0.2734 from 0.0100 to 0.0200
\put( 0.0059, 0.7266){\line(1,0){ 0.0059}}
% line 0.2969 from 0.0100 to 0.0400
\put( 0.0059, 0.7031){\line(1,0){ 0.0177}}
% line 0.2891 from 0.0100 to 0.0200
\put( 0.0059, 0.7109){\line(1,0){ 0.0059}}
% line 0.3047 from 0.0100 to 0.0200
\put( 0.0059, 0.6953){\line(1,0){ 0.0059}}
% 01010 at 0.0100, 0.3281
\put( 0.0387, 0.6719){\makebox(0,0)[l]{\tt01010}}
% line 0.3438 from 0.0100 to 0.1500
\put( 0.0059, 0.6562){\line(1,0){ 0.0828}}
% 01011 at 0.0100, 0.3594
\put( 0.0387, 0.6406){\makebox(0,0)[l]{\tt01011}}
% line 0.3281 from 0.0100 to 0.0400
\put( 0.0059, 0.6719){\line(1,0){ 0.0177}}
% line 0.3203 from 0.0100 to 0.0200
\put( 0.0059, 0.6797){\line(1,0){ 0.0059}}
% line 0.3359 from 0.0100 to 0.0200
\put( 0.0059, 0.6641){\line(1,0){ 0.0059}}
% line 0.3594 from 0.0100 to 0.0400
\put( 0.0059, 0.6406){\line(1,0){ 0.0177}}
% line 0.3516 from 0.0100 to 0.0200
\put( 0.0059, 0.6484){\line(1,0){ 0.0059}}
% line 0.3672 from 0.0100 to 0.0200
\put( 0.0059, 0.6328){\line(1,0){ 0.0059}}
% 0110 at 0.0100, 0.4062
\put( 0.1207, 0.5938){\makebox(0,0)[l]{\tt0110}}
% line 0.4375 from 0.0100 to 0.2800
\put( 0.0059, 0.5625){\line(1,0){ 0.1597}}
% 0111 at 0.0100, 0.4688
\put( 0.1207, 0.5312){\makebox(0,0)[l]{\tt0111}}
% 01100 at 0.0100, 0.3906
\put( 0.0387, 0.6094){\makebox(0,0)[l]{\tt01100}}
% line 0.4062 from 0.0100 to 0.1500
\put( 0.0059, 0.5938){\line(1,0){ 0.0828}}
% 01101 at 0.0100, 0.4219
\put( 0.0387, 0.5781){\makebox(0,0)[l]{\tt01101}}
% line 0.3906 from 0.0100 to 0.0400
\put( 0.0059, 0.6094){\line(1,0){ 0.0177}}
% line 0.3828 from 0.0100 to 0.0200
\put( 0.0059, 0.6172){\line(1,0){ 0.0059}}
% line 0.3984 from 0.0100 to 0.0200
\put( 0.0059, 0.6016){\line(1,0){ 0.0059}}
% line 0.4219 from 0.0100 to 0.0400
\put( 0.0059, 0.5781){\line(1,0){ 0.0177}}
% line 0.4141 from 0.0100 to 0.0200
\put( 0.0059, 0.5859){\line(1,0){ 0.0059}}
% line 0.4297 from 0.0100 to 0.0200
\put( 0.0059, 0.5703){\line(1,0){ 0.0059}}
% 01110 at 0.0100, 0.4531
\put( 0.0387, 0.5469){\makebox(0,0)[l]{\tt01110}}
% line 0.4688 from 0.0100 to 0.1500
\put( 0.0059, 0.5312){\line(1,0){ 0.0828}}
% 01111 at 0.0100, 0.4844
\put( 0.0387, 0.5156){\makebox(0,0)[l]{\tt01111}}
% line 0.4531 from 0.0100 to 0.0400
\put( 0.0059, 0.5469){\line(1,0){ 0.0177}}
% line 0.4453 from 0.0100 to 0.0200
\put( 0.0059, 0.5547){\line(1,0){ 0.0059}}
% line 0.4609 from 0.0100 to 0.0200
\put( 0.0059, 0.5391){\line(1,0){ 0.0059}}
% line 0.4844 from 0.0100 to 0.0400
\put( 0.0059, 0.5156){\line(1,0){ 0.0177}}
% line 0.4766 from 0.0100 to 0.0200
\put( 0.0059, 0.5234){\line(1,0){ 0.0059}}
% line 0.4922 from 0.0100 to 0.0200
\put( 0.0059, 0.5078){\line(1,0){ 0.0059}}
% 10 at 0.0100, 0.6250
\put( 0.2397, 0.3750){\makebox(0,0)[l]{\tt10}}
% line 0.7500 from 0.0100 to 0.4500
\put( 0.0059, 0.2500){\line(1,0){ 0.2602}}
% 11 at 0.0100, 0.8750
\put( 0.2397, 0.1250){\makebox(0,0)[l]{\tt11}}
% 100 at 0.0100, 0.5625
\put( 0.1806, 0.4375){\makebox(0,0)[l]{\tt100}}
% line 0.6250 from 0.0100 to 0.3800
\put( 0.0059, 0.3750){\line(1,0){ 0.2188}}
% 101 at 0.0100, 0.6875
\put( 0.1806, 0.3125){\makebox(0,0)[l]{\tt101}}
% 1000 at 0.0100, 0.5312
\put( 0.1207, 0.4688){\makebox(0,0)[l]{\tt1000}}
% line 0.5625 from 0.0100 to 0.2800
\put( 0.0059, 0.4375){\line(1,0){ 0.1597}}
% 1001 at 0.0100, 0.5938
\put( 0.1207, 0.4062){\makebox(0,0)[l]{\tt1001}}
% 10000 at 0.0100, 0.5156
\put( 0.0387, 0.4844){\makebox(0,0)[l]{\tt10000}}
% line 0.5312 from 0.0100 to 0.1500
\put( 0.0059, 0.4688){\line(1,0){ 0.0828}}
% 10001 at 0.0100, 0.5469
\put( 0.0387, 0.4531){\makebox(0,0)[l]{\tt10001}}
% line 0.5156 from 0.0100 to 0.0400
\put( 0.0059, 0.4844){\line(1,0){ 0.0177}}
% line 0.5078 from 0.0100 to 0.0200
\put( 0.0059, 0.4922){\line(1,0){ 0.0059}}
% line 0.5234 from 0.0100 to 0.0200
\put( 0.0059, 0.4766){\line(1,0){ 0.0059}}
% line 0.5469 from 0.0100 to 0.0400
\put( 0.0059, 0.4531){\line(1,0){ 0.0177}}
% line 0.5391 from 0.0100 to 0.0200
\put( 0.0059, 0.4609){\line(1,0){ 0.0059}}
% line 0.5547 from 0.0100 to 0.0200
\put( 0.0059, 0.4453){\line(1,0){ 0.0059}}
% 10010 at 0.0100, 0.5781
\put( 0.0387, 0.4219){\makebox(0,0)[l]{\tt10010}}
% line 0.5938 from 0.0100 to 0.1500
\put( 0.0059, 0.4062){\line(1,0){ 0.0828}}
% 10011 at 0.0100, 0.6094
\put( 0.0387, 0.3906){\makebox(0,0)[l]{\tt10011}}
% line 0.5781 from 0.0100 to 0.0400
\put( 0.0059, 0.4219){\line(1,0){ 0.0177}}
% line 0.5703 from 0.0100 to 0.0200
\put( 0.0059, 0.4297){\line(1,0){ 0.0059}}
% line 0.5859 from 0.0100 to 0.0200
\put( 0.0059, 0.4141){\line(1,0){ 0.0059}}
% line 0.6094 from 0.0100 to 0.0400
\put( 0.0059, 0.3906){\line(1,0){ 0.0177}}
% line 0.6016 from 0.0100 to 0.0200
\put( 0.0059, 0.3984){\line(1,0){ 0.0059}}
% line 0.6172 from 0.0100 to 0.0200
\put( 0.0059, 0.3828){\line(1,0){ 0.0059}}
% 1010 at 0.0100, 0.6562
\put( 0.1207, 0.3438){\makebox(0,0)[l]{\tt1010}}
% line 0.6875 from 0.0100 to 0.2800
\put( 0.0059, 0.3125){\line(1,0){ 0.1597}}
% 1011 at 0.0100, 0.7188
\put( 0.1207, 0.2812){\makebox(0,0)[l]{\tt1011}}
% 10100 at 0.0100, 0.6406
\put( 0.0387, 0.3594){\makebox(0,0)[l]{\tt10100}}
% line 0.6562 from 0.0100 to 0.1500
\put( 0.0059, 0.3438){\line(1,0){ 0.0828}}
% 10101 at 0.0100, 0.6719
\put( 0.0387, 0.3281){\makebox(0,0)[l]{\tt10101}}
% line 0.6406 from 0.0100 to 0.0400
\put( 0.0059, 0.3594){\line(1,0){ 0.0177}}
% line 0.6328 from 0.0100 to 0.0200
\put( 0.0059, 0.3672){\line(1,0){ 0.0059}}
% line 0.6484 from 0.0100 to 0.0200
\put( 0.0059, 0.3516){\line(1,0){ 0.0059}}
% line 0.6719 from 0.0100 to 0.0400
\put( 0.0059, 0.3281){\line(1,0){ 0.0177}}
% line 0.6641 from 0.0100 to 0.0200
\put( 0.0059, 0.3359){\line(1,0){ 0.0059}}
% line 0.6797 from 0.0100 to 0.0200
\put( 0.0059, 0.3203){\line(1,0){ 0.0059}}
% 10110 at 0.0100, 0.7031
\put( 0.0387, 0.2969){\makebox(0,0)[l]{\tt10110}}
% line 0.7188 from 0.0100 to 0.1500
\put( 0.0059, 0.2812){\line(1,0){ 0.0828}}
% 10111 at 0.0100, 0.7344
\put( 0.0387, 0.2656){\makebox(0,0)[l]{\tt10111}}
% line 0.7031 from 0.0100 to 0.0400
\put( 0.0059, 0.2969){\line(1,0){ 0.0177}}
% line 0.6953 from 0.0100 to 0.0200
\put( 0.0059, 0.3047){\line(1,0){ 0.0059}}
% line 0.7109 from 0.0100 to 0.0200
\put( 0.0059, 0.2891){\line(1,0){ 0.0059}}
% line 0.7344 from 0.0100 to 0.0400
\put( 0.0059, 0.2656){\line(1,0){ 0.0177}}
% line 0.7266 from 0.0100 to 0.0200
\put( 0.0059, 0.2734){\line(1,0){ 0.0059}}
% line 0.7422 from 0.0100 to 0.0200
\put( 0.0059, 0.2578){\line(1,0){ 0.0059}}
% 110 at 0.0100, 0.8125
\put( 0.1806, 0.1875){\makebox(0,0)[l]{\tt110}}
% line 0.8750 from 0.0100 to 0.3800
\put( 0.0059, 0.1250){\line(1,0){ 0.2188}}
% 111 at 0.0100, 0.9375
\put( 0.1806, 0.0625){\makebox(0,0)[l]{\tt111}}
% 1100 at 0.0100, 0.7812
\put( 0.1207, 0.2188){\makebox(0,0)[l]{\tt1100}}
% line 0.8125 from 0.0100 to 0.2800
\put( 0.0059, 0.1875){\line(1,0){ 0.1597}}
% 1101 at 0.0100, 0.8438
\put( 0.1207, 0.1562){\makebox(0,0)[l]{\tt1101}}
% 11000 at 0.0100, 0.7656
\put( 0.0387, 0.2344){\makebox(0,0)[l]{\tt11000}}
% line 0.7812 from 0.0100 to 0.1500
\put( 0.0059, 0.2188){\line(1,0){ 0.0828}}
% 11001 at 0.0100, 0.7969
\put( 0.0387, 0.2031){\makebox(0,0)[l]{\tt11001}}
% line 0.7656 from 0.0100 to 0.0400
\put( 0.0059, 0.2344){\line(1,0){ 0.0177}}
% line 0.7578 from 0.0100 to 0.0200
\put( 0.0059, 0.2422){\line(1,0){ 0.0059}}
% line 0.7734 from 0.0100 to 0.0200
\put( 0.0059, 0.2266){\line(1,0){ 0.0059}}
% line 0.7969 from 0.0100 to 0.0400
\put( 0.0059, 0.2031){\line(1,0){ 0.0177}}
% line 0.7891 from 0.0100 to 0.0200
\put( 0.0059, 0.2109){\line(1,0){ 0.0059}}
% line 0.8047 from 0.0100 to 0.0200
\put( 0.0059, 0.1953){\line(1,0){ 0.0059}}
% 11010 at 0.0100, 0.8281
\put( 0.0387, 0.1719){\makebox(0,0)[l]{\tt11010}}
% line 0.8438 from 0.0100 to 0.1500
\put( 0.0059, 0.1562){\line(1,0){ 0.0828}}
% 11011 at 0.0100, 0.8594
\put( 0.0387, 0.1406){\makebox(0,0)[l]{\tt11011}}
% line 0.8281 from 0.0100 to 0.0400
\put( 0.0059, 0.1719){\line(1,0){ 0.0177}}
% line 0.8203 from 0.0100 to 0.0200
\put( 0.0059, 0.1797){\line(1,0){ 0.0059}}
% line 0.8359 from 0.0100 to 0.0200
\put( 0.0059, 0.1641){\line(1,0){ 0.0059}}
% line 0.8594 from 0.0100 to 0.0400
\put( 0.0059, 0.1406){\line(1,0){ 0.0177}}
% line 0.8516 from 0.0100 to 0.0200
\put( 0.0059, 0.1484){\line(1,0){ 0.0059}}
% line 0.8672 from 0.0100 to 0.0200
\put( 0.0059, 0.1328){\line(1,0){ 0.0059}}
% 1110 at 0.0100, 0.9062
\put( 0.1207, 0.0938){\makebox(0,0)[l]{\tt1110}}
% line 0.9375 from 0.0100 to 0.2800
\put( 0.0059, 0.0625){\line(1,0){ 0.1597}}
% 1111 at 0.0100, 0.9688
\put( 0.1207, 0.0312){\makebox(0,0)[l]{\tt1111}}
% 11100 at 0.0100, 0.8906
\put( 0.0387, 0.1094){\makebox(0,0)[l]{\tt11100}}
% line 0.9062 from 0.0100 to 0.1500
\put( 0.0059, 0.0938){\line(1,0){ 0.0828}}
% 11101 at 0.0100, 0.9219
\put( 0.0387, 0.0781){\makebox(0,0)[l]{\tt11101}}
% line 0.8906 from 0.0100 to 0.0400
\put( 0.0059, 0.1094){\line(1,0){ 0.0177}}
% line 0.8828 from 0.0100 to 0.0200
\put( 0.0059, 0.1172){\line(1,0){ 0.0059}}
% line 0.8984 from 0.0100 to 0.0200
\put( 0.0059, 0.1016){\line(1,0){ 0.0059}}
% line 0.9219 from 0.0100 to 0.0400
\put( 0.0059, 0.0781){\line(1,0){ 0.0177}}
% line 0.9141 from 0.0100 to 0.0200
\put( 0.0059, 0.0859){\line(1,0){ 0.0059}}
% line 0.9297 from 0.0100 to 0.0200
\put( 0.0059, 0.0703){\line(1,0){ 0.0059}}
% 11110 at 0.0100, 0.9531
\put( 0.0387, 0.0469){\makebox(0,0)[l]{\tt11110}}
% line 0.9688 from 0.0100 to 0.1500
\put( 0.0059, 0.0312){\line(1,0){ 0.0828}}
% 11111 at 0.0100, 0.9844
\put( 0.0387, 0.0156){\makebox(0,0)[l]{\tt11111}}
% line 0.9531 from 0.0100 to 0.0400
\put( 0.0059, 0.0469){\line(1,0){ 0.0177}}
% line 0.9453 from 0.0100 to 0.0200
\put( 0.0059, 0.0547){\line(1,0){ 0.0059}}
% line 0.9609 from 0.0100 to 0.0200
\put( 0.0059, 0.0391){\line(1,0){ 0.0059}}
% line 0.9844 from 0.0100 to 0.0400
\put( 0.0059, 0.0156){\line(1,0){ 0.0177}}
% line 0.9766 from 0.0100 to 0.0200
\put( 0.0059, 0.0234){\line(1,0){ 0.0059}}
% line 0.9922 from 0.0100 to 0.0200
\put( 0.0059, 0.0078){\line(1,0){ 0.0059}}
\end{picture}
\hspace{-0.04in}% was -.25
\raisebox{1.1895in}{% was 1.425
\setlength{\unitlength}{33.39in}
%\setlength{\unitlength}{40in}
\begin{picture}(0.085,0.04)(-0.0425,0.37)
\thinlines
%
% wings added by hand
\put( -0.0408 , 0.4082){\line(-1,-3){ 0.005}}
\put( -0.0408 , 0.3730){\line(-1,3){ 0.005}}
%
% arrow identifying the final interval added by hand
% the center of the interval is 0010 below this point
% 10011110 (0.3809)
% 0.0017 is the length of the stubby lines
%
% want vector's tip to end at height 0.37995 and x=0.0010
% 4*34 = 136 -> 36635
% this was perfectly positioned
%\put( 0.0040, 0.36635){\makebox(0,0)[tl]{\tt100111101}}
%\put( 0.0044, 0.36635){\vector(-1,4){0.0034}}
% but I shifted it to this for arty reasons
\put( 0.0048, 0.36635){\makebox(0,0)[tl]{\tt100111101}}
\put( 0.0052, 0.36635){\vector(-1,4){0.0034}}
%
% line 0.5966 from -0.4800 to 0.0000
\put( -0.0408, 0.4034){\line(1,0){ 0.0408}}
% bbba at -0.2800, 0.6096
\put( -0.0252, 0.3904){\makebox(0,0)[r]{\tt{bbba}}}
% line 0.6227 from -0.4200 to 0.0000
\put( -0.0357, 0.3773){\line(1,0){ 0.0357}}
% bbbaa at -0.1000, 0.6003
\put( -0.0099, 0.3997){\makebox(0,0)[r]{\tt{bbbaa}}}
% line 0.6040 from -0.2800 to 0.0000
\put( -0.0238, 0.3960){\line(1,0){ 0.0238}}
% bbbab at -0.1000, 0.6114
\put( -0.0099, 0.3886){\makebox(0,0)[r]{\tt{bbbab}}}
% line 0.6188 from -0.2800 to 0.0000
\put( -0.0238, 0.3812){\line(1,0){ 0.0238}}
% bbba\eof at -0.1000, 0.6207
\put( -0.0099, 0.3793){\makebox(0,0)[r]{\tt{bbba\teof}}}
% line 0.6250 from 0.0100 to 0.4900
\put( 0.0008, 0.3750){\line(1,0){ 0.0408}}
% line 0.5938 from 0.0100 to 0.4200
\put( 0.0008, 0.4062){\line(1,0){ 0.0348}}
% 10011 at 0.0100, 0.6094
\put( 0.0299, 0.3906){\makebox(0,0)[l]{\tt10011}} % moved left a bit, was.0329
% 10010111 at 0.0100, 0.5918
\put( 0.0040, 0.4082){\makebox(0,0)[l]{\tt10010111}}
% line 0.5918 from 0.0100 to 0.0300
\put( 0.0008, 0.4082){\line(1,0){ 0.0017}}
% line 0.6094 from 0.0100 to 0.3700 % shortened, was .0306
\put( 0.0008, 0.3906){\line(1,0){ 0.0276}}
% line 0.6016 from 0.0100 to 0.3000
\put( 0.0008, 0.3984){\line(1,0){ 0.0246}}
% 10011000 at 0.0100, 0.5957
\put( 0.0040, 0.4043){\makebox(0,0)[l]{\tt10011000}}
% line 0.5977 from 0.0100 to 0.2100
\put( 0.0008, 0.4023){\line(1,0){ 0.0170}}
% 10011001 at 0.0100, 0.5996
\put( 0.0040, 0.4004){\makebox(0,0)[l]{\tt10011001}}
% line 0.5957 from 0.0100 to 0.0300
\put( 0.0008, 0.4043){\line(1,0){ 0.0017}}
% line 0.5996 from 0.0100 to 0.0300
\put( 0.0008, 0.4004){\line(1,0){ 0.0017}}
% 10011010 at 0.0100, 0.6035
\put( 0.0040, 0.3965){\makebox(0,0)[l]{\tt10011010}}
% line 0.6055 from 0.0100 to 0.2100
\put( 0.0008, 0.3945){\line(1,0){ 0.0170}}
% 10011011 at 0.0100, 0.6074
\put( 0.0040, 0.3926){\makebox(0,0)[l]{\tt10011011}}
% line 0.6035 from 0.0100 to 0.0300
\put( 0.0008, 0.3965){\line(1,0){ 0.0017}}
% line 0.6074 from 0.0100 to 0.0300
\put( 0.0008, 0.3926){\line(1,0){ 0.0017}}
% line 0.6172 from 0.0100 to 0.3000
\put( 0.0008, 0.3828){\line(1,0){ 0.0246}}
% 10011100 at 0.0100, 0.6113
\put( 0.0040, 0.3887){\makebox(0,0)[l]{\tt10011100}}
% line 0.6133 from 0.0100 to 0.2100
\put( 0.0008, 0.3867){\line(1,0){ 0.0170}}
% 10011101 at 0.0100, 0.6152
\put( 0.0040, 0.3848){\makebox(0,0)[l]{\tt10011101}}
% line 0.6113 from 0.0100 to 0.0300
\put( 0.0008, 0.3887){\line(1,0){ 0.0017}}
% line 0.6152 from 0.0100 to 0.0300
\put( 0.0008, 0.3848){\line(1,0){ 0.0017}}
% 10011110 at 0.0100, 0.6191
\put( 0.0040, 0.3809){\makebox(0,0)[l]{\tt10011110}}
% line 0.6211 from 0.0100 to 0.2100
\put( 0.0008, 0.3789){\line(1,0){ 0.0170}}
% 10011111 at 0.0100, 0.6230
\put( 0.0040, 0.3770){\makebox(0,0)[l]{\tt10011111}}
% line 0.6191 from 0.0100 to 0.0300
\put( 0.0008, 0.3809){\line(1,0){ 0.0017}}
% line 0.6230 from 0.0100 to 0.0300
\put( 0.0008, 0.3770){\line(1,0){ 0.0017}}
% 10100000 at 0.0100, 0.6270
\put( 0.0040, 0.3730){\makebox(0,0)[l]{\tt10100000}}
% line 0.6289 from 0.0100 to 0.2100
\put( 0.0008, 0.3711){\line(1,0){ 0.0170}}
% line 0.6270 from 0.0100 to 0.0300
\put( 0.0008, 0.3730){\line(1,0){ 0.0017}}
\end{picture}
}
}
\end{center}
}{%
\caption[a]{Illustration of the arithmetic coding
process as the sequence {$\tt bbba\eof$} is
transmitted.}
\label{fig.ac}
}%
\end{figure}
When the first symbol `$\tb$' is observed, the encoder
knows that the encoded string will start `{\tt{01}}',
`{\tt{10}}', or `{\tt{11}}',
but does not know which. The encoder writes nothing
for the time being, and
examines the next symbol, which is `$\tb$'.
The interval `$\tt bb$' lies wholly within interval `{\tt{1}}', so
the encoder can write the first bit: `{\tt{1}}'.
The third symbol `$\tt b$' narrows down the interval
a little, but not quite enough for it to lie
wholly within interval `{\tt{10}}'. Only when the next `$\tt a$'
is read from the source can we transmit some more
bits. Interval `$\tt bbba$' lies wholly within the interval `{\tt{1001}}',
so the encoder adds `{\tt{001}}'
to the `{\tt{1}}' it has written. Finally when the `$\eof$'
arrives, we need a procedure for terminating the encoding.
Magnifying the interval `$\tt bbba\eof$' (\figref{fig.ac}, right)
we note that the marked interval `{\tt{100111101}}'
is wholly contained by $\tt bbba\eof$, so the encoding can be completed by
appending `{\tt{11101}}'.
\exercissxA{2}{ex.ac.terminate}{
Show that the overhead required to terminate a message
is never more than 2 bits, relative to the ideal message length given the
probabilistic model $\H$,
$h(\bx \given \H) = \log [ 1/ P(\bx \given \H)]$.
}
% \begin{center}
% % created by ac.p sub=1 unit=40 only_show_data=1 > ac/ac_sub_data.tex
% \input{figs/ac/ac_sub_data.tex}
% \end{center}
This is an important result. Arithmetic coding is
very nearly optimal. The message length is always
within two bits of the \ind{Shannon information content}\index{information content} of the entire
source string,
so the expected message length is within two bits of the
entropy of the entire message.
\subsubsection{Decoding\subsubpunc}
The decoder receives the string `{\tt{100111101}}'
and passes along it one
symbol at a time. First, the probabilities $P(\ta), P(\tb), P(\eof)$ are computed
using the identical program that the encoder used and the intervals
`$\ta$', `$\tb$' and `$\eof$' are deduced. Once the first two
bits `{\tt{10}}' have been examined, it is certain that the original string
must have been started with a `$\tb$', since the interval `{\tt{10}}' lies wholly within
interval `$\tb$'. The decoder can then use the model to compute $P(\ta \given \tb),
P(\tb \given \tb), P(\eof \given \tb)$ and deduce the boundaries of the intervals
`$\tb\ta$', `$\tb\tb$' and `$\tb\eof$'.
Continuing, we decode the second $\tb$ once we reach `{\tt{1001}}',
the third $\tb$ once we reach `{\tt{100111}}', and so forth, with the
unambiguous identification of `$\tb\tb\tb\ta\eof$' once the whole binary
string has been
read. With the convention that `$\eof$' denotes the end of the message, the decoder
knows to stop decoding.
\subsubsection{Transmission of multiple files\subsubpunc}
How might one use arithmetic coding to communicate
several distinct files over the binary channel?
Once the $\eof$ character has been
transmitted, we imagine that the decoder is
reset into its initial state. There is no
transfer of the learnt statistics of the first file
to the second file.
% We start a fresh arithmetic code.
If, however, we did believe that there is a relationship
among the files that we are going to compress, we could
define our alphabet differently, introducing
a second end-of-file character that
marks the end of the file but instructs
the encoder and decoder to continue using the
same probabilistic model.
% If we went this route,
% we would only be able to uncompress the second file
% after first uncompressing the first file.
\subsection{The big picture}
Notice that to communicate a string of $N$ letters
% coming from an alphabet of size $|\A| = I$
both the encoder and the decoder needed to compute only $N|\A|$
conditional probabilities -- the probabilities of each possible letter
in each context actually encountered -- just
as in the guessing game.\index{guessing game} This cost can be contrasted with the alternative
of using a Huffman code\index{Huffman code!disadvantages}
with a large block size (in order to
reduce the possible one-bit-per-symbol overhead discussed in
% the previous chapter
section \ref{sec.huffman.probs}), where {\em all\/} block sequences that could
occur
% be encoded
% in a block
must be considered and their probabilities evaluated.
Notice how flexible arithmetic coding is:
it can be used with any source alphabet
and any encoded alphabet. The size of the source alphabet and the encoded
alphabet can
change with
time. Arithmetic coding can be used with any probability distribution,
which can change utterly from context to context.
Furthermore, if we would like the symbols of the encoding alphabet (say, {\tt 0} and {\tt 1})
to be used with {\em unequal\/} frequency, that can easily be arranged by subdividing
the right-hand interval in proportion to the required frequencies.
\subsection{How the probabilistic model might make its predictions}
The technique of arithmetic coding does not force one to
produce the predictive probability in any particular way, but
the predictive distributions might naturally be produced by a
Bayesian model.
\Figref{fig.ac} was generated using a
simple model that always assigns a probability of 0.15 to $\eof$,
and assigns the remaining 0.85 to $\ta$ and $\tb$, divided
in proportion to probabilities given by Laplace's
rule,
\beq
P_{\rm L}(\ta \given x_1,\ldots,x_{n-1})=\frac{F_{\ta}+1}{F_{\ta}+F_{\tb}+2} ,
\label{eq.laplaceagain}
\eeq
where
$F_{\ta}(x_1,\ldots,x_{n-1})$ is the number of times that $\ta$ has
occurred so far, and $F_{\tb}$ is the count of $\tb$s.
These predictions corresponds to a simple
Bayesian model that expects and adapts to
% is able to learn
a non-equal frequency
of use of the source symbols $\ta$ and $\tb$ within a file.
% The end result will be an encoder that can adapt to a nonuniform source.
\Figref{fig.ac2} displays the intervals corresponding to
a number of strings of length up to five. Note that if the string so far
has contained a large number of $\tb$s then the probability of
$\tb$ relative to $\ta$ is increased, and conversely if many $\ta$s
occur then $\ta$s are made more probable. Larger intervals, remember,
require fewer bits to encode.
%
\begin{figure}[tbp]
\figuremargin{%
\begin{center}
% created by ac.p only_show_data=1 > ac/ac_data.tex
\mbox{
\setlength{\unitlength}{5.75in}
\begin{picture}(0.59130434782608698452,1)(-0.29565217391304349226,0)
\thinlines
% line 0.0000 from -0.5000 to 0.0000
\put( -0.2957, 1.0000){\line(1,0){ 0.2957}}
% a at -0.4500, 0.2125
\put( -0.2811, 0.7875){\makebox(0,0)[r]{\tt{a}}}
% line 0.4250 from -0.5000 to 0.0000
\put( -0.2957, 0.5750){\line(1,0){ 0.2957}}
% b at -0.4500, 0.6375
\put( -0.2811, 0.3625){\makebox(0,0)[r]{\tt{b}}}
% line 0.8500 from -0.5000 to 0.0000
\put( -0.2957, 0.1500){\line(1,0){ 0.2957}}
% \eof at -0.4500, 0.9250
\put( -0.2811, 0.0750){\makebox(0,0)[r]{\tt{\teof}}}
% line 1.0000 from -0.5000 to 0.0000
\put( -0.2957, 0.0000){\line(1,0){ 0.2957}}
% aa at -0.3500, 0.1204
\put( -0.2220, 0.8796){\makebox(0,0)[r]{\tt{aa}}}
% line 0.2408 from -0.4500 to 0.0000
\put( -0.2661, 0.7592){\line(1,0){ 0.2661}}
% ab at -0.3500, 0.3010
\put( -0.2220, 0.6990){\makebox(0,0)[r]{\tt{ab}}}
% line 0.3612 from -0.4500 to 0.0000
\put( -0.2661, 0.6388){\line(1,0){ 0.2661}}
% a\eof at -0.3500, 0.3931
\put( -0.2220, 0.6069){\makebox(0,0)[r]{\tt{a\teof}}}
% aaa at -0.2300, 0.0768
\put( -0.1510, 0.9232){\makebox(0,0)[r]{\tt{aaa}}}
% line 0.1535 from -0.3500 to 0.0000
\put( -0.2070, 0.8465){\line(1,0){ 0.2070}}
% aab at -0.2300, 0.1791
\put( -0.1510, 0.8209){\makebox(0,0)[r]{\tt{aab}}}
% line 0.2047 from -0.3500 to 0.0000
\put( -0.2070, 0.7953){\line(1,0){ 0.2070}}
% aa\eof at -0.2300, 0.2228
\put( -0.1510, 0.7772){\makebox(0,0)[r]{\tt{aa\teof}}}
% aaaa at -0.1000, 0.0522
\put( -0.0741, 0.9478){\makebox(0,0)[r]{\tt{aaaa}}}
% line 0.1044 from -0.2300 to 0.0000
\put( -0.1360, 0.8956){\line(1,0){ 0.1360}}
% aaab at -0.1000, 0.1175
\put( -0.0741, 0.8825){\makebox(0,0)[r]{\tt{aaab}}}
% line 0.1305 from -0.2300 to 0.0000
\put( -0.1360, 0.8695){\line(1,0){ 0.1360}}
% line 0.0740 from -0.1000 to 0.0000
\put( -0.0591, 0.9260){\line(1,0){ 0.0591}}
% line 0.0887 from -0.1000 to 0.0000
\put( -0.0591, 0.9113){\line(1,0){ 0.0591}}
% line 0.1192 from -0.1000 to 0.0000
\put( -0.0591, 0.8808){\line(1,0){ 0.0591}}
% line 0.1266 from -0.1000 to 0.0000
\put( -0.0591, 0.8734){\line(1,0){ 0.0591}}
% aaba at -0.1000, 0.1666
\put( -0.0741, 0.8334){\makebox(0,0)[r]{\tt{aaba}}}
% line 0.1796 from -0.2300 to 0.0000
\put( -0.1360, 0.8204){\line(1,0){ 0.1360}}
% aabb at -0.1000, 0.1883
\put( -0.0741, 0.8117){\makebox(0,0)[r]{\tt{aabb}}}
% line 0.1970 from -0.2300 to 0.0000
\put( -0.1360, 0.8030){\line(1,0){ 0.1360}}
% line 0.1683 from -0.1000 to 0.0000
\put( -0.0591, 0.8317){\line(1,0){ 0.0591}}
% line 0.1757 from -0.1000 to 0.0000
\put( -0.0591, 0.8243){\line(1,0){ 0.0591}}
% line 0.1870 from -0.1000 to 0.0000
\put( -0.0591, 0.8130){\line(1,0){ 0.0591}}
% line 0.1944 from -0.1000 to 0.0000
\put( -0.0591, 0.8056){\line(1,0){ 0.0591}}
% aba at -0.2300, 0.2664
\put( -0.1510, 0.7336){\makebox(0,0)[r]{\tt{aba}}}
% line 0.2920 from -0.3500 to 0.0000
\put( -0.2070, 0.7080){\line(1,0){ 0.2070}}
% abb at -0.2300, 0.3176
\put( -0.1510, 0.6824){\makebox(0,0)[r]{\tt{abb}}}
% line 0.3432 from -0.3500 to 0.0000
\put( -0.2070, 0.6568){\line(1,0){ 0.2070}}
% ab\eof at -0.2300, 0.3522
\put( -0.1510, 0.6478){\makebox(0,0)[r]{\tt{ab\teof}}}
% abaa at -0.1000, 0.2539
\put( -0.0741, 0.7461){\makebox(0,0)[r]{\tt{abaa}}}
% line 0.2669 from -0.2300 to 0.0000
\put( -0.1360, 0.7331){\line(1,0){ 0.1360}}
% abab at -0.1000, 0.2756
\put( -0.0741, 0.7244){\makebox(0,0)[r]{\tt{abab}}}
% line 0.2843 from -0.2300 to 0.0000
\put( -0.1360, 0.7157){\line(1,0){ 0.1360}}
% line 0.2556 from -0.1000 to 0.0000
\put( -0.0591, 0.7444){\line(1,0){ 0.0591}}
% line 0.2630 from -0.1000 to 0.0000
\put( -0.0591, 0.7370){\line(1,0){ 0.0591}}
% line 0.2743 from -0.1000 to 0.0000
\put( -0.0591, 0.7257){\line(1,0){ 0.0591}}
% line 0.2817 from -0.1000 to 0.0000
\put( -0.0591, 0.7183){\line(1,0){ 0.0591}}
% abba at -0.1000, 0.3007
\put( -0.0741, 0.6993){\makebox(0,0)[r]{\tt{abba}}}
% line 0.3094 from -0.2300 to 0.0000
\put( -0.1360, 0.6906){\line(1,0){ 0.1360}}
% abbb at -0.1000, 0.3225
\put( -0.0741, 0.6775){\makebox(0,0)[r]{\tt{abbb}}}
% line 0.3355 from -0.2300 to 0.0000
\put( -0.1360, 0.6645){\line(1,0){ 0.1360}}
% line 0.2994 from -0.1000 to 0.0000
\put( -0.0591, 0.7006){\line(1,0){ 0.0591}}
% line 0.3068 from -0.1000 to 0.0000
\put( -0.0591, 0.6932){\line(1,0){ 0.0591}}
% line 0.3168 from -0.1000 to 0.0000
\put( -0.0591, 0.6832){\line(1,0){ 0.0591}}
% line 0.3316 from -0.1000 to 0.0000
\put( -0.0591, 0.6684){\line(1,0){ 0.0591}}
% ba at -0.3500, 0.4852
\put( -0.2220, 0.5148){\makebox(0,0)[r]{\tt{ba}}}
% line 0.5454 from -0.4500 to 0.0000
\put( -0.2661, 0.4546){\line(1,0){ 0.2661}}
% bb at -0.3500, 0.6658
\put( -0.2220, 0.3342){\makebox(0,0)[r]{\tt{bb}}}
% line 0.7862 from -0.4500 to 0.0000
\put( -0.2661, 0.2138){\line(1,0){ 0.2661}}
% b\eof at -0.3500, 0.8181
\put( -0.2220, 0.1819){\makebox(0,0)[r]{\tt{b\teof}}}
% baa at -0.2300, 0.4506
\put( -0.1510, 0.5494){\makebox(0,0)[r]{\tt{baa}}}
% line 0.4762 from -0.3500 to 0.0000
\put( -0.2070, 0.5238){\line(1,0){ 0.2070}}
% bab at -0.2300, 0.5018
\put( -0.1510, 0.4982){\makebox(0,0)[r]{\tt{bab}}}
% line 0.5274 from -0.3500 to 0.0000
\put( -0.2070, 0.4726){\line(1,0){ 0.2070}}
% ba\eof at -0.2300, 0.5364
\put( -0.1510, 0.4636){\makebox(0,0)[r]{\tt{ba\teof}}}
% baaa at -0.1000, 0.4381
\put( -0.0741, 0.5619){\makebox(0,0)[r]{\tt{baaa}}}
% line 0.4511 from -0.2300 to 0.0000
\put( -0.1360, 0.5489){\line(1,0){ 0.1360}}
% baab at -0.1000, 0.4598
\put( -0.0741, 0.5402){\makebox(0,0)[r]{\tt{baab}}}
% line 0.4685 from -0.2300 to 0.0000
\put( -0.1360, 0.5315){\line(1,0){ 0.1360}}
% line 0.4398 from -0.1000 to 0.0000
\put( -0.0591, 0.5602){\line(1,0){ 0.0591}}
% line 0.4472 from -0.1000 to 0.0000
\put( -0.0591, 0.5528){\line(1,0){ 0.0591}}
% line 0.4585 from -0.1000 to 0.0000
\put( -0.0591, 0.5415){\line(1,0){ 0.0591}}
% line 0.4659 from -0.1000 to 0.0000
\put( -0.0591, 0.5341){\line(1,0){ 0.0591}}
% baba at -0.1000, 0.4849
\put( -0.0741, 0.5151){\makebox(0,0)[r]{\tt{baba}}}
% line 0.4936 from -0.2300 to 0.0000
\put( -0.1360, 0.5064){\line(1,0){ 0.1360}}
% babb at -0.1000, 0.5066
\put( -0.0741, 0.4934){\makebox(0,0)[r]{\tt{babb}}}
% line 0.5197 from -0.2300 to 0.0000
\put( -0.1360, 0.4803){\line(1,0){ 0.1360}}
% line 0.4836 from -0.1000 to 0.0000
\put( -0.0591, 0.5164){\line(1,0){ 0.0591}}
% line 0.4910 from -0.1000 to 0.0000
\put( -0.0591, 0.5090){\line(1,0){ 0.0591}}
% line 0.5010 from -0.1000 to 0.0000
\put( -0.0591, 0.4990){\line(1,0){ 0.0591}}
% line 0.5158 from -0.1000 to 0.0000
\put( -0.0591, 0.4842){\line(1,0){ 0.0591}}
% bba at -0.2300, 0.5710
\put( -0.1510, 0.4290){\makebox(0,0)[r]{\tt{bba}}}
% line 0.5966 from -0.3500 to 0.0000
\put( -0.2070, 0.4034){\line(1,0){ 0.2070}}
% bbb at -0.2300, 0.6734
\put( -0.1510, 0.3266){\makebox(0,0)[r]{\tt{bbb}}}
% line 0.7501 from -0.3500 to 0.0000
\put( -0.2070, 0.2499){\line(1,0){ 0.2070}}
% bb\eof at -0.2300, 0.7682
\put( -0.1510, 0.2318){\makebox(0,0)[r]{\tt{bb\teof}}}
% bbaa at -0.1000, 0.5541
\put( -0.0741, 0.4459){\makebox(0,0)[r]{\tt{bbaa}}}
% line 0.5628 from -0.2300 to 0.0000
\put( -0.1360, 0.4372){\line(1,0){ 0.1360}}
% bbab at -0.1000, 0.5759
\put( -0.0741, 0.4241){\makebox(0,0)[r]{\tt{bbab}}}
% line 0.5889 from -0.2300 to 0.0000
\put( -0.1360, 0.4111){\line(1,0){ 0.1360}}
% line 0.5528 from -0.1000 to 0.0000
\put( -0.0591, 0.4472){\line(1,0){ 0.0591}}
% line 0.5602 from -0.1000 to 0.0000
\put( -0.0591, 0.4398){\line(1,0){ 0.0591}}
% line 0.5702 from -0.1000 to 0.0000
\put( -0.0591, 0.4298){\line(1,0){ 0.0591}}
% line 0.5850 from -0.1000 to 0.0000
\put( -0.0591, 0.4150){\line(1,0){ 0.0591}}
% bbba at -0.1000, 0.6096
\put( -0.0741, 0.3904){\makebox(0,0)[r]{\tt{bbba}}}
% line 0.6227 from -0.2300 to 0.0000
\put( -0.1360, 0.3773){\line(1,0){ 0.1360}}
% bbbb at -0.1000, 0.6749
\put( -0.0741, 0.3251){\makebox(0,0)[r]{\tt{bbbb}}}
% line 0.7271 from -0.2300 to 0.0000
\put( -0.1360, 0.2729){\line(1,0){ 0.1360}}
% line 0.6040 from -0.1000 to 0.0000
\put( -0.0591, 0.3960){\line(1,0){ 0.0591}}
% line 0.6188 from -0.1000 to 0.0000
\put( -0.0591, 0.3812){\line(1,0){ 0.0591}}
% line 0.6375 from -0.1000 to 0.0000
\put( -0.0591, 0.3625){\line(1,0){ 0.0591}}
% line 0.7114 from -0.1000 to 0.0000
\put( -0.0591, 0.2886){\line(1,0){ 0.0591}}
% line 0.0000 from 0.0100 to 0.5000
\put( 0.0059, 1.0000){\line(1,0){ 0.2897}}
% 0 at 0.0100, 0.2500
\put( 0.2811, 0.7500){\makebox(0,0)[l]{\tt0}}
% line 0.5000 from 0.0100 to 0.5000
\put( 0.0059, 0.5000){\line(1,0){ 0.2897}}
% 1 at 0.0100, 0.7500
\put( 0.2811, 0.2500){\makebox(0,0)[l]{\tt1}}
% line 1.0000 from 0.0100 to 0.5000
\put( 0.0059, 0.0000){\line(1,0){ 0.2897}}
% 00 at 0.0100, 0.1250
\put( 0.2397, 0.8750){\makebox(0,0)[l]{\tt00}}
% line 0.2500 from 0.0100 to 0.4500
\put( 0.0059, 0.7500){\line(1,0){ 0.2602}}
% 01 at 0.0100, 0.3750
\put( 0.2397, 0.6250){\makebox(0,0)[l]{\tt01}}
% 000 at 0.0100, 0.0625
\put( 0.1806, 0.9375){\makebox(0,0)[l]{\tt000}}
% line 0.1250 from 0.0100 to 0.3800
\put( 0.0059, 0.8750){\line(1,0){ 0.2188}}
% 001 at 0.0100, 0.1875
\put( 0.1806, 0.8125){\makebox(0,0)[l]{\tt001}}
% 0000 at 0.0100, 0.0312
\put( 0.1207, 0.9688){\makebox(0,0)[l]{\tt0000}}
% line 0.0625 from 0.0100 to 0.2800
\put( 0.0059, 0.9375){\line(1,0){ 0.1597}}
% 0001 at 0.0100, 0.0938
\put( 0.1207, 0.9062){\makebox(0,0)[l]{\tt0001}}
% 00000 at 0.0100, 0.0156
\put( 0.0387, 0.9844){\makebox(0,0)[l]{\tt00000}}
% line 0.0312 from 0.0100 to 0.1500
\put( 0.0059, 0.9688){\line(1,0){ 0.0828}}
% 00001 at 0.0100, 0.0469
\put( 0.0387, 0.9531){\makebox(0,0)[l]{\tt00001}}
% line 0.0156 from 0.0100 to 0.0400
\put( 0.0059, 0.9844){\line(1,0){ 0.0177}}
% line 0.0078 from 0.0100 to 0.0200
\put( 0.0059, 0.9922){\line(1,0){ 0.0059}}
% line 0.0234 from 0.0100 to 0.0200
\put( 0.0059, 0.9766){\line(1,0){ 0.0059}}
% line 0.0469 from 0.0100 to 0.0400
\put( 0.0059, 0.9531){\line(1,0){ 0.0177}}
% line 0.0391 from 0.0100 to 0.0200
\put( 0.0059, 0.9609){\line(1,0){ 0.0059}}
% line 0.0547 from 0.0100 to 0.0200
\put( 0.0059, 0.9453){\line(1,0){ 0.0059}}
% 00010 at 0.0100, 0.0781
\put( 0.0387, 0.9219){\makebox(0,0)[l]{\tt00010}}
% line 0.0938 from 0.0100 to 0.1500
\put( 0.0059, 0.9062){\line(1,0){ 0.0828}}
% 00011 at 0.0100, 0.1094
\put( 0.0387, 0.8906){\makebox(0,0)[l]{\tt00011}}
% line 0.0781 from 0.0100 to 0.0400
\put( 0.0059, 0.9219){\line(1,0){ 0.0177}}
% line 0.0703 from 0.0100 to 0.0200
\put( 0.0059, 0.9297){\line(1,0){ 0.0059}}
% line 0.0859 from 0.0100 to 0.0200
\put( 0.0059, 0.9141){\line(1,0){ 0.0059}}
% line 0.1094 from 0.0100 to 0.0400
\put( 0.0059, 0.8906){\line(1,0){ 0.0177}}
% line 0.1016 from 0.0100 to 0.0200
\put( 0.0059, 0.8984){\line(1,0){ 0.0059}}
% line 0.1172 from 0.0100 to 0.0200
\put( 0.0059, 0.8828){\line(1,0){ 0.0059}}
% 0010 at 0.0100, 0.1562
\put( 0.1207, 0.8438){\makebox(0,0)[l]{\tt0010}}
% line 0.1875 from 0.0100 to 0.2800
\put( 0.0059, 0.8125){\line(1,0){ 0.1597}}
% 0011 at 0.0100, 0.2188
\put( 0.1207, 0.7812){\makebox(0,0)[l]{\tt0011}}
% 00100 at 0.0100, 0.1406
\put( 0.0387, 0.8594){\makebox(0,0)[l]{\tt00100}}
% line 0.1562 from 0.0100 to 0.1500
\put( 0.0059, 0.8438){\line(1,0){ 0.0828}}
% 00101 at 0.0100, 0.1719
\put( 0.0387, 0.8281){\makebox(0,0)[l]{\tt00101}}
% line 0.1406 from 0.0100 to 0.0400
\put( 0.0059, 0.8594){\line(1,0){ 0.0177}}
% line 0.1328 from 0.0100 to 0.0200
\put( 0.0059, 0.8672){\line(1,0){ 0.0059}}
% line 0.1484 from 0.0100 to 0.0200
\put( 0.0059, 0.8516){\line(1,0){ 0.0059}}
% line 0.1719 from 0.0100 to 0.0400
\put( 0.0059, 0.8281){\line(1,0){ 0.0177}}
% line 0.1641 from 0.0100 to 0.0200
\put( 0.0059, 0.8359){\line(1,0){ 0.0059}}
% line 0.1797 from 0.0100 to 0.0200
\put( 0.0059, 0.8203){\line(1,0){ 0.0059}}
% 00110 at 0.0100, 0.2031
\put( 0.0387, 0.7969){\makebox(0,0)[l]{\tt00110}}
% line 0.2188 from 0.0100 to 0.1500
\put( 0.0059, 0.7812){\line(1,0){ 0.0828}}
% 00111 at 0.0100, 0.2344
\put( 0.0387, 0.7656){\makebox(0,0)[l]{\tt00111}}
% line 0.2031 from 0.0100 to 0.0400
\put( 0.0059, 0.7969){\line(1,0){ 0.0177}}
% line 0.1953 from 0.0100 to 0.0200
\put( 0.0059, 0.8047){\line(1,0){ 0.0059}}
% line 0.2109 from 0.0100 to 0.0200
\put( 0.0059, 0.7891){\line(1,0){ 0.0059}}
% line 0.2344 from 0.0100 to 0.0400
\put( 0.0059, 0.7656){\line(1,0){ 0.0177}}
% line 0.2266 from 0.0100 to 0.0200
\put( 0.0059, 0.7734){\line(1,0){ 0.0059}}
% line 0.2422 from 0.0100 to 0.0200
\put( 0.0059, 0.7578){\line(1,0){ 0.0059}}
% 010 at 0.0100, 0.3125
\put( 0.1806, 0.6875){\makebox(0,0)[l]{\tt010}}
% line 0.3750 from 0.0100 to 0.3800
\put( 0.0059, 0.6250){\line(1,0){ 0.2188}}
% 011 at 0.0100, 0.4375
\put( 0.1806, 0.5625){\makebox(0,0)[l]{\tt011}}
% 0100 at 0.0100, 0.2812
\put( 0.1207, 0.7188){\makebox(0,0)[l]{\tt0100}}
% line 0.3125 from 0.0100 to 0.2800
\put( 0.0059, 0.6875){\line(1,0){ 0.1597}}
% 0101 at 0.0100, 0.3438
\put( 0.1207, 0.6562){\makebox(0,0)[l]{\tt0101}}
% 01000 at 0.0100, 0.2656
\put( 0.0387, 0.7344){\makebox(0,0)[l]{\tt01000}}
% line 0.2812 from 0.0100 to 0.1500
\put( 0.0059, 0.7188){\line(1,0){ 0.0828}}
% 01001 at 0.0100, 0.2969
\put( 0.0387, 0.7031){\makebox(0,0)[l]{\tt01001}}
% line 0.2656 from 0.0100 to 0.0400
\put( 0.0059, 0.7344){\line(1,0){ 0.0177}}
% line 0.2578 from 0.0100 to 0.0200
\put( 0.0059, 0.7422){\line(1,0){ 0.0059}}
% line 0.2734 from 0.0100 to 0.0200
\put( 0.0059, 0.7266){\line(1,0){ 0.0059}}
% line 0.2969 from 0.0100 to 0.0400
\put( 0.0059, 0.7031){\line(1,0){ 0.0177}}
% line 0.2891 from 0.0100 to 0.0200
\put( 0.0059, 0.7109){\line(1,0){ 0.0059}}
% line 0.3047 from 0.0100 to 0.0200
\put( 0.0059, 0.6953){\line(1,0){ 0.0059}}
% 01010 at 0.0100, 0.3281
\put( 0.0387, 0.6719){\makebox(0,0)[l]{\tt01010}}
% line 0.3438 from 0.0100 to 0.1500
\put( 0.0059, 0.6562){\line(1,0){ 0.0828}}
% 01011 at 0.0100, 0.3594
\put( 0.0387, 0.6406){\makebox(0,0)[l]{\tt01011}}
% line 0.3281 from 0.0100 to 0.0400
\put( 0.0059, 0.6719){\line(1,0){ 0.0177}}
% line 0.3203 from 0.0100 to 0.0200
\put( 0.0059, 0.6797){\line(1,0){ 0.0059}}
% line 0.3359 from 0.0100 to 0.0200
\put( 0.0059, 0.6641){\line(1,0){ 0.0059}}
% line 0.3594 from 0.0100 to 0.0400
\put( 0.0059, 0.6406){\line(1,0){ 0.0177}}
% line 0.3516 from 0.0100 to 0.0200
\put( 0.0059, 0.6484){\line(1,0){ 0.0059}}
% line 0.3672 from 0.0100 to 0.0200
\put( 0.0059, 0.6328){\line(1,0){ 0.0059}}
% 0110 at 0.0100, 0.4062
\put( 0.1207, 0.5938){\makebox(0,0)[l]{\tt0110}}
% line 0.4375 from 0.0100 to 0.2800
\put( 0.0059, 0.5625){\line(1,0){ 0.1597}}
% 0111 at 0.0100, 0.4688
\put( 0.1207, 0.5312){\makebox(0,0)[l]{\tt0111}}
% 01100 at 0.0100, 0.3906
\put( 0.0387, 0.6094){\makebox(0,0)[l]{\tt01100}}
% line 0.4062 from 0.0100 to 0.1500
\put( 0.0059, 0.5938){\line(1,0){ 0.0828}}
% 01101 at 0.0100, 0.4219
\put( 0.0387, 0.5781){\makebox(0,0)[l]{\tt01101}}
% line 0.3906 from 0.0100 to 0.0400
\put( 0.0059, 0.6094){\line(1,0){ 0.0177}}
% line 0.3828 from 0.0100 to 0.0200
\put( 0.0059, 0.6172){\line(1,0){ 0.0059}}
% line 0.3984 from 0.0100 to 0.0200
\put( 0.0059, 0.6016){\line(1,0){ 0.0059}}
% line 0.4219 from 0.0100 to 0.0400
\put( 0.0059, 0.5781){\line(1,0){ 0.0177}}
% line 0.4141 from 0.0100 to 0.0200
\put( 0.0059, 0.5859){\line(1,0){ 0.0059}}
% line 0.4297 from 0.0100 to 0.0200
\put( 0.0059, 0.5703){\line(1,0){ 0.0059}}
% 01110 at 0.0100, 0.4531
\put( 0.0387, 0.5469){\makebox(0,0)[l]{\tt01110}}
% line 0.4688 from 0.0100 to 0.1500
\put( 0.0059, 0.5312){\line(1,0){ 0.0828}}
% 01111 at 0.0100, 0.4844
\put( 0.0387, 0.5156){\makebox(0,0)[l]{\tt01111}}
% line 0.4531 from 0.0100 to 0.0400
\put( 0.0059, 0.5469){\line(1,0){ 0.0177}}
% line 0.4453 from 0.0100 to 0.0200
\put( 0.0059, 0.5547){\line(1,0){ 0.0059}}
% line 0.4609 from 0.0100 to 0.0200
\put( 0.0059, 0.5391){\line(1,0){ 0.0059}}
% line 0.4844 from 0.0100 to 0.0400
\put( 0.0059, 0.5156){\line(1,0){ 0.0177}}
% line 0.4766 from 0.0100 to 0.0200
\put( 0.0059, 0.5234){\line(1,0){ 0.0059}}
% line 0.4922 from 0.0100 to 0.0200
\put( 0.0059, 0.5078){\line(1,0){ 0.0059}}
% 10 at 0.0100, 0.6250
\put( 0.2397, 0.3750){\makebox(0,0)[l]{\tt10}}
% line 0.7500 from 0.0100 to 0.4500
\put( 0.0059, 0.2500){\line(1,0){ 0.2602}}
% 11 at 0.0100, 0.8750
\put( 0.2397, 0.1250){\makebox(0,0)[l]{\tt11}}
% 100 at 0.0100, 0.5625
\put( 0.1806, 0.4375){\makebox(0,0)[l]{\tt100}}
% line 0.6250 from 0.0100 to 0.3800
\put( 0.0059, 0.3750){\line(1,0){ 0.2188}}
% 101 at 0.0100, 0.6875
\put( 0.1806, 0.3125){\makebox(0,0)[l]{\tt101}}
% 1000 at 0.0100, 0.5312
\put( 0.1207, 0.4688){\makebox(0,0)[l]{\tt1000}}
% line 0.5625 from 0.0100 to 0.2800
\put( 0.0059, 0.4375){\line(1,0){ 0.1597}}
% 1001 at 0.0100, 0.5938
\put( 0.1207, 0.4062){\makebox(0,0)[l]{\tt1001}}
% 10000 at 0.0100, 0.5156
\put( 0.0387, 0.4844){\makebox(0,0)[l]{\tt10000}}
% line 0.5312 from 0.0100 to 0.1500
\put( 0.0059, 0.4688){\line(1,0){ 0.0828}}
% 10001 at 0.0100, 0.5469
\put( 0.0387, 0.4531){\makebox(0,0)[l]{\tt10001}}
% line 0.5156 from 0.0100 to 0.0400
\put( 0.0059, 0.4844){\line(1,0){ 0.0177}}
% line 0.5078 from 0.0100 to 0.0200
\put( 0.0059, 0.4922){\line(1,0){ 0.0059}}
% line 0.5234 from 0.0100 to 0.0200
\put( 0.0059, 0.4766){\line(1,0){ 0.0059}}
% line 0.5469 from 0.0100 to 0.0400
\put( 0.0059, 0.4531){\line(1,0){ 0.0177}}
% line 0.5391 from 0.0100 to 0.0200
\put( 0.0059, 0.4609){\line(1,0){ 0.0059}}
% line 0.5547 from 0.0100 to 0.0200
\put( 0.0059, 0.4453){\line(1,0){ 0.0059}}
% 10010 at 0.0100, 0.5781
\put( 0.0387, 0.4219){\makebox(0,0)[l]{\tt10010}}
% line 0.5938 from 0.0100 to 0.1500
\put( 0.0059, 0.4062){\line(1,0){ 0.0828}}
% 10011 at 0.0100, 0.6094
\put( 0.0387, 0.3906){\makebox(0,0)[l]{\tt10011}}
% line 0.5781 from 0.0100 to 0.0400
\put( 0.0059, 0.4219){\line(1,0){ 0.0177}}
% line 0.5703 from 0.0100 to 0.0200
\put( 0.0059, 0.4297){\line(1,0){ 0.0059}}
% line 0.5859 from 0.0100 to 0.0200
\put( 0.0059, 0.4141){\line(1,0){ 0.0059}}
% line 0.6094 from 0.0100 to 0.0400
\put( 0.0059, 0.3906){\line(1,0){ 0.0177}}
% line 0.6016 from 0.0100 to 0.0200
\put( 0.0059, 0.3984){\line(1,0){ 0.0059}}
% line 0.6172 from 0.0100 to 0.0200
\put( 0.0059, 0.3828){\line(1,0){ 0.0059}}
% 1010 at 0.0100, 0.6562
\put( 0.1207, 0.3438){\makebox(0,0)[l]{\tt1010}}
% line 0.6875 from 0.0100 to 0.2800
\put( 0.0059, 0.3125){\line(1,0){ 0.1597}}
% 1011 at 0.0100, 0.7188
\put( 0.1207, 0.2812){\makebox(0,0)[l]{\tt1011}}
% 10100 at 0.0100, 0.6406
\put( 0.0387, 0.3594){\makebox(0,0)[l]{\tt10100}}
% line 0.6562 from 0.0100 to 0.1500
\put( 0.0059, 0.3438){\line(1,0){ 0.0828}}
% 10101 at 0.0100, 0.6719
\put( 0.0387, 0.3281){\makebox(0,0)[l]{\tt10101}}
% line 0.6406 from 0.0100 to 0.0400
\put( 0.0059, 0.3594){\line(1,0){ 0.0177}}
% line 0.6328 from 0.0100 to 0.0200
\put( 0.0059, 0.3672){\line(1,0){ 0.0059}}
% line 0.6484 from 0.0100 to 0.0200
\put( 0.0059, 0.3516){\line(1,0){ 0.0059}}
% line 0.6719 from 0.0100 to 0.0400
\put( 0.0059, 0.3281){\line(1,0){ 0.0177}}
% line 0.6641 from 0.0100 to 0.0200
\put( 0.0059, 0.3359){\line(1,0){ 0.0059}}
% line 0.6797 from 0.0100 to 0.0200
\put( 0.0059, 0.3203){\line(1,0){ 0.0059}}
% 10110 at 0.0100, 0.7031
\put( 0.0387, 0.2969){\makebox(0,0)[l]{\tt10110}}
% line 0.7188 from 0.0100 to 0.1500
\put( 0.0059, 0.2812){\line(1,0){ 0.0828}}
% 10111 at 0.0100, 0.7344
\put( 0.0387, 0.2656){\makebox(0,0)[l]{\tt10111}}
% line 0.7031 from 0.0100 to 0.0400
\put( 0.0059, 0.2969){\line(1,0){ 0.0177}}
% line 0.6953 from 0.0100 to 0.0200
\put( 0.0059, 0.3047){\line(1,0){ 0.0059}}
% line 0.7109 from 0.0100 to 0.0200
\put( 0.0059, 0.2891){\line(1,0){ 0.0059}}
% line 0.7344 from 0.0100 to 0.0400
\put( 0.0059, 0.2656){\line(1,0){ 0.0177}}
% line 0.7266 from 0.0100 to 0.0200
\put( 0.0059, 0.2734){\line(1,0){ 0.0059}}
% line 0.7422 from 0.0100 to 0.0200
\put( 0.0059, 0.2578){\line(1,0){ 0.0059}}
% 110 at 0.0100, 0.8125
\put( 0.1806, 0.1875){\makebox(0,0)[l]{\tt110}}
% line 0.8750 from 0.0100 to 0.3800
\put( 0.0059, 0.1250){\line(1,0){ 0.2188}}
% 111 at 0.0100, 0.9375
\put( 0.1806, 0.0625){\makebox(0,0)[l]{\tt111}}
% 1100 at 0.0100, 0.7812
\put( 0.1207, 0.2188){\makebox(0,0)[l]{\tt1100}}
% line 0.8125 from 0.0100 to 0.2800
\put( 0.0059, 0.1875){\line(1,0){ 0.1597}}
% 1101 at 0.0100, 0.8438
\put( 0.1207, 0.1562){\makebox(0,0)[l]{\tt1101}}
% 11000 at 0.0100, 0.7656
\put( 0.0387, 0.2344){\makebox(0,0)[l]{\tt11000}}
% line 0.7812 from 0.0100 to 0.1500
\put( 0.0059, 0.2188){\line(1,0){ 0.0828}}
% 11001 at 0.0100, 0.7969
\put( 0.0387, 0.2031){\makebox(0,0)[l]{\tt11001}}
% line 0.7656 from 0.0100 to 0.0400
\put( 0.0059, 0.2344){\line(1,0){ 0.0177}}
% line 0.7578 from 0.0100 to 0.0200
\put( 0.0059, 0.2422){\line(1,0){ 0.0059}}
% line 0.7734 from 0.0100 to 0.0200
\put( 0.0059, 0.2266){\line(1,0){ 0.0059}}
% line 0.7969 from 0.0100 to 0.0400
\put( 0.0059, 0.2031){\line(1,0){ 0.0177}}
% line 0.7891 from 0.0100 to 0.0200
\put( 0.0059, 0.2109){\line(1,0){ 0.0059}}
% line 0.8047 from 0.0100 to 0.0200
\put( 0.0059, 0.1953){\line(1,0){ 0.0059}}
% 11010 at 0.0100, 0.8281
\put( 0.0387, 0.1719){\makebox(0,0)[l]{\tt11010}}
% line 0.8438 from 0.0100 to 0.1500
\put( 0.0059, 0.1562){\line(1,0){ 0.0828}}
% 11011 at 0.0100, 0.8594
\put( 0.0387, 0.1406){\makebox(0,0)[l]{\tt11011}}
% line 0.8281 from 0.0100 to 0.0400
\put( 0.0059, 0.1719){\line(1,0){ 0.0177}}
% line 0.8203 from 0.0100 to 0.0200
\put( 0.0059, 0.1797){\line(1,0){ 0.0059}}
% line 0.8359 from 0.0100 to 0.0200
\put( 0.0059, 0.1641){\line(1,0){ 0.0059}}
% line 0.8594 from 0.0100 to 0.0400
\put( 0.0059, 0.1406){\line(1,0){ 0.0177}}
% line 0.8516 from 0.0100 to 0.0200
\put( 0.0059, 0.1484){\line(1,0){ 0.0059}}
% line 0.8672 from 0.0100 to 0.0200
\put( 0.0059, 0.1328){\line(1,0){ 0.0059}}
% 1110 at 0.0100, 0.9062
\put( 0.1207, 0.0938){\makebox(0,0)[l]{\tt1110}}
% line 0.9375 from 0.0100 to 0.2800
\put( 0.0059, 0.0625){\line(1,0){ 0.1597}}
% 1111 at 0.0100, 0.9688
\put( 0.1207, 0.0312){\makebox(0,0)[l]{\tt1111}}
% 11100 at 0.0100, 0.8906
\put( 0.0387, 0.1094){\makebox(0,0)[l]{\tt11100}}
% line 0.9062 from 0.0100 to 0.1500
\put( 0.0059, 0.0938){\line(1,0){ 0.0828}}
% 11101 at 0.0100, 0.9219
\put( 0.0387, 0.0781){\makebox(0,0)[l]{\tt11101}}
% line 0.8906 from 0.0100 to 0.0400
\put( 0.0059, 0.1094){\line(1,0){ 0.0177}}
% line 0.8828 from 0.0100 to 0.0200
\put( 0.0059, 0.1172){\line(1,0){ 0.0059}}
% line 0.8984 from 0.0100 to 0.0200
\put( 0.0059, 0.1016){\line(1,0){ 0.0059}}
% line 0.9219 from 0.0100 to 0.0400
\put( 0.0059, 0.0781){\line(1,0){ 0.0177}}
% line 0.9141 from 0.0100 to 0.0200
\put( 0.0059, 0.0859){\line(1,0){ 0.0059}}
% line 0.9297 from 0.0100 to 0.0200
\put( 0.0059, 0.0703){\line(1,0){ 0.0059}}
% 11110 at 0.0100, 0.9531
\put( 0.0387, 0.0469){\makebox(0,0)[l]{\tt11110}}
% line 0.9688 from 0.0100 to 0.1500
\put( 0.0059, 0.0312){\line(1,0){ 0.0828}}
% 11111 at 0.0100, 0.9844
\put( 0.0387, 0.0156){\makebox(0,0)[l]{\tt11111}}
% line 0.9531 from 0.0100 to 0.0400
\put( 0.0059, 0.0469){\line(1,0){ 0.0177}}
% line 0.9453 from 0.0100 to 0.0200
\put( 0.0059, 0.0547){\line(1,0){ 0.0059}}
% line 0.9609 from 0.0100 to 0.0200
\put( 0.0059, 0.0391){\line(1,0){ 0.0059}}
% line 0.9844 from 0.0100 to 0.0400
\put( 0.0059, 0.0156){\line(1,0){ 0.0177}}
% line 0.9766 from 0.0100 to 0.0200
\put( 0.0059, 0.0234){\line(1,0){ 0.0059}}
% line 0.9922 from 0.0100 to 0.0200
\put( 0.0059, 0.0078){\line(1,0){ 0.0059}}
\end{picture}
}
\end{center}
}{%
\caption[a]{Illustration of the intervals defined by a
simple Bayesian probabilistic model. The size of an intervals is proportional
to the probability of the string.
This model anticipates that
the source is likely to be biased towards one of {\tt{a}} and
{\tt{b}}, so sequences having lots of {\tt{a}}s or lots of
{\tt{b}}s have larger intervals than sequences of the same length
that are 50:50 {\tt{a}}s and
{\tt{b}}s.}
\label{fig.ac2}
}%
\end{figure}
\begin{aside}
\subsection{Details of the Bayesian model}
Having emphasized that any model could be used -- arithmetic coding
is not wedded to any particular set of probabilities -- let me explain
the simple adaptive probabilistic model used in the preceding example;
we first encountered this model
in
% chapter \ref{ch1}
% (page \pageref{ex.postpa})
\exerciseref{ex.postpa}.
%
%
% {\em (This material may be a repetition of material in \chref{ch1}.)}
%
\subsubsection{Assumptions}
The model will be described using parameters
$p_{\eof}$, $p_{\ta}$ and $p_{\tb}$, defined below,
which should not be confused with the predictive
probabilities {\em in a particular context\/},
for example,
$P(\ta \given \bs\eq {\tb\ta\ta} )$.
% An analogy for this model, as I indicated
% at the start,
% is the tossing of a bent coin (\secref{sec.bentcoin}).
A bent coin labelled
$\ta$ and $\tb$ is tossed some number of times $l$,
which we don't know beforehand. The coin's probability
of coming up $\ta$ when tossed is $p_{\ta}$, and $p_{\tb} = 1-p_{\ta}$; the parameters
$p_{\ta},p_{\tb}$ are not known beforehand. The source string $\bs = \tt baaba\eof$
indicates that $l$ was 5 and the sequence of outcomes was $\tt baaba$.
\ben
\item
It is assumed that the length of the string $l$ has an exponential
probability distribution
\beq
P(l) = (1 - p_{\eof})^l p_{\eof}
.
\eeq
This distribution corresponds to assuming a constant probability
$p_{\eof}$ for the termination symbol `$\eof$' at each character.
\item
It is assumed that the non-terminal
characters in the string are selected independently at random
from an ensemble with probabilities
% distribution
$\P = \{p_{\ta},p_{\tb}\}$; the
probability $p_{\ta}$ is fixed throughout the string to some
unknown value that could be anywhere between $0$ and $1$.
The probability of an $\ta$ occurring as the next symbol, given
$p_{\ta}$ (if only we knew it), is $(1-p_{\eof})p_{\ta}$.
% given that it is not
The probability, given $p_{\ta}$, that an unterminated string of length $F$
is a given string $\bs$
that contains $\{F_{\ta},F_{\tb}\}$ counts of the two outcomes
% $\{ a , b \}$
is the \ind{Bernoulli distribution}
\beq
P( \bs \given p_{\ta} , F ) = p_{\ta}^{F_{\ta}} (1-p_{\ta})^{F_{\tb}} .
\label{eq.pa.like}
\eeq
\item
We assume a uniform prior distribution for $p_{\ta}$,
\beq
P(p_{\ta}) = 1 , \: \: \: \: \: \: p_{\ta} \in [0,1] ,
\label{eq.pa.prior}
\eeq
and define $p_{\tb} \equiv 1-p_{\ta}$.
It would be easy to assume other priors on $p_{\ta}$, with beta distributions
being the most convenient to handle.
\een
This model was studied in \secref{sec.bentcoin}.
The key result we require is the predictive distribution for
the next symbol, given the string so far, $\bs$.
This probability that the next character is $\ta$
or $\tb$ (assuming that it is not `$\eof$')
was derived in \eqref{eq.laplacederived} and is precisely
Laplace's rule (\ref{eq.laplaceagain}).
\end{aside}
\exercisaxB{3}{ex.ac.vs.huffman}{
Compare the expected message length
when an ASCII file is compressed by the following
three methods.
\begin{description}
\item[Huffman-with-header\puncspace] Read the whole
file, find the empirical frequency of each symbol,
construct a Huffman code for those frequencies,
transmit the code by transmitting
the lengths of the Huffman codewords, then transmit
the file using the Huffman code.
(The actual codewords don't need to be transmitted,
since we can use a deterministic method for
building the tree given the codelengths.)
\item[Arithmetic code using the \ind{Laplace model}\puncspace]
\beq
P_{\rm L}(\ta \given x_1,\ldots,x_{n-1})=\frac{F_{\ta}+1}
{\sum_{{\ta'}}(F_{{\ta'}}+1)}.
\eeq
\item[Arithmetic code using a \ind{Dirichlet model}\puncspace]
This model's predictions are:
\beq
P_{\rm D}(\ta \given x_1,\ldots,x_{n-1})=\frac{F_{\ta}+\alpha}
{\sum_{{\ta'}}(F_{{\ta'}}+\alpha)},
\eeq
where $\alpha$ is fixed to a number such as 0.01.
A small value of $\alpha$ corresponds to a more responsive version of the
Laplace model; the probability over characters
is expected to be more nonuniform;
$\alpha=1$ reproduces the Laplace model.
\end{description}
Take care that the header of your Huffman message
is self-delimiting.
Special cases worth considering are (a) short files
with just a few hundred characters; (b) large files
in which some characters are never used.
}
\section{Further applications of arithmetic coding}
\subsection{Efficient generation of random samples}
\label{sec.ac.efficient}
Arithmetic coding not only offers a way to compress strings
believed to come from a given model; it also offers a way to generate
random strings from a model. Imagine sticking a
pin into the unit interval at random, that line
having been divided into subintervals in proportion
to probabilities $p_i$; the probability that your pin will
lie in interval $i$ is $p_i$.
So to generate a sample from a model, all we need to do is feed ordinary
random bits into an arithmetic {\em decoder\/}\index{arithmetic coding!decoder} for that
model.\index{arithmetic coding!uses beyond compression} An infinite random
bit sequence corresponds to the selection of a point
at random from the line $[0,1)$, so the decoder will
then select a string at random from the assumed distribution.
This arithmetic method is guaranteed to use very nearly the smallest
number of random bits possible to make the selection -- an important
point in communities where random numbers are expensive!
[{This is
not a joke. Large amounts of money are spent on generating random bits
in software and hardware. Random numbers are valuable.}]
A simple example of the use of this technique is in the
generation of random bits with a nonuniform distribution $\{ p_0,p_1 \}$.
% This is a useful technique
\exercissxA{2}{ex.usebits}{
Compare the following two techniques for generating random symbols
from a nonuniform distribution $\{ p_0,p_1 \} = \{ 0.99,0.01\}$:
\ben
\item The standard method: use a standard random number generator
to generate an integer between 1 and $2^{32}$. Rescale the integer
to $(0,1)$. Test whether this uniformly distributed random variable is
less than $0.99$, and emit a {\tt{0}} or {\tt{1}} accordingly.
\item
Arithmetic coding using the correct model, fed with standard
random bits.
\een
Roughly how many random bits will each method use to generate a thousand
samples from this sparse distribution?
}
\subsection{Efficient data-entry devices}
When we enter text into a computer, we make gestures of some sort --
maybe we tap a keyboard, or scribble with a pointer, or click with a mouse;
an {\em efficient\/}
\index{user interfaces}\index{data entry}\ind{text entry} system is
one where the
number of gestures required to enter a given text string is {\em small\/}.
Writing\index{writing}\index{text entry}%
\marginfignocaption{\small
\begin{center}
\begin{tabular}{rcl}
\multicolumn{3}{l}{Compression:}\\
text& $\rightarrow$ &bits\\[0.2in]
\multicolumn{3}{l}{Writing:} \\
text &$\leftarrow$& gestures\\[0.2in]
\end{tabular}
\end{center}
}
can be viewed as an inverse process\index{arithmetic coding!uses beyond compression}
to data compression. In data compression, the aim is to map
a given text string into a {\em small\/} number of bits.
In text entry, we want a small sequence of gestures
to produce our intended text.
By inverting an arithmetic coder,
we can obtain \index{inverse-arithmetic-coder}an information-efficient
text entry device that is driven by continuous pointing
gestures \cite{ward2000}. In this system, called \ind{Dasher},\index{human--machine interfaces}\index{software!Dasher}
the user zooms in on the unit interval to locate the\index{text entry}
interval corresponding to their intended string,
in the same style as \figref{fig.ac}. A \ind{language
model} (exactly as used in text compression) controls the
sizes of the intervals such that probable strings are
quick and easy to identify.
After an hour's practice,
a novice
user can write with one \ind{finger} driving {Dasher}
at about 25 words per minute -- that's about
half their normal ten-finger
\index{QWERTY}typing speed on a regular \ind{keyboard}.
It's even possible to write at 25 words per minute, {\em hands-free},
using gaze direction to drive Dasher \cite{wardmackay2002}.
Dasher is available as free software for various
platforms.\footnote{ {\tt http://www.inference.phy.cam.ac.uk/dasher/}}
\label{sec.stopbeforeLZ}
\section{Lempel--Ziv coding\nonexaminable}
The \index{Lempel--Ziv coding|(}Lempel--Ziv algorithms, which are widely used for data compression
(\eg, the {\tt\ind{compress}} and {\tt\ind{gzip}} commands), are different in philosophy to arithmetic
coding. There is no separation between modelling and coding,\index{philosophy}
and no opportunity for explicit modelling.\index{source code!algorithms}
\subsection{Basic Lempel--Ziv algorithm}
The method of compression is to replace a \ind{substring} with a \ind{pointer} to
an earlier occurrence of the same substring.
For example if the string is {\tt{1011010100010}}\ldots, we \ind{parse} it into
an ordered {\dem\ind{dictionary}\/} of substrings that have not appeared before
as follows:
$\l$, {\tt{1}}, {\tt{0}}, {\tt{11}}, {\tt{01}}, {\tt{010}}, {\tt{00}}, {\tt{10}}, \dots.
We include the \index{empty string}empty substring \ind{$\lambda$} as
the first substring in the dictionary and order the substrings in the dictionary
by the order in which they emerged from the source.
After every comma, we look along the next part of the
input sequence until we have read a
substring that has not been marked off before. A moment's
reflection will confirm that
this substring is longer by one bit than a substring that has occurred
earlier in the dictionary. This means that we can encode each substring by
giving a {\dem pointer\/} to the earlier occurrence of that prefix and then sending
the extra bit by which the new substring in the dictionary differs from
the earlier substring. If, at the $n$th bit, we have enumerated
$s(n)$ substrings, then we can
give the value of the pointer in
$\lceil \log_2 s(n) \rceil$ bits. The code for the above sequence
is then as shown in the fourth line of the following table (with
punctuation included for clarity), the upper lines indicating the source
string and the value of $s(n)$:
%
%
\beginfullpagewidth%% defined in chapternotes.sty, uses {narrow}
\[
\begin{array}{l|*{8}{l}}
\mbox{source substrings}&\lambda & {\tt{1}} & {\tt{0}} & {\tt{11}} & {\tt{01}} & {\tt{010}} & {\tt{00}} & {\tt{10}} \\
s(n) & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\
s(n)_{\rm binary}
& {\tt{000}} & {\tt{001}} & {\tt{010}} & {\tt{011}} & {\tt{100}} & {\tt{101}} & {\tt{110}} & {\tt{111}} \\
(\mbox{pointer},\mbox{bit})& & (,{\tt{1}}) & ({\tt{0}},{\tt{0}}) & ({\tt{01}},{\tt{1}}) & ({\tt{10}},{\tt{1}}) & ({\tt{100}},{\tt{0}})& ({\tt{010}},{\tt{0}}) & ({\tt{001}},{\tt{0}})
\end{array}
\]
\end{narrow}
% The pointer
Notice that the first pointer we send is empty,
because, given that there is only one substring in the
dictionary -- the string $\lambda$ --
no bits are needed to convey the `choice' of that substring
as the prefix.
The encoded string is {\tt 100011101100001000010}.
The encoding, in this
simple case, is actually a longer string than the source string, because
there was no obvious redundancy in the source string.
\exercisaxB{2}{ex.Clengthen}{
Prove that {\em any\/} uniquely decodeable code from $\{{\tt{0}},{\tt{1}}\}^+$ to
$\{{\tt{0}},{\tt{1}}\}^+$ necessarily makes some strings longer if it makes
some strings shorter.
}
One reason why the algorithm described above lengthens
a lot of strings
is because it is inefficient --
it transmits unnecessary bits; to put it another
way, its code is not complete.\label{sec.LZprune}
% is not necessarily the explanation for the above lengthening,
% however, because
% see also {ex.LZprune}{
% the algorithm described is certainly inefficient: o
Once a substring in
the {dictionary} has been joined there by both of its children, then
we can be sure that it will not be needed (except possibly as part
of our protocol for terminating a message); so at that point we
could drop it from our dictionary of substrings and shuffle them all along
one, thereby reducing the length of subsequent pointer messages. Equivalently,
we could write the second prefix into the dictionary at the point
previously occupied by the parent. A second unnecessary overhead
is the transmission of the new bit in these cases -- the second
time a prefix is used, we can be sure of the identity of the next bit.
% This is easy to do in a computer but not so easy for a human
% to cope with.
\subsubsection{Decoding}
The decoder again involves an identical twin at the decoding end
who constructs the dictionary of substrings as the
data are decoded.
\exercissxB{2}{ex.LZencode}{
Encode the string {\tt{000000000000100000000000}}
using the basic
Lempel--Ziv algorithm described above.
}
% lambda 0 00 000 0000 001 00000 000000
% 000 001 010 011 100 101 110 111
% ,0 1,0 10,0 11,0 010,1 100,0 110,0
% answer
% 010100110010110001100
\exercissxB{2}{ex.LZdecode}{
Decode the string
\begin{center}
{\tt{00101011101100100100011010101000011}}
\end{center}
that was encoded using the basic
Lempel--Ziv algorithm.
}
% answer
% 0100001000100010101000001 001000001000000
% lamda, 0, 1, 00, 001, 000, 10, 0010, 101, 0000, 01, 00100, 0001, 00000
% 0 , 1, 10,11, 100, 101,110, 111, 1000, 1001, 1010, 1011,1100, 1101
% ,0 0,1 01,0 11,1 011,0 010,0 100,0 110,1 0101,0 0001,1 bored!
% 10101011101100100100011010101000011
%
% see tcl/lempelziv.tcl
\subsubsection{Practicalities}
In this description I have not discussed the method for terminating
a string.
There are many variations on the Lempel--Ziv algorithm, all exploiting
the same idea but using different procedures for dictionary management,
etc.
% Two of the best known
% variations are called the Ziv-Lempel algorithm
% and the LZW algorithm.
%
The resulting programs are fast, but their performance on compression
of English text, although useful,
does not match the standards set in the arithmetic coding literature.
\subsection{Theoretical properties}
In contrast to the block code, Huffman code, and arithmetic coding
methods we discussed in the last three chapters,
the Lempel--Ziv algorithm is defined without making any mention
of a \ind{probabilistic model} for the source. Yet,
% in fact,
given any \ind{ergodic}
%\footnote{Need to clarify this. It means
% the source is memoryless on sufficiently long timescales.}
source (\ie, one that is memoryless on sufficiently long timescales),
the Lempel--Ziv algorithm can be
proven {\em asymptotically\/} to
compress down to the entropy of the source. This is why it is called
a `\ind{universal}' compression algorithm. For a proof
of this property, see \citeasnoun{Cover&Thomas}.
% Cover and Thomas (1991).
It achieves its compression,
however, only by {\em memorizing\/} substrings that have happened
so that it has a short name for them the next time they occur.
The asymptotic timescale on which this universal performance
is achieved
%%is likely to be the time that it takes for
%% if the source has not been observed long enough for
% {\em all\/} typical sequences of length $n^*$
% to occur, where $n^*$ is the longest lengthscale associated with the
% statistical fluctuations in the source.
%the longest lengthscale on
% which there are correlations in .
% red then
% For many sources the time for all typical sequences to
% occur is
may, for many sources, be unfeasibly long, because
the number of typical substrings that need memorizing
may be enormous.
%
The useful performance
of the algorithm in practice is a reflection of the fact that
many files contain multiple repetitions of particular
short sequences of characters,
a form of redundancy to which the algorithm is well suited.
\subsection{Common ground}
I have emphasized the difference in philosophy behind arithmetic coding
and Lempel--Ziv coding. There is common ground
between them, though: in principle, one can design
adaptive probabilistic models, and thence arithmetic codes, that
are `\ind{universal}', that is, models that will asymptotically compress
{\em any source in some class\/} to within some factor (preferably 1)
of its entropy.\index{compression!universal}
However, {for practical purposes\/}, I think such universal models can only be
constructed if the class of sources is severely restricted.
A general purpose compressor that can discover the probability
distribution of {\em any\/} source would be a general purpose
\ind{artificial intelligence}! A general purpose artificial
intelligence does not yet exist.
% \subsection{Comments}
% The Lempel--Ziv algorithm can be generalized to any finite alphabet
% as long as the input and output alphabets are the same. I believe
% it is not convenient to use unequal alphabets.
\section{Demonstration}
An interactive aid for exploring arithmetic coding, {\tt dasher.tcl}, is
available.\footnote{{\tt http://www.inference.phy.cam.ac.uk/mackay/itprnn/softwareI.html}}
% http://www.inference.phy.cam.ac.uk/mackay/itprnn/code/tcl/dasher.tcl
A demonstration arithmetic-coding\index{source code!algorithms}
\index{arithmetic coding!software}\index{software!arithmetic coding}software
package written by \index{Neal, Radford}{Radford Neal}\footnote{%
% is available from \\ \noindent
{\tt ftp://ftp.cs.toronto.edu/pub/radford/www/ac.software.html}}
% This package
consists of encoding and decoding modules to which the
user adds a module defining the probabilistic model. It
should be emphasized that there is no single
general-purpose arithmetic-coding compressor; a new model has to be written
for each type of source.
% application.
%
Radford Neal's\index{Neal, Radford}
package includes a simple adaptive model similar to the
Bayesian model demonstrated in section \ref{sec.ac}.
The results using this Laplace model should
be viewed as a basic benchmark since it is
the simplest possible probabilistic model -- it
% These results are anecdotal and should not be taken too
% seriously, but it is interesting that the highly developed gzip
% software only does a little better than the benchmark
% of the simple Laplace model,
simply assumes the characters in the file come independently
from a fixed ensemble.
The counts $\{ F_i \}$ of the symbols $\{ a_i \}$ are rescaled
and rounded as the file is read such that all the counts lie
between 1 and 256.
\index{DjVu}\index{deja vu}\index{Le Cun, Yann}\index{Bottou, Leon}
% Yann Le Cun, Leon Bottou and colleagues at AT{\&}T Labs
% have written a
A state-of-the-art compressor for documents
containing text and images, {\tt{DjVu}},
uses arithmetic coding.\footnote{%
% {\tt{DjVu}} is described at
\tt http://www.djvuzone.org/}
% (better Reference for deja vu?)
It uses a carefully designed approximate
arithmetic coder for binary
alphabets called the Z-coder \cite{bottou98coder},
which is much faster than the
arithmetic coding software described above. One of
the neat tricks the Z-coder uses is this: the adaptive model
adapts only occasionally (to save on computer time),
with the decision about when to adapt being pseudo-randomly
controlled by
whether the arithmetic encoder emitted a bit.
The JBIG image compression standard for binary images
uses arithmetic coding with a context-dependent
model, which adapts using a rule similar to Laplace's rule.
PPM \cite{Teahan95a} is a leading method for text compression,
and it uses arithmetic coding.
There are many Lempel--Ziv-based programs.
{\tt gzip} is based on a version of Lempel--Ziv
called `{\tt LZ77}' \cite{Ziv_Lempel77}\nocite{Ziv_Lempel78}. {\tt compress} is based on `{\tt LZW}'
\cite{Welch84}.
In my experience the
best is {\tt gzip}, with {\tt compress} being inferior
on most files.
% To
% give further credit to {\tt gzip}, it stores additional information in
% the compressed file such as the name of the file and its
% last modification date.
{\tt bzip} is
a {\dem{\ind{block-sorting} file compressor\/}}, which makes
use of a neat hack called the {\dem\ind{Burrows--Wheeler transform}}\index{source code!Burrows--Wheeler transform}\index{source code!block-sorting compression}
\cite{bwt}. This method is not based on an explicit probabilistic
model, and it only works well for files larger than several
thousand characters; but in practice it is a very effective
compressor for files in which the context of a character
is a good predictor for that character.%
% Maybe I'll describe it in a future edition of this
% book.
\footnote{There is a lot of information about the
Burrows--Wheeler transform on the net.
{\tt{http://dogma.net/DataCompression/BWT.shtml}}
}
%bzip2 compresses files using the Burrows--Wheeler block-sorting text compression algorithm, and Huffman
%coding. Compression is generally considerably better than that achieved by more conventional
%LZ77/LZ78-based compressors, and approaches the performance of the PPM family of statistical
%compressors.
\subsubsection{Compression of a text file}
Table \ref{tab.zipcompare1} gives the computer time in seconds taken and the
compression achieved when these programs are applied to
the \LaTeX\ file containing the text of
this chapter, of size 20,942 bytes.
\begin{table}[htbp]
\figuremargin{
\begin{center}
\begin{tabular}{lccc} \toprule
Method & Compression & Compressed size & Uncompression \\
& time$ \,/\, $sec & (\%age of 20,942) & time$ \,/\, $sec \\ \midrule
%Adaptive encoder,
Laplace model &
0.28 & $12\,974$ (61\%) & 0.32 \\
%{\tt gzip / gunzip} &
{\tt gzip} &
0.10 & \hspace{0.06in}$ 8\,177$ (39\%) & {\bf 0.01} \\
{\tt compress}
%/ uncompress}
&
0.05 & $10\,816$ (51\%) & 0.05 \\ \midrule
{\tt bzip}
% / bunzip}
&
& \hspace{0.06in}$ 7\,495$ (36\%) & \\
{\tt bzip2}
%/ bunzip2}
&
& \hspace{0.06in}$ 7\,640$ (36\%) & \\
{\tt ppmz } &
& \hspace{0.06in}{\bf 6$\,$800 (32\%)} & \\
\bottomrule
\end{tabular}
\end{center}
}{
\caption[a]{Comparison of compression algorithms applied to a text file.
}
\label{tab.zipcompare1}
}
\end{table}
% I will report the value of ``u''
% django:
% 0.410u 0.060s 0:00.60 78.3% 0+0k 0+0io 109pf+0w
% 6800 Nov 25 18:05 ../l4.tex.ppm
% time ppmz ../l4.tex.ppm ../l4.tex.up
% 0.480u 0.040s 0:00.60 86.6% 0+0k 0+0io 109pf+0w
%
% 108:wol:/home/mackay/_tools/ac0> time adaptive_encode < ~/_courses/itprnn/l4.tex > l4.tex.aez
% 0.280u 0.040s 0:00.55 58.1% 0+105k 2+3io 0pf+0w
% 109:wol:/home/mackay/_tools/ac0> time gzip ~/_courses/itprnn/l4.tex
% 0.100u 0.060s 0:00.28 57.1% 0+161k 2+12io 0pf+0w
% 110:wol:/home/mackay/_tools/ac0> ls -lisa ~/_courses/itprnn/l4.tex.gz
% 110131 8 8177 Jan 10 15:40 /home/mackay/_courses/itprnn/l4.tex.gz
% 111:wol:/home/mackay/_tools/ac0> gunzip ~/_courses/itprnn/l4.tex.gz
% 112:wol:/home/mackay/_tools/ac0> ls -lisa ~/_courses/itprnn/l4.tex l4.tex.aez
% 109904 21 20942 Jan 10 15:40 /home/mackay/_courses/itprnn/l4.tex
% 444691 13 12974 Jan 10 15:40 l4.tex.aez
% 113:wol:/home/mackay/_tools/ac0> time gzip ~/_courses/itprnn/l4.tex
% 0.100u 0.050s 0:00.24 62.5% 0+150k 0+13io 0pf+0w
% 114:wol:/home/mackay/_tools/ac0> time gunzip ~/_courses/itprnn/l4.tex.gz
% 0.010u 0.060s 0:00.17 41.1% 0+80k 0+8io 0pf+0w
% 115:wol:/home/mackay/_tools/ac0> time adaptive_decode < l4.tex.aez > l4.tex
% 0.320u 0.030s 0:00.39 89.7% 0+101k 6+4io 5pf+0w
%
% django: bzip and gunzip:
% 149:django.ucsf.edu:/home/mackay/_tools/ac0> time bzip l4.tex
% BZIP, a block-sorting file compressor. Version 0.21, 25-August-96.
% 0.060u 0.020s 0:00.22 36.3% 0+0k 0+0io 107pf+0w
% 7495 Jan 10 1997 l4.tex.bz
% 153:django.ucsf.edu:/home/mackay/_tools/ac0> time bunzip l4.tex.bz
% 0.020u 0.010s 0:00.14 21.4% 0+0k 0+0io 93pf+0w
% 20942 Jan 10 1997 l4.tex
% 155:django.ucsf.edu:/home/mackay/_tools/ac0> time bzip2 l4.tex
% 0.050u 0.000s 0:00.37 13.5% 0+0k 0+0io 90pf+0w
% 7640 Jan 10 1997 l4.tex.bz2
% 157:django.ucsf.edu:/home/mackay/_tools/ac0> time bunzip2 l4.tex.bz2
% 0.020u 0.000s 0:00.15 13.3% 0+0k 0+0io 85pf+0w
% time gzip l4.tex
% 0.010u 0.010s 0:00.28 7.1% 0+0k 0+0io 84pf+0w
% 8177 Jan 10 1997 l4.tex.gz
% time gunzip l4.tex
% 0.000u 0.010s 0:00.12 8.3% 0+0k 0+0io 87pf+0w
%
\subsubsection{Compression of a sparse file}
Interestingly, {\tt gzip} does not always do so well.
Table \ref{tab.zipcompare2} gives the
% computer time in seconds taken and the
compression achieved when these programs are applied to
a text file containing $10^6$ characters, each of which is
either {\tt0} and {\tt1} with probabilities
0.99 and 0.01. The Laplace model is quite
well matched to this source,
and the benchmark arithmetic coder
gives good performance, followed closely by {\tt compress}; {\tt gzip}
% , interestingly,
is worst.
% see /home/mackay/_tools/ac0
%
% , and {\tt gzip --best} does no better.
% has identical performance to {\tt gzip} on this example.}]
An ideal model for this source would compress the
file into about $10^6 H_2(0.01)/8 \simeq 10\,100$ bytes. The Laplace model
compressor falls short of this performance because it is implemented
using only eight-bit precision. The {\tt{ppmz}} compressor compresses
the best of all, but takes much more computer time.\index{Lempel--Ziv coding|)}
\begin{table}[htbp]
\figuremargin{
\begin{center}
\begin{tabular}{lccc} \toprule
Method & Compression & Compressed size & Uncompression \\
& time$ \,/\, $sec & $ \,/\, $bytes & time$ \,/\, $sec \\ \midrule
% Adaptive encoder,
% Laplace model &
% 6.4 & 14089 (1.4\%)\hspace{0.06in} & 9.2 \\
%{\tt gzip / gunzip} &
% 2.1 & 20548 (2.1\%)\hspace{0.06in} & 0.43 \\
%{\tt compress / uncompress} &
% 0.73 & 14692 (1.47\%) & 0.76 \\ \bottomrule
%{\tt bzip / bunzip} &
% & & (\%) & \\
%{\tt bzip2 / bunzip2} &
% & & (\%) & \\ \hline
Laplace model &
0.45 & $14\,143$ (1.4\%)\hspace{0.06in} & 0.57 \\
{\tt gzip } &
0.22 & $20\,646$ (2.1\%)\hspace{0.06in} & 0.04 \\
{\tt gzip {\tt-}{\tt-}best+} &
%{\tt gzip \verb+--best+} &
1.63 & $15\,553$ (1.6\%)\hspace{0.06in} & 0.05 \\
{\tt compress} &
0.13 & $14\,785$ (1.5\%)\hspace{0.06in} & 0.03 \\ \midrule
{\tt bzip } &
0.30 & $10\,903$ (1.09\%) & 0.17 \\
{\tt bzip2} &
0.19 & $11\,260$ (1.12\%) & 0.05 \\
{\tt ppmz} &
533 & {\bf 10$\,$447 (1.04\%)} & 535 \\
\bottomrule
\end{tabular}
\end{center}
% ideal length = 0.0807931 * 10^6 = 80793 bits = 10099 bytes
% /home/mackay/_tools/ac0/README1
}{
\caption[a]{Comparison of compression algorithms applied to a random file
of $10^6$ characters, 99\% {\tt0}s and 1\% {\tt1}s.
}
\label{tab.zipcompare2}
}
\end{table}
\section{Summary}
In the last three chapters
we have studied three classes of data compression codes.
\begin{description}
\item[Fixed-length block codes] (Chapter \chtwo). These are mappings
from a fixed number of source symbols to a fixed-length binary message.
% Most source strings are given no encoding;
Only a tiny fraction of
the source strings are given an encoding.
These codes were fun for identifying the entropy as the measure
of compressibility but they are of little practical use.
\item[Symbol codes] (Chapter \chthree). Symbol codes employ a variable-length
code for each symbol in the source alphabet, the codelengths being
integer lengths determined by the probabilities of the symbols.
Huffman's algorithm constructs an optimal symbol code for a given
set of symbol probabilities.
Every source string has a uniquely decodeable encoding, and if
the source symbols come from the assumed distribution then the symbol
code will compress
to an expected length $L$ lying in the interval $[H,H\!+\!1)$.
Statistical fluctuations in the source may make the actual length
longer or shorter than this mean length.
If the source is not well matched to the assumed distribution then
the mean length is increased by the relative entropy $D_{\rm KL}$
between the source distribution and the code's implicit distribution.
For sources with small entropy, the symbol has to emit
at least one bit per source symbol; compression
below one bit per source symbol can only be achieved
by the cumbersome procedure of putting the source data into blocks.
\item[Stream codes\puncspace]
The distinctive property of stream codes, compared with
symbol codes, is that they are not constrained to emit at least one bit for every
symbol read from the source stream. So large numbers of
source symbols may be
coded into a smaller number of bits.
% , but unlike block codes, this is achieved
This property could only be obtained using a symbol code
if the source stream were somehow chopped into blocks.
\bit
\item {Arithmetic codes}
combine a probabilistic model with an encoding algorithm
that identifies each string with a sub-interval of $[0,1)$
of size equal to the probability of that string under the model.
This code is almost optimal in the sense that
the compressed length of a string $\bx$ closely matches
the Shannon information content of $\bx$ given
the probabilistic model. Arithmetic codes fit with
the philosophy that good compression requires
%intelligence
{\dem data modelling}, in the form of an adaptive Bayesian model.
\item
% [Stream codes: Lempel--Ziv codes\puncspace]
Lempel--Ziv codes are adaptive in the sense that they memorize strings
that have already occurred. They are built on the philosophy that
we don't know anything at all about
what the probability distribution of the source will be, and we want
a compression algorithm that will perform reasonably well
whatever that distribution is.
\eit
\end{description}
%\section{Optimal compression must involve artificial intelligence}
%\subsection{A rant about `universal' compression}
% moved this to rant.tex for the time being
Both arithmetic codes and Lempel--Ziv codes will fail to decode
correctly if any of the bits of the compressed file are altered.
So if compressed files are to be stored or transmitted over
noisy media, error-correcting codes will be essential.
Reliable communication over unreliable channels is
the topic of \partnoun\ \noisypart.
% the next few chapters.
%Exercises
\section{Exercises on stream codes}%{Problems}
\exercisaxA{2}{ex.AC52}{
Describe an arithmetic coding algorithm to encode random
bit strings of length $N$ and weight $K$ (\ie, $K$ ones and $N-K$
zeroes) where $N$ and $K$ are given.
For the case $N\eq 5$, $K \eq 2$ show in detail the intervals corresponding to
all source substrings of lengths 1--5.
}
\exercissxB{2}{ex.AC52b}{
How many bits are needed to specify a selection of
% an unordered collection of
$K$ objects from $N$ objects? ($N$ and $K$ are assumed to be known and
the selection of $K$ objects is unordered.)
How might such a selection
be made at random without being wasteful of random bits?
}
\exercisaxB{2}{ex.HuffvAC}{
% from 2001 exam
A binary source $X$ emits independent identically
distributed symbols with probability distribution $\{ f_{0},f_1 \}$,
where $f_1 = 0.01$.
Find an optimal uniquely-decodeable symbol code for a string
$\bx=x_1x_2x_3$ of {\bf{three}} successive
samples from this source.
Estimate (to one decimal place) the factor
by which the expected length of this optimal code is greater
than the entropy of the three-bit string $\bx$.
[$H_2(0.01) \simeq 0.08$, where
$H_2(x) = x \log_2 (1/x) + (1-x) \log_2 (1/(1-x))$.]
%\medskip
An {{arithmetic code}\/} is used to compress a string of $1000$ samples
from the source $X$. Estimate the mean and standard deviation of
the length of the compressed file.
% This is example 6.3, identical, except we are talking about compressing
% rather than generating.
}
\exercisaxB{2}{ex.ACNf}{
Describe an arithmetic coding algorithm to generate random
bit strings of length $N$ with density $f$ (\ie, each
bit has probability $f$ of being a one) where $N$ is given.
}
\exercisaxC{2}{ex.LZprune}{
Use a modified Lempel--Ziv algorithm in which, as discussed
on \pref{sec.LZprune}, the dictionary of prefixes
is
% effectively
pruned by writing new prefixes into the
space occupied by prefixes that will not be needed again.
Such prefixes can be identified when
both their children have been added to the dictionary of prefixes.
(You may neglect the issue of termination of encoding.)
Use this algorithm to encode the string
{\tt{0100001000100010101000001}}.
Highlight the bits that follow a prefix on the
second occasion that that prefix is used. (As discussed earlier,
these bits could be omitted.)
% from the encoding if we adopted the convention (discussed
% earlier)
% of not transmitting the bit that follows a prefix on the
% second occasion that that prefix is used.
% nb this is same as an earlier example.
% i get
% ,0 0,1 1,0 10,1 10,0 00,0 011,0 100,1 010,0 001,1
}
\exercissxC{2}{ex.LZcomplete}{
Show that this modified Lempel--Ziv code is still not `complete',
that is, there are binary strings that are not encodings of any string.
}
% answer: this is because there are illegal prefix names, e.g. at the
% 5th step, 111 is not legal.
%
\exercissxB{3}{ex.LZfail}{
Give examples of simple sources that have low entropy
but would not be compressed well by the Lempel--Ziv algorithm.
}
%
% Ideas: add a figure showing the flow diagram -- source, model.
%
%
% \begin{thebibliography}{}
% \bibitem[\protect\citeauthoryear{Witten {\em et~al.\/}}{1987}]{arith_coding}
% {\sc Witten, I.~H.}, {\sc Neal, R.~M.}, \lsaand {\sc Cleary, J.~G.}
% \newblock (1987)
% \newblock Arithmetic coding for data compression.
% \newblock {\em Communications of the ACM\/} {\bf 30} (6):~520-540.
%
% \end{thebibliography}
% \part{Noisy Channel Coding}
% \end{document}
\dvips
%
\section{Further exercises on data compression}
%\chapter{Further Exercises on Data Compression}
\label{ch_f4}
%
% _f4.tex: exercises to follow chapter 4 in a 'review, revision, further topics'
% exercise zone.
%
\fakesection{Post-compression general extra exercises}
The following exercises may be skipped by the reader who
is eager to learn about noisy channels.
%
% DOES THIS BELONG HERE? Maybe move to p92.
%
\fakesection{RNGaussian}
\exercissxA{3}{ex.RNGaussian}{
\index{life in high dimensions}\index{high dimensions, life in}
%
Consider a Gaussian distribution\index{Gaussian distribution!$N$--dimensional} in $N$ dimensions,
\beq
P(\bx) = \frac{1}{(2 \pi \sigma^2)^{N/2}} \exp \left( - \frac{\sum_n x_n^2}{2 \sigma^2} \right) .
\label{first.gaussian}
\eeq
% Show that
Define the radius of a point $\bx$ to be $r = \left( {\sum_n
x_n^2} \right)^{1/2}$.
Estimate the mean and variance of the square of the radius,
$r^2 = \left( {\sum_n x_n^2} \right)$.
\begin{aside}%{\small
You may find helpful the integral
\beq
\int \! \d x\: \frac{1}{(2 \pi \sigma^2)^{1/2}} \: x^4
\exp \left( - \frac{x^2}{2 \sigma^2} \right) = 3 \sigma^4 ,
\label{eq.gaussian4thmoment}
\eeq
though you should be able to estimate the required quantities
without it.
\end{aside}
% If you like gamma integrals
% derive the probability density of the radius $r = \left( {\sum_n
% x_n^2} \right)^{1/2}$, and find the most probable
% radius.
%\amarginfig{b}{% in first printing, before asides changed
\amarginfig{t}{%
\setlength{\unitlength}{0.7mm}
% there is a strip without ink at the left, hence I use -19
% instead of -21 as the left coordinate
\begin{picture}(42,42)(-19,-21)% original is 6in by 6in, so 7unitlength=1in
% use 42 unitlength for width
\put(-21,-21){\makebox(42,42){\psfig{figure=figs/typicalG.ps,angle=-90,width=29.4mm}}}
%\put(14,14){\makebox(0,0)[l]{\small probability density is maximized here}}
\put(10,18){\makebox(0,0)[bl]{\small probability density}}
\put(13,13){\makebox(0,0)[bl]{\small is maximized here}}
%\put(14,-14){\makebox(0,0)[l]{\small almost all probability mass is here}}
\put(9,-16){\makebox(0,0)[l]{\small almost all}}
\put(2,-21){\makebox(0,0)[l]{\small probability mass is here}}
%\put(15,-26){\makebox(0,0)[l]{\small is here}}
\put(-2,-2){\makebox(0,0)[tr]{\small $\sqrt{N} \sigma$}}
\end{picture}
\caption[a]{Schematic representation of the typical
set of an $N$-dimensional Gaussian distribution.}
}
Assuming that $N$ is large,
show that nearly all the probability of a Gaussian is contained in
a \ind{thin shell} of radius $\sqrt{N} \sigma$. Find the thickness of the
shell.
Evaluate the probability density
% in $\bx$ space
(\ref{first.gaussian}) at a point in
that thin shell and at the origin $\bx=0$ and compare.
Use the case $N=1000$ as an example.
Notice that nearly all the probability mass
% the bulk of the probability density
is located in a
different part of the space from the region of highest probability
density.
%
}
%
% extra exercises that are appropriate once source compression has been
% discussed.
%
% contents:
%
% simple huffman question
% Phone chat using rings (originally in mockexam.tex, now in M.tex)
% Bridge bidding as communication (where?)
%
\fakesection{Compression exercises: bidding in bridge, etc}
%
\exercisaxA{2}{ex.source_code}{
%
Explain what is meant by an {\em optimal binary symbol code\/}.
Find an optimal binary symbol code for the ensemble:
\[
\A = \{ {\tt{a}},{\tt{b}},{\tt{c}},{\tt{d}},{\tt{e}},{\tt{f}},{\tt{g}},{\tt{h}},{\tt{i}},{\tt{j}} \} ,
\]
\[
\P = \left\{ \frac{1}{100} ,
\frac{2}{100} ,
\frac{4}{100} ,
\frac{5}{100} ,
\frac{6}{100} ,
\frac{8}{100} ,
\frac{9}{100} ,
\frac{10}{100} ,
\frac{25}{100} ,
\frac{30}{100} \right\} ,
\]
and compute the expected length of the code.
}
\exercisaxA{2}{ex.doublet.huffman}{
A string $\by=x_1 x_2$ consists of {\em two\/} independent samples from an ensemble
\[
X : {\cal A}_X = \{ {\tt{a}} , {\tt{b}} , {\tt{c}} \} ; {\cal P}_X = \left\{ \frac{1}{10} , \frac{3}{10} ,
\frac{6}{10} \right\} .
\]
What is the entropy of $\by$?
Construct an optimal binary symbol code for the string $\by$, and find
its expected length.
}
\exercisaxA{2}{ex.ac_expected}{
% (Cambridge University Part III Maths examination, 1998.)
%
Strings of $N$ independent samples from an ensemble
with $\P = \{ 0.1 , 0.9 \}$ are compressed using
an {arithmetic code} that is matched to that ensemble.
Estimate the mean and standard deviation of
the compressed strings' lengths
for the case $N=1000$.
%
[$H_2(0.1) \simeq 0.47$]
% ; $\log_2(9) \simeq 3$.]
% .47, 3.17
% my answer: 470 pm 30
}
% from M.tex, in which model solns are found too
\exercisaxA{3}{ex.phone_chat}{%(Cambridge University Part III Maths examination, 1998.)
{\sf Source coding with variable-length symbols.}
% -- Source coding / optimal use of channel}
\begin{quote}
In the chapters on source coding, we assumed that
we were encoding into a binary alphabet $\{ {\tt0} , {\tt1} \}$ in which both symbols\index{source code!variable symbol durations}
% had the same associated cost. Clearly a good compression algorithm
% uses both these symbols with equal frequency, and the capacity of
% this alphabet is one bit per character.
should be used with equal frequency.
In this question we explore how the encoding alphabet should be
used
% what happens
if the symbols take different times to transmit.
% have different costs.
% the
\end{quote}
%
A poverty-stricken \ind{student} communicates for free with a friend
using a \ind{telephone} by selecting an integer
$n \in \{ 1,2,3\ldots \}$,
making the friend's
phone ring $n$ times, then hanging up in the middle of the $n$th ring.
This process is repeated so that a string of symbols
$n_1 n_2 n_3 \ldots$ is received. What is the optimal way to communicate?
If large integers $n$ are selected
then the message takes longer to communicate. If only
small integers $n$ are used then the information content per symbol is
small.
We aim to maximize the rate of information transfer, per unit time.
Assume that the time taken to transmit
a number of rings $n$ and to redial
%, including the space that separates them from the next sequence of rings
is $l_n$ seconds. Consider a probability distribution over $n$,
$\{ p_n \}$.
Defining the average duration {\em per symbol\/} to be
\beq
L(\bp) = \sum_n p_n l_n
\eeq
and the entropy {\em per symbol\/} to be
\beq
H(\bp) = \sum_n p_n \log_2 \frac{1}{p_n } ,
\eeq
show that for the average information
rate {\em per second\/} to be maximized,
the symbols must be used with probabilities
of the form
\beq
p_n = \frac{1}{Z} 2^{-\beta l_n}
\label{eq.phone.1}
\eeq
where
% $\beta$ is a Lagrange multiplier
%and
$Z = \sum_n 2^{-\beta l_n}$
and $\beta$ satisfies the implicit equation
% \marginpar{[6]}
\beq
\beta = \frac{H(\bp)}{L(\bp)} ,
\label{eq.phone.2}
\eeq
that is, $\beta$ is the rate of communication.
%is set so as to maximize
%\beq
% R(\beta) = - \beta - \frac{\log Z(\beta)}{L(\beta)}
%\eeq
% where $L(\beta)=\sum p_n l_n$.
% By differentiating $R(\beta)$, show that
% $\beta^*$ satisfies
Show that these two equations
(\ref{eq.phone.1}, \ref{eq.phone.2}) imply that $\beta$ must be set
such that
\beq
\log Z =0.
\label{eq.phone.3}
\eeq
%
Assuming that the channel has the property
% redialling takes the same time as one ring, so that
\beq
l_n = n \: \mbox{seconds},
\label{eq.phone.4}
\eeq
find the optimal distribution $\bp$ and show that
the maximal information rate is 1 bit per second.
% $\log xxxx$
% and that the mean number of rings
% in a group is xxxx and that the information per
% ring is xxxx.
How does this compare with the information rate per second achieved
if $\bp$ is set to
$(1/2,1/2,0,0,0,0,\ldots)$ --- that is,
only the symbols $n=1$ and $n=2$ are selected,
and they have equal probability?
Discuss the relationship between the results
(\ref{eq.phone.1}, \ref{eq.phone.3}) derived above,
and the Kraft inequality from source coding theory.
How might a random binary source
be efficiently encoded into a sequence of symbols
$n_1 n_2 n_3 \ldots$ for transmission over the channel defined
in \eqref{eq.phone.4}?
}
\exercisaxB{1}{ex.shuffle}{How many bits
does it take to shuffle a pack of cards?
% [In case this is not clear, here's the long-winded
% version: imagine using a random number generator
% to generate perfect shuffles of a deck of cards.
% What is the smallest number of random bits
% needed per shuffle?]
}
\exercisaxB{2}{ex.bridge}{In the card game\index{game!Bridge}
Bridge,\index{Bridge}
the four players receive 13 cards each from the deck of 52 and
start each game by looking at their own hand
and bidding. The legal bids are, in ascending order
$1 \clubsuit, 1 \diamondsuit, 1 \heartsuit, 1\spadesuit,$ $1NT,$
$2 \clubsuit,$ $2 \diamondsuit,$
% 2 \heartsuit, 2\spadesuit, 2NT,
$\ldots$
% 7 \clubsuit, 7 \diamondsuit,
$7 \heartsuit, 7\spadesuit, 7NT$,
and successive bids must follow this order;
a bid of, say, $2 \heartsuit$ may only be
followed by higher bids such as $2\spadesuit$ or $3 \clubsuit$ or $7 NT$.
(Let us neglect the `double' bid.)
% The outcome of the bidding process determines the subsequent
% game.
The players have several aims when bidding. One of the
aims is for two partners to communicate to each other
as much as possible about what cards are in their hands.
% There are many bidding systems whose aim is, among other things,
% to communicate this information.
Let us concentrate on this task.
\begin{enumerate}
\item
After the cards have been dealt,
how many bits are needed for North to convey to South what
her hand is?
\item
Assuming that E and W do not bid at all, what
is the maximum total information that N and S can convey to each
other while bidding? Assume that N starts the bidding, and that
once either N or S stops bidding, the bidding stops.
\end{enumerate}
}
\exercisaxB{2}{ex.microwave}{
My old `\ind{arabic}' \ind{microwave oven}\index{human--machine interfaces}
had 11 buttons for entering
cooking times, and my new `\ind{roman}' microwave has just five.
The buttons of the roman microwave are labelled `10 minutes',
`1 minute', `10 seconds', `1 second', and `Start'; I'll abbreviate
these five strings to the symbols {\tt M}, {\tt C}, {\tt X}, {\tt I}, $\Box$.
% The two keypads then look as follows.
% included by _e4.tex
\amarginfig{b}{%
\begin{center}
\begin{tabular}[t]{c}%%%%%%%%%% table containing microwave buttons
%\toprule
Arabic \\ \midrule
% The keypad
\begin{tabular}[t]{*{3}{p{.1in}}}
\framebox{1} & \framebox{2} & \framebox{3} \\
\framebox{4} & \framebox{5} & \framebox{6} \\
\framebox{7} & \framebox{8} & \framebox{9} \\
& \framebox{0} & \framebox{$\!\Box\!$} \\
\end{tabular}
\\
%\bottomrule
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% end all micro table
\end{tabular}
\begin{tabular}[t]{c}%%%%%%%%%% table containing microwave buttons
%\toprule
Roman \\ \midrule
% The keypad
\begin{tabular}[t]{*{3}{p{.1in}}}
\framebox{{\tt{M}}} & \framebox{{\tt{X}}} & \\
\framebox{{\tt{C}}} & \framebox{{\tt{I}}} & \framebox{$\!\Box\!$} \\
\end{tabular}
\\
%\bottomrule
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% end all micro table
\end{tabular}\\
\mbox{$\:$}
\end{center}
\caption[a]{Alternative keypads for microwave ovens.}
}
To enter one minute and twenty-three seconds (1:23), the arabic sequence
is
\beq
{\tt{123}}\Box,
\eeq
and the roman sequence is
\beq
{\tt{CXXIII}}\Box .
\eeq
Each of these keypads defines a code mapping the
3599 cooking times from 0:01 to 59:59 into a string of symbols.
\ben
\item
Which times can be produced with two or three symbols? (For example,
0:20 can be produced by three symbols in either code:
${\tt{XX}}\Box$ and
${\tt{20}}\Box$.)
\item
Are the two codes complete?
Give a detailed answer.
% Discuss all the ways in which these two codes are not complete.
\item
For each code, name a cooking time
% couple of times
that it can produce in
four symbols that the other code cannot.
\item
Discuss the implicit probability distributions over times to which
each of these codes is best matched.
\item
Concoct a plausible probability distribution over times
that a real user might use, and evaluate roughly the expected number of
symbols, and maximum number of symbols, that each code
requires. Discuss the ways in which
each code is inefficient or efficient.
\item
Invent a more efficient cooking-time-encoding system for a microwave oven.
\een
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
}
%
\fakesection{Cinteger}
%\input{tex/_Cinteger}
\exercissxC{2}{ex.Cinteger}{
Is the standard binary representation for positive
integers (\eg\ $c_{\rm b}(5) = {\tt 101}$)
a uniquely decodeable code?
Design a binary code for the positive integers,
\ie, a mapping from $n \in \{ 1,2,3,\ldots \}$ to $c(n) \in
\{{\tt 0},{\tt 1}\}^+$,
that is uniquely decodeable.
Try to design codes that are prefix codes and that satisfy the
\Kraft\ equality $\sum_n 2^{-l_n} \eq 1$.
%
% Not a typo.
%
\begin{aside}
Motivations: any data file terminated by a special
end of file character can be mapped onto an integer,
so a prefix code for integers can be used as a self-delimiting
encoding of files too. Large files correspond to large integers.
Also, one of the building blocks of a `universal' coding scheme --
that is, a coding scheme that will work OK for a large variety
of sources -- is the ability to encode integers. Finally,
in microwave ovens, cooking times are positive integers!
\end{aside}
Discuss criteria by which one might compare alternative codes
for integers (or, equivalently, alternative self-delimiting codes for
files).
}
%
%
%
\section{Solutions}% to Chapter \protect\ref{ch4}'s exercises}
%
% solns to exercises in l4.tex
%
\fakesection{solns to exercises in l4.tex}
\soln{ex.ac.terminate}{
The worst-case situation is when the
interval to be represented lies just inside
a binary interval. In this case, we may choose either of
two binary intervals as shown in \figref{fig.ac.worst.case}.
These binary intervals are no
smaller than $P(\bx|\H)/4$, so the binary encoding has a length
no greater than $\log_2 1/ P(\bx|\H) + \log_2 4$, which is
two bits more than the ideal message length.
}
%
% HELP HELP HELP RESTORE ME!
% \input{tex/acvshuffman.tex}
%
%
\soln{ex.usebits}{
The standard method uses
32 random bits per generated
symbol and so requires $32\,000$ bits
to generate one thousand samples.
% this is displaced down a bit.
\begin{figure}%[htbp]
\figuremargin{%
\begin{center}
% created by ac.p only_show_data=1 > ac/ac_data.tex
\mbox{
\small
\setlength{\unitlength}{1.62in}
\begin{picture}(2,1.2)(0,0)
\thicklines
% desired interval on left
\put( 0.0, 1.01){\makebox(0,0)[bl]{Source string's interval}}
\put( 0.5, 0.5){\makebox(0,0){$P(\bx|\H)$}}
\put( 0.0, 0.05){\line(1,0){ 1.0}}
\put( 0.0, 0.95){\line(1,0){ 1.0}}
%
% binary intervals
\put( 1.0, 1.03){\makebox(0,0)[bl]{Binary intervals}}
\put( 1.0, 0.0){\line(1,0){ 1.0}}
\put( 1.0, 1.0){\line(1,0){ 1.0}}
%
\thinlines
%
\put( 0.5, 0.4){\vector(0,-1){0.35}}
\put( 0.5, 0.6){\vector(0,1){0.35}}
%
\put( 1.0, 0.5){\line(1,0){ 0.5}}
\put( 1.0, 0.25){\line(1,0){ 0.25}}
\put( 1.0, 0.75){\line(1,0){ 0.25}}
%
\put( 1.125, 0.625){\vector(0,1){0.125}}
\put( 1.125, 0.625){\vector(0,-1){0.125}}
\put( 1.125, 0.375){\vector(0,1){0.125}}
\put( 1.125, 0.375){\vector(0,-1){0.125}}
\end{picture}
}
\end{center}
}{%
\caption[a]{Termination of arithmetic coding in the worst case, where
there is a two bit overhead. Either of the two binary intervals marked on the
right-hand side may be chosen. These binary intervals are no
smaller than $P(\bx|\H)/4$.}
\label{fig.ac.worst.case}
}%
\end{figure}
Arithmetic coding uses on average
about $H_2 (0.01)=0.081$ bits per generated symbol, and so
requires about 83 bits to generate one thousand samples
(assuming an overhead of roughly two bits associated with termination).
Fluctuations in the number of {\tt{1}}s would produce variations
around this mean with standard deviation 21.
}
% 57
%\soln{ex.Clengthen}{
% moved to cutsolutions.tex
\soln{ex.LZencode}{
The encoding is {\tt010100110010110001100}, which comes from the
parsing
\beq
\tt 0, 00, 000, 0000, 001, 00000, 000000
\eeq
which is encoded thus:
\beq
{\tt (,0),(1,0),(10,0),(11,0),(010,1),(100,0),(110,0) } .
\eeq
}
\soln{ex.LZdecode}{
The decoding is
\begin{center}
{\tt 0100001000100010101000001}.
\end{center}
}
%\soln{ex.AC52}{
\soln{ex.AC52b}{
This problem is equivalent to \exerciseref{ex.AC52}.
The selection of $K$ objects from $N$ objects requires
$\lceil \log_2 {N \choose K}\rceil$ bits $\simeq N H_2(K/N)$ bits.
%
This selection could be made using arithmetic coding. The selection
corresponds to a binary string of length $N$ in which the {\tt{1}} bits represent
which objects are selected. Initially the probability of a {\tt{1}} is
$K/N$ and the probability of a {\tt{0}} is $(N\!-\!K)/N$. Thereafter, given that
the emitted string thus far, of length $n$, contains $k$ {\tt{1}}s,
the probability of a {\tt{1}} is
$(K\!-\!k)/(N\!-\!n)$ and the probability of a {\tt{0}} is $1 - (K\!-\!k)/(N\!-\!n)$.
}
\soln{ex.LZcomplete}{
This modified Lempel--Ziv code is still not `complete', because,
for example, after five prefixes have been collected,
the pointer could be any of the strings $\tt000$, $\tt001$, $\tt010$,
$\tt011$, $\tt100$, but
it cannot be $\tt101$, $\tt110$ or $\tt111$. Thus there are some binary strings
that cannot be produced as encodings.
}
\soln{ex.LZfail}{
Sources with low entropy that are not well compressed by Lempel--Ziv
include:\index{Lempel--Ziv coding!criticisms}
\ben
\item
Sources with some symbols that have
long range correlations and intervening
random junk. An ideal model should capture what's correlated
and compress it. Lempel--Ziv can only compress the correlated features
by memorizing all cases of the intervening junk.
As a simple example, consider a
\index{telephone number}\index{phone number}telephone book in which
every line contains an (old number, new number) pair:
\begin{center}
{\tt{285-3820:572-5892}}\teof\\
{\tt{258-8302:593-2010}}\teof\\
\end{center}
The number of characters per line is 18, drawn from the 13-character
alphabet
$\{ {\tt{0}},{\tt{1}},\ldots,{\tt{9}},{\tt{-}},{\tt{:}},\eof\}$.
The characters `{\tt{-}}',
`{\tt{:}}' and `\teof' occur in a predictable sequence, so
the true information content per line, assuming
all the phone numbers are seven digits long, and assuming
that they are random sequences,
is about 14 \dits. (A \dit\ is the information content
of a random integer between 0 and 9.)
A finite state language model could easily capture
the regularities in these data.
A Lempel--Ziv algorithm will take a long time before
it compresses such a file down to 14 bans per line,
% by a factor of $14/18$,
however, because in order for it to `learn' that
the string {\tt{:}}$ddd$ is always followed by {\tt{-}},
for any three digits $ddd$, it will have to {\em see\/}
all those strings. So near-optimal compression
will only be achieved after thousands of lines of the
file have been read.\medskip
% figs/wallpaper.ps made by pepper.p
\begin{figure}[htbp]
\fullwidthfigureright{%
%\figuremargin{%
\small
\begin{center}
\mbox{%(a)
\psfig{figure=figs/wallpaper.ps}}\\
%\mbox{(b) \psfig{figure=figs/wallpaperc.ps}}\\
%\mbox{(c) \psfig{figure=figs/wallpaperb.ps}}
\end{center}
}{%
\caption[a]{
A source with low entropy that is not well compressed by Lempel--Ziv.
The bit sequence is read from left to right.
Each line differs from the line above in $f=5$\% of its bits.
The image width is 400 pixels.
%
% Three
% sources with low entropy that are not well compressed by Lempel--Ziv.
% The bit sequence is read from left to right. The image width is 400 pixels
% in each case.
%
% (a) Each line differs from the line above in $p=$5\% of its bits.
%
% (b)
% Each column $c$ has its own transition probability $p_c$ such that
% successive vertical bits are identical with probability $p_c$. The
% probabilities $p_c$ are drawn from a uniform distribution over $[0,0.5]$.
%
% (c) As in b, but the probabilities $p_c$ are drawn from a uniform
% distribution over $[0,1]$.
}
% ; in columns with $p_c \simeq 1$, successive
% vertical bits are likely to be opposite to each other. }
%
\label{fig.pepper}
}%
\end{figure}
%
% this is beautiful but gratuitous
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%55
%\begin{figure}[htbp]
%\figuremargin{%
%\begin{center}
%\mbox{\psfig{figure=figs/automaton346.big1.ps,height=7in}}\\
%\end{center}
%}{%
%\caption[a]{A longer cellular automaton history.
%}
%\label{fig.automatonII}
%\end{figure}
\vspace*{-10pt}% included to undo the cumulation of item space and figure space.
\item
Sources with long range correlations, for example two-dimensional
images that are represented by a sequence of pixels, row by row,
so that vertically adjacent pixels are a distance $w$
apart in the source stream, where $w$ is the image width.
Consider, for example, a fax transmission in which each line
is very similar to the previous line (\figref{fig.pepper}).
The true entropy is only $H_2(f)$ per pixel, where $f$
is the probability that a pixel differs from its parent.
% except for a light peppering
% of noise.
% Each line is somewhat similar to the previous line but not identical,
% so there is no previous occurrence of a long string
% to point to; some algorithms in the Lempel--Ziv class
% will achieve a certain degree of compression
% by memorizing recent short strings, but the compression achieved
% will not equal the true entropy.
% and after a few lines,
% the pattern has moved on by a random walk, so memorizing ancient patterns
% is of no use.
Lempel--Ziv algorithms will only compress
down to the entropy once {\em all\/} strings of length $2^w = 2^{400}$
have occurred and their successors have been memorized.
There are only about $2^{300}$ particles in the universe, so we
can confidently say that
Lempel--Ziv codes will {\em never\/} capture the redundancy
of such an image.
% figs/wallpaper.ps made by pepper.p
\begin{figure}[htbp]
%\figuremargin{%
\fullwidthfigureright{%
\begin{center}
%\mbox{(a) \psfig{figure=figs/wallpaperx.ps}}\\
\mbox{%(b)
\psfig{figure=figs/wallpaperx2.ps}}\\
%\mbox{(c) \psfig{figure=figs/automaton346.2.ps}}\\
% see also figs/automaton346.big1.pbm
\end{center}
}{%
\caption[a]{%A second source with low entropy that is not optimally compressed by Lempel--Ziv.
A texture consisting of horizontal and vertical pins
dropped at random on the plane.
% (c) The 100-step time-history of a cellular automaton with 400 cells.
}
\label{fig.wallpaper}
}%
\end{figure}
Another highly redundant texture is shown in \figref{fig.wallpaper}.
The image was made
by dropping horizontal and vertical pins randomly on the plane.
It contains both long-range vertical correlations and long-range horizontal
correlations. There is no practical way that Lempel--Ziv, fed with a pixel-by-pixel scan
of this image, could capture both these correlations.
% gzip on the pbm gives: 2374 wallpaperx.pbm.gz
% That is better than 50%.
% Saved as a gif, wallpaperx.pbm is 2926 characters. Original 40000 pixels would be 5000 characters.
% That is worse than 50% compression.
% cf. perl program, stripwallpaper.p
% is
% 0 8 274 /home/mackay/bin/stripwallpaper.p.gz
% 0 16 631 wallpaperx.asc.gz
% 0 24 905 total <--------
% 18 65 368 /home/mackay/bin/stripwallpaper.p
% 162 484 1390 wallpaperx.asc
% 180 549 1758 total
% lossless jpg is terrible!:
% 38828 wallpaperx.jpg
% would be nice to try JBIG on this.
% It is worth emphasizing that b
Biological computational systems
can readily identify the redundancy in these images and in images
that are much more complex; thus we might anticipate that
the best data compression algorithms will result from the development
of \ind{artificial intelligence} methods.\index{compression!future methods}
\item
Sources with intricate redundancy, such as files generated
by computers. For example, a \LaTeX\ file
followed by its encoding into a PostScript file. The information content
of this pair of files is roughly equal to the information content of the
\LaTeX\ file alone.
\item
A picture of the Mandelbrot set. The picture has an information content
equal to the number of bits required to specify the range of the
complex plane studied, the pixel sizes,
and the colouring rule used.
% mapping of set membership to pixel colour.
% \item
% Encoded transmissions arising from an error-correcting code of rate $K/N$.
% These are very easily compressed by a factor
% $K/N$ if the generator operation is known.
% see README2 in /home/mackay/_courses/comput/newising_mc
\item
A picture of a ground state of
a frustrated antiferromagnetic \ind{Ising model} (\figref{fig.ising.ground}),
which we will discuss
in \chref{ch.ising}.
Like \figref{fig.wallpaper}, this binary image has interesting
correlations in two directions.
\begin{figure}[htbp]
\figuremargin{%
\begin{center}
\mbox{\bighisingsample{hexagon2}}
\end{center}
}{%
\caption[a]{Frustrated triangular
Ising model in one of its ground states.}
\label{fig.ising.ground}
}%
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item
Cellular automata -- \figref{fig.wallpaperc} shows
the state history of 100 steps of a \ind{cellular automaton}
with 400 cells. The update rule, in which each cell's new state depends on the state of five
preceding cells, was selected at random. The information content is equal to the information
in the boundary (400 bits), and the propagation rule, which here can be described in 32 bits.
An optimal compressor will thus give a compressed file length which
is essentially constant, independent of the vertical height of the image.
Lempel--Ziv would only give this zero-cost compression once the
cellular automaton has entered a periodic limit cycle, which
could easily take about $2^{100}$ iterations.
In contrast, the JBIG compression method, which models the probability of
a pixel given its local context and uses
arithmetic coding, would do a good job on these images.
%\item
% And finally, an example relating to error-correcting codes:
% the\index{error-correcting code!and compression}\index{difficulty of compression}\index{compression!difficulty of}
% received transmissions arising when encoded transmissions are
% sent over a noisy channel. Such received strings have an entropy
% equal to the source entropy plus the channel noise's
% entropy. If a \index{Lempel--Ziv coding|)}Lempel--Ziv
% algorithm could compress these strings,
% this would be tantamount to solving the decoding problem for
% the error-correcting code!
%
% We have not got to this topic yet, but we will see later that
% the decoding of a general error-correcting code is
% a challenging intractable problem.
% automaton.p
\begin{figure}%[htbp]
%\figuremargin{%
\fullwidthfigureright{%
\begin{center}
\mbox{%(c)
\psfig{figure=figs/automaton346.2.ps}}\\
% see also figs/automaton346.big1.pbm
\end{center}
}{%
\caption[a]{% Another source with low entropy that is not optimally compressed by Lempel--Ziv.
The 100-step time-history of a cellular automaton with 400 cells.
}
\label{fig.wallpaperc}
}%
\end{figure}
\een
}
\index{source code!stream codes|)}\index{stream codes|)}
\dvipsb{solutions stream codes}
%
%
%\section{Solutions}% to Chapter \protect\ref{ch_f4}'s exercises}
% \section{Solutions to section \protect\ref{ch_f4}'s exercises}
\fakesection{RNGaussian}
\soln{ex.RNGaussian}{
For a one-dimensional Gaussian, the
variance of $x$, $\Exp[x^2]$, is $\sigma^2$.
So the mean value of $r^2$ in $N$ dimensions,
since the components of $\bx$ are independent
random variables, is
\beq
\Exp[ r^2] = N \sigma^2 .
\eeq
The variance of $r^2$, similarly,
is $N$ times the variance of $x^2$,
where $x$ is a one-dimensional Gaussian
variable.
\beq
\var (x^2 ) = \int \! \d x \:
\frac{1}{(2 \pi \sigma^2)^{1/2}} x^4 \exp \left( - \frac{x^2}{2 \sigma^2} \right)
- \sigma^4 .
\eeq
The integral is found to be $3 \sigma^4$ (\eqref{eq.gaussian4thmoment}),
so $\var(x^2) = 2 \sigma^4$.
Thus the variance of $r^2$ is $2 N \sigma^4$.
For large $N$, the \ind{central-limit theorem}
% law of large numbers
indicates that
$r^2$ has a Gaussian distribution with mean $N \sigma^2$ and standard
deviation $\sqrt{2 N} \sigma^2$, so the probability density of $r$
must similarly be concentrated about $r \simeq \sqrt{N} \sigma$.
The thickness of this shell is given by turning the standard deviation of
$r^2$ into a standard deviation on $r$: for small
$\delta r/r$,
$\delta \log r = \delta r/r = (\dhalf) \delta \log r^2 = (\dhalf) \delta (r^2)/r^2$,
so setting $\delta (r^2) = \sqrt{2 N} \sigma^2$, $r$ has standard deviation
$\delta r = (\dhalf) r \delta (r^2)/r^2$
% $=$ $(\dhalf) \sqrt{2 N} \sigma^2 / \sqrt{( N \sigma^2)}$
$=\sigma/\sqrt{2}$.
The probability density of the Gaussian at a point $\bx_{\rm shell}$ where
$r = \sqrt{N} \sigma$ is
\beq
P(\bx_{\rm shell}) = \frac{1}{(2 \pi \sigma^2)^{N/2}}
\exp \left( - \frac{N \sigma^2}{2 \sigma^2} \right)
= \frac{1}{(2 \pi \sigma^2)^{N/2}}
\exp \left( - \frac{N}{2} \right) .
\eeq
Whereas the probability density at the origin is
\beq
P(\bx\eq 0) = \frac{1}{(2 \pi \sigma^2)^{N/2}} .
\eeq
Thus $P(\bx_{\rm shell})/P(\bx\eq 0) = \exp \left( - \linefrac{N}{2} \right) .$
The probability density at the typical radius is $e^{-N/2}$ times
smaller than the density at the origin. If $N=1000$, then the probability
density at the origin is $e^{500}$ times greater.
%
}
%
%
% for _e4.tex
%
\fakesection{Source coding problems solutions}
%\soln{ex.forward-backward-language}{
%% (Draft.)
%%
% If we write down a language model for strings in forward-English,
% the same model defines a probability distribution over strings
% of backward English. The probability distributions have
% identical entropy, so the average information contents
% of the reversed
% language and the forward language are equal.
%}
%\soln{ex.microwave}{
% moved to cutsolutions.tex
% removed to cutsolutions.tex
% \soln{ex.bridge}{(Draft.)
\dvipsb{solutions further data compression f4}
%\subchapter{Codes for integers \nonexaminable}
\chapter{Codes for Integers \nonexaminable}
\label{ch.codesforintegers}
This chapter is an aside, which may safely be skipped.
\section*{Solution to \protect\exerciseref{ex.Cinteger}}% was fiftythree
\label{sec.codes.for.integers}\label{ex.Cinteger.sol}% special by hand
%\soln{ex.Cinteger}{
%}
\fakesection{Cinteger Solutions to problems}
%
% original integer stuff is in old/s_integer.tex
%
% chapter 2 , coding of integers
To discuss the coding of integers\index{source code!for integers}
we need some definitions.\index{binary representations}
\begin{description}
\item[The standard binary representation of a positive
integer] $n$ will be denoted by $c_{\rm b}(n)$,
\eg, $c_{\rm b}(5) = {\tt 101}$, $c_{\rm b}(45) = {\tt 101101}$.
\item[The standard binary length of a positive
integer] $n$, $l_{\rm b}(n)$, is the length
of the string $c_{\rm b}(n)$.
For example, $l_{\rm b}(5) = 3$, $l_{\rm b}(45) = 6$.
\end{description}
The standard binary representation $c_{\rm b}(n)$
is {\em not\/} a uniquely decodeable code for integers
since there is no way of knowing when an integer has ended.
For example, $c_{\rm b}(5)c_{\rm b}(5)$ is identical to $c_{\rm b}(45)$.
It would be uniquely decodeable if we knew the
standard binary length of each integer
before it was received.
Noticing that all positive integers have a standard binary representation
that starts with a {\tt{1}}, we might define another representation:
\begin{description}
\item[The headless binary representation of a positive
integer] $n$ will be denoted by $c_{\rm B}(n)$,
\eg, $c_{\rm B}(5) = {\tt 01}$, $c_{\rm B}(45) = {\tt 01101}$
and $c_{\rm B}(1) = \lambda$ (where $\l$ denotes the null
string).
\end{description}
This representation would be uniquely decodeable if we knew the
length $l_{\rm b}(n)$ of the integer.
So, how can we make a uniquely decodeable code for integers?
Two strategies can be distinguished.
\ben
\item {\bf Self-delimiting codes}.
We first communicate somehow
% An alternative strategy is to make the code self-delimiting
\index{symbol code!self-delimiting}\index{self-delimiting}the length of the integer, $l_{\rm b}(n)$,
which is also a positive integer; then communicate the original
integer $n$ itself using $c_{\rm B}(n)$.
\item {\bf Codes with `end of file' characters}.
We code the integer into blocks of length
$b$ bits, and reserve one of the $2^b$ symbols to
have the special meaning `end of file'. The coding
of integers into blocks is arranged so that
this reserved symbol is not needed for any other purpose.
\een
The simplest uniquely decodeable code for integers is the unary code,
which can be viewed as a code with an end of file character.
\begin{description}
\item[Unary code\puncspace]
An integer $n$ is encoded by sending a string of $n\!-\!1$ {\tt 0}s
% zeroes
followed by a {\tt 1}.
\[
\begin{array}{cl} \toprule
n & c_{\rm U}(n) \\ \midrule
1 & {\tt 1} \\
2 & {\tt 01} \\
3 & {\tt 001} \\
4 & {\tt 0001} \\
5 & {\tt 00001} \\
\vdots & \\
45 & {\tt 000000000000000000000000000000000000000000001} \\ \bottomrule
\end{array}
\]
The unary code has length $l_{\rm U}(n) = n$.
The unary code is the optimal code for integers if the probability
distribution over $n$ is $p_{\rm U}(n) = 2^{-{n}}$.
\end{description}
\subsubsection*{Self-delimiting codes}
We can use the unary code to encode the {\em length\/} of the binary
encoding of $n$ and make a self-delimiting code:
\begin{description}
\item[Code $C_\alpha$\puncspace]
% The length of the standard binary representation is a positive integer
We send the unary code for $l_{\rm b}(n)$, followed
by the headless binary representation of $n$.
\beq
c_{\alpha}(n) = c_{\rm U}[ l_{\rm b}(n) ] c_{\rm B}(n) .
\eeq
Table \ref{tab.calpha} shows the codes for some integers. The overlining
indicates the division of each string into the parts $c_{\rm U}[ l_{\rm b}(n) ]$
and $c_{\rm B}(n)$.
\margintab{\footnotesize
\[
\begin{array}{clll} \toprule
n & c_{\rm b}(n) & \makebox[0in][c]{$l_{\rm b}(n)$} & c_{\alpha}(n)
% = c_{\rm U}[ l_{\rm b}(n) ] c_{\rm B}(n)
\\ \midrule
1 & {\tt 1 } & 1 & {\tt {\overline{1}}} \\
2 & {\tt 10 } & 2 & {\tt {\overline{01}}0} \\
3 & {\tt 11 } & 2 & {\tt {\overline{01}}1} \\
4 & {\tt 100} & 3 & {\tt {\overline{001}}00} \\
5 & {\tt 101} & 3 & {\tt {\overline{001}}01} \\
6 & {\tt 110} & 3 & {\tt {\overline{001}}10} \\
\vdots & \\
45 & {\tt 101101} & 6 & {\tt {\overline{000001}}01101} \\ \bottomrule
\end{array}
\]
\caption[a]{$C_\alpha$.}
\label{tab.calpha}
}
We might equivalently view $c_{\alpha}(n)$ as consisting of a string
of $(l_{\rm b}(n)-1)$ zeroes followed by the standard binary representation
of $n$, $c_{\rm b}(n)$.
The codeword $c_{\alpha}(n)$ has length $l_{\alpha}(n) = 2 l_{\rm b}(n) - 1$.
The implicit probability distribution over $n$ for the code
$C_{\alpha}$ is separable
into the product of a probability distribution over the length $l$,
\beq
P(l) = 2^{-l} ,
\eeq
and a uniform distribution over integers having that length,
\beq
P(n\given l) = \left\{ \begin{array}{cl} 2^{-l+1} & l_{\rm b}(n) = l \\
0 & \mbox{otherwise}.
\end{array} \right.
\eeq
\end{description}
Now, for the above code, the header that communicates
the length always occupies the same number
of bits as the standard binary representation of the integer (give or take
one). If we are expecting to encounter large integers (large files)
then this representation seems suboptimal, since it leads to
all files occupying a size that is double their original
uncoded size. Instead of using the unary
code to encode the length $l_{\rm b}(n)$, we could use $C_{\alpha}$.%
% see graveyard for original
\margintab{{\footnotesize
\[
\begin{array}{cll} \toprule
n & c_{\beta}(n) & c_{\gamma}(n)
\\ \midrule
1 & {\tt{\overline{1}}} & {\tt{\overline{1}}} \\
2 & {\tt{\overline{010}}0} & {\tt{\overline{0100}}0} \\
3 & {\tt{\overline{010}}1} & {\tt{\overline{0100}}1} \\
4 & {\tt{\overline{011}}00}& {\tt{\overline{0101}}00} \\
5 & {\tt{\overline{011}}01}& {\tt{\overline{0101}}01} \\
6 & {\tt{\overline{011}}10}& {\tt{\overline{0101}}10} \\
\vdots & \\
45 & {\tt{\overline{00110}}01101} & {\tt{\overline{01110}}01101} \\ \bottomrule
\end{array}
\]
}
\caption[a]{$C_\beta$ and $C_{\gamma}$.}
\label{tab.cbeta}
}
\begin{description}
\item[Code $C_\beta$\puncspace]
% The length of the standard binary representation is a positive integer
We send the length $l_{\rm b}(n)$ using $C_{\alpha}$, followed
by the headless binary representation of $n$.
\beq
c_{\beta}(n) = c_{\alpha}[ l_{\rm b}(n) ] c_{\rm B}(n) .
\eeq
\end{description}
Iterating this procedure, we can define a sequence of codes.
\begin{description}
\item[Code $C_{\gamma}$\puncspace]
\beq
c_{\gamma}(n) = c_{\beta}[ l_{\rm b}(n) ] c_{\rm B}(n) .
\eeq
% see graveyard for gamma table
\item[Code $C_\delta$\puncspace]
\beq
c_{\delta}(n) = c_{\gamma}[ l_{\rm b}(n) ] c_{\rm B}(n) .
\eeq
\end{description}
\subsection{Codes with end-of-file symbols}
We can also make byte-based representations.
(Let's use the term byte
flexibly here, to denote any fixed-length string of bits, not
just a string of length 8 bits.)
If we encode the number in some base, for example decimal, then
we can represent each digit in a byte. In order to represent
a digit from 0 to 9 in a byte we need four bits.
Because $2^4 = 16$, this leaves 6 extra four-bit symbols,
$\{${\tt 1010}, {\tt 1011}, {\tt 1100}, {\tt 1101}, {\tt 1110},
{\tt 1111}$\}$,
that correspond to no decimal digit. We can use these
as end-of-file symbols to indicate the end of our positive
integer.
% Such a code can also code the integer zero, for which
% we have not been providing a code up till now.
Clearly it is redundant to have more than one end-of-file
symbol, so a more efficient code would encode the integer
into base 15, and use just the sixteenth symbol, {\tt 1111},
as the punctuation character.
Generalizing this idea, we can make similar byte-based
codes for integers in bases 3 and 7, and in any base of
the form $2^n-1$.
\margintab{\small
\[
\begin{array}{cll} \toprule
n & c_3(n) & c_{7}(n)
% = c_{\rm U}[ l_{\rm b}(n) ] c_{\rm B}(n)
\\ \midrule
1 & {\tt 01\, 11 } & {\tt 001\, 111} \\
2 & {\tt 10\, 11 } & {\tt 010\, 111} \\
3 & {\tt 01\, 00\, 11 } & {\tt 011\, 111} \\
\vdots & \\
45 & {\tt 01\, 10\, 00\, 00\, 11} & {\tt 110\, 011\, 111} \\ \bottomrule
\end{array}
\]
\caption[a]{Two codes with end-of-file symbols,
$C_3$ and $C_7$. Spaces have been included to show the
byte boundaries.
}
}
These codes are almost complete. (Recall that
a code is `complete' if it satisfies the
Kraft inequality with equality.) The codes'
remaining inefficiency is that they provide the
ability to encode the integer zero and the empty string,
neither of which was required.
\exercissxB{2}{ex.intEOF}{
Consider the implicit probability distribution over integers
corresponding to the code with an end-of-file character.
\ben
\item
If the code has eight-bit blocks (\ie, the integer is
coded in base 255), what is the mean length in bits
of the integer, under the implicit distribution?
\item
If one wishes to encode binary files of expected size about one hundred
\kilobytes\ using a code with an end-of-file character, what is the optimal
block size?
\een
}
\subsection*{Encoding a tiny file}
% see claude.p in itp/tex
To illustrate the codes we have discussed, we now use each
code to encode
a small file consisting of just 14 characters,
\[
\framebox{\tt{Claude Shannon}}.
\]
\bit
\item
If we map the ASCII characters onto seven-bit
symbols (\eg, in decimal,
${\tt C}=67$, ${\tt l}=108$, etc.), this 14 character file corresponds to the
integer
\[
n = 167\,987\,786\,364\,950\,891\,085\,602\,469\,870 \:\:\mbox{(decimal)}.
\]
\item
The unary code for $n$ consists of this many (less one) zeroes,
followed by a one. If all the oceans were turned into ink, and if we
wrote a hundred bits with every cubic millimeter,
% or microlitre
there
% would be roughly
might be enough ink to write $c_{\rm U}(n)$.
\item
The standard binary representation of $n$ is this length-98 sequence of bits:
\beqa
c_{\rm b}(n) &=& \begin{array}[t]{l}
\tt 1000011110110011000011110101110010011001010100000 \\
\tt 1010011110100011000011101110110111011011111101110.
\end{array}
\eeqa
% To store this self-delimiting file
% on a disc, we would need
\eit
\exercisaxB{2}{ex.claudeshannonn}{
Write down or describe the following
self-delimiting representations of the above number $n$:
$c_{\alpha}(n)$,
$c_{\beta}(n)$,
$c_{\gamma}(n)$,
$c_{\delta}(n)$,
$c_{3}(n)$,
$c_{7}(n)$, and
$c_{15}(n)$.
Which of these encodings is the shortest? [{\sf{Answer:}} $c_{15}$.]
}
%
% solution moved to cutsolutions.tex
%
\subsection{Comparing the codes}
One could answer the question `which of two codes is
superior?' by a sentence of the form `For $n>k$, code 1 is
superior, for $n
Secondly, the depiction in terms of Venn diagrams
encourages one to believe that all the areas correspond to
positive quantities. In the special case of two random variables
it is indeed true that $H(X \given Y)$, $\I(X;Y)$ and $H(Y \given X)$ are positive
quantities. But as soon as we progress to three-variable ensembles,
we obtain a diagram with positive-looking areas that
may actually correspond to negative quantities. \Figref{fig.venn3}
correctly shows relationships such as
\beq
H(X) + H(Z \given X) + H(Y \given X,Z) = H(X,Y,Z) .
\eeq
But it gives the misleading impression that
the conditional mutual information $\I(X;Y \given Z)$ is
{\em less than\/} the mutual information
$\I(X;Y)$.
\begin{figure}
\figuremargin{%3/4
\begin{center}
\mbox{\psfig{figure=figs/venn3.ps,angle=-90,width=5.25in}}
\end{center}
}{%
\caption[a]{A misleading representation of entropies, continued.}
\label{fig.venn3}
}%
\end{figure}
In fact the area labelled $A$ can correspond to a {\em negative\/}
quantity. Consider the joint ensemble
$(X,Y,Z)$ in which $x \in \{0,1\}$ and $y \in \{0,1\}$
are independent binary variables and $z \in \{0,1\}$ is defined
to be $z=x+y \mod 2$.
Then clearly $H(X) = H(Y) = 1$ bit. Also $H(Z) = 1$ bit.
And $H(Y \given X) = H(Y) = 1$ since the two variables are independent.
So the mutual information between $X$ and $Y$ is zero.
$\I(X;Y) = 0$. However, if $z$ is observed, $X$ and $Y$ become dependent ---
% correlated ---
knowing $x$, given $z$, tells you what $y$ is: $y = z - x \mod 2$.
So $\I(X;Y \given Z) = 1$ bit. Thus the area labelled $A$ must correspond
to $-1$ bits for the figure to give the correct answers.
The above example is not at all a capricious or exceptional
illustration.
The binary symmetric channel with input $X$, noise $Y$, and output $Z$
% The classic\index{earthquake and burglar alarm}\index{burglar alarm and earthquake}
% earthquake-burglar-alarm ensemble \exercisebref{ex.burglar}\
%% (section ???),
% with
% earthquake $= X$,
% burglar $ = Y$ and alarm $= Z$,
% is a perfect example of a
is a
situation in which $\I(X;Y)=0$ (input and noise are independent)
% uncorrelated
but $\I(X;Y \given Z) > 0$ (once you see the output, the unknown input and the unknown noise
are intimately related!).
The Venn diagram representation is therefore valid only if one is aware
that positive areas may represent negative quantities.
With this proviso
% As long as this possibility is
kept in mind, the
interpretation of entropies in terms
of sets can be helpful \cite{Yeung1991}.
% The quantity corresponding to $A$ is denoted $I(X;Y;Z)$
% by \citeasnoun{Yeung1991}.
}
\soln{ex.dataprocineq}{% BORDERLINE
%{\bf New answer:}
For any joint ensemble $XYZ$, the following chain rule
for mutual information holds.
\beq
\I(X;Y,Z) = \I(X;Y) + \I(X;Z \given Y) .
\eeq
Now, in the case $w \rightarrow d \rightarrow r$,
$w$ and $r$ are independent given $d$, so
$\I(W;R \given D) = 0$. Using the chain rule twice, we have:
\beq
\I(W;D,R) = \I(W;D)
\eeq
and
\beq
\I(W;D,R) = \I(W;R) + \I(W;D \given R) ,
\eeq
so
\beq
\I(W;R) - \I(W;D) \leq 0 .
\eeq
% for more solutions to this problem see
% Igraveyard.tex
}
\prechapter{About Chapter}
\fakesection{prerequisites for chapter 5}
Before reading \chref{ch.five}, you should have read \chapterref{ch.one}
and worked
on
\exerciseref{ex.rel.ent}, and
\exerciserefrange{ex.Hcondnal}{ex.zxymod2}.
% \exfifteen--\exeighteen,
% \extwenty--\extwentyone, and \extwentythree.
% uvw to HXY>0
% {ex.Hmutualineq}{ex.joint},
% \exerciserefrangeshort{ex.rel.ent}
% load of H() and I() stuff shoved in here now.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\ENDprechapter
\chapter{Communication over a Noisy Channel}
\label{ch.five}
% % l5.tex
%
% useful program: bin/capacity.p for checking channel
% capacities
%
% % \part{Noisy Channel Coding}
% \chapter{Communication over a noisy channel}
% % The noisy-channel coding theorem, part a}
% % \chapter{The noisy channel coding theorem, part a}
\label{ch5}
\section{The big picture}
%
\setlength{\unitlength}{1mm}
\begin{realcenter}
%\begin{floatingfigure}[l]{3.2in}
\begin{picture}(85,50)(-40,5)
\thinlines
\put(0,5){\framebox(25,10){\begin{tabular}{c}Noisy\\ channel\end{tabular}}}
\put(-20,20){\framebox(25,10){\begin{tabular}{c}Encoder\end{tabular}}}
\put(20,20){\framebox(25,10){\begin{tabular}{c}Decoder\end{tabular}}}
\put(-20,40){\framebox(25,10){\begin{tabular}{c}Compressor\end{tabular}}}
\put(20,40){\framebox(25,10){\begin{tabular}{c}Decompressor\end{tabular}}}
\put(-40,40){\makebox(15,10){\begin{tabular}{c}{\sc Source}\\{\sc coding}\end{tabular}}}
\put(-40,20){\makebox(15,10){\begin{tabular}{c}{\sc Channel}\\{\sc coding}\end{tabular}}}
\put(-20,55){\makebox(25,10){Source}}
%
\put(-7.5,18){\line(0,-1){8}}
\put(-7.5,10){\vector(1,0){6}}
\put(32.5,10){\vector(0,1){8}}
\put(32.5,10){\line(-1,0){6}}
%
\put(32.5,31){\vector(0,1){8}}
\put(32.5,51){\vector(0,1){6}}
\put(-7.5,39){\vector(0,-1){8}}
\put(-7.5,57){\vector(0,-1){6}}
\end{picture}
\end{realcenter}
%
In\index{channel!noisy} Chapters \ref{ch2}--\ref{ch4},
we discussed source coding with block
codes, symbol codes and stream codes. We implicitly assumed that
the channel from the compressor to the decompressor
was noise-free. Real channels are noisy. We will now spend two
chapters on the subject of noisy-channel coding -- the fundamental
possibilities and limitations of error-free \ind{communication} through a
noisy channel. The aim of channel coding
is to make the noisy channel behave like a noiseless channel.
We will assume that the data to be transmitted
has been through a good compressor, so the bit stream has no
obvious redundancy. The channel code, which makes the transmission,
will put\index{redundancy!in channel code}
back
% into the transmission
redundancy of a special sort, designed
to make the noisy received signal decodeable.\index{decoder}
Suppose we transmit 1000 bits per second\index{channel!binary symmetric}
with $p_0 = p_1 = \dhalf$
over a noisy channel that flips bits with probability
$f = 0.1$. What is the rate of
transmission of information?
% shannon p.35
We might guess that the rate is 900 bits per second by subtracting
the expected number of errors per second. But this is not correct, because
the recipient does not know where the errors occurred.
Consider the case where the noise is so great that
the received symbols are independent of the
transmitted symbols. This corresponds to a noise level of $f=0.5$,
since half of the received symbols are correct due to chance alone.
But when $f=0.5$, no information is transmitted at all.
% ? cut this clearly?
\label{sec.ch5.intro}
% refer to exercise {ex.zxymod2}.
Given what we have learnt about entropy, it seems reasonable that
a measure of the information transmitted is given by the \ind{mutual
information} between the source and the received signal, that is, the
entropy of the source minus the \ind{conditional entropy}
of the source given the received signal.
%
% shannon calls the conditional entropy the equivocation
% and points out that the equivocation is the amount of extra
% information needed for a correcting device to figure out
% what is going on
We will now review the definition of conditional entropy
and mutual information. Then we will examine
% progress to the question of
whether it is possible to use such a noisy channel to communicate
{\em reliably}.
We will
% Our aim here is to
show that for any channel $Q$ there is a non-zero rate,
the \inds{capacity}\index{channel!capacity}
$C(Q)$, up to which information can be sent with arbitrarily
small probability of error.
\section{Review of probability and information}
% conditional, joint and mutual information}
% We now build on
As an example, we take the joint distribution $XY$ from
\extwentyone.
%
% A useful picture breaks down the total information content $H(X,Y)$
% of a joint ensemble thus:
% \begin{center}
% \setlength{\unitlength}{1in}
% \begin{picture}(3,1.13)(0,-0.2)
% \put(0,0.7){\framebox(3,0.20){$H(X,Y)$}}
% \put(0,0.4){\framebox(2.2,0.20){$H(X)$}}
% \put(1.5,0.1){\framebox(1.5,0.20){$H(Y)$}}
% \put(1.5125,-0.2){\framebox(0.675,0.20){$\I(X;Y)$}}
% \put(0,-0.2){\framebox(1.475,0.20){$H(X \given Y)$}}
% \put(2.225,-0.2){\framebox(0.775,0.20){$H(Y \specialgiven X)$}}
% \end{picture}
% \end{center}
%
% \subsection{Example of a joint ensemble}
% A joint ensemble $XY$ has the following joint distribution.
The
marginal distributions $P(x)$ and $P(y)$ are shown in the
margins.
% $P(x,y)$:
\[
\begin{array}{cc|cccc|c}
\multicolumn{2}{c}{P(x,y)} & \multicolumn{4}{|c|}{x} & P(y) \\[0.051in]
& & 1 & 2 & 3 & 4 & \\[0.011in]
\hline
\strutf
&1 & \dfrac{1}{8} & \dfrac{1}{16} & \dfrac{1}{32} & \dfrac{1}{32} & \dfrac{1}{4} \\[0.01in]
\raisebox{0mm}{\mbox{$y$}}
&2 & \dfrac{1}{16} & \dfrac{1}{8} & \dfrac{1}{32} & \dfrac{1}{32} & \dfrac{1}{4} \\[0.01in]
&3 & \dfrac{1}{16} & \dfrac{1}{16} & \dfrac{1}{16} & \dfrac{1}{16} & \dfrac{1}{4} \\[0.01in]
&4 & \dfrac{1}{4} & 0 & 0 & 0 & \dfrac{1}{4} \\[0.01in]
\hline
\multicolumn{2}{c|}{P(x)}
& \strutf\dfrac{1}{2} & \dfrac{1}{4} & \dfrac{1}{8} & \dfrac{1}{8} & \\[0.051in]
\end{array}
\]
The joint entropy is $H(X,Y)=27/8$ bits.
The marginal entropies are $H(X) = 7/4$ bits and $H(Y) = 2$ bits.
We can compute the conditional distribution of $x$ for each value of $y$,
and the entropy of each of those conditional distributions:
\[
\begin{array}{cc|cccc|c}
\multicolumn{2}{c|}{P(x \given y)} & \multicolumn{4}{c|}{x} & H(X \given y) / \mbox{bits} \\[0.051in]
& & 1 & 2 & 3 & 4 & \\[0.011in]
\hline
\strutf
&1 & \dfrac{1}{2} & \dfrac{1}{4} & \dfrac{1}{8} & \dfrac{1}{8} & \dfrac{7}{4} \\[0.01in]
\raisebox{0mm}{\mbox{$y$}}
&2 & \dfrac{1}{4} & \dfrac{1}{2} & \dfrac{1}{8} & \dfrac{1}{8} & \dfrac{7}{4} \\[0.01in]
&3 & \dfrac{1}{4} & \dfrac{1}{4} & \dfrac{1}{4} & \dfrac{1}{4} & 2 \\[0.01in]
&4 & 1 & 0 & 0 & 0 & 0 \\[0.01in]
\hline
\multicolumn{3}{c}{\strutf
} & \multicolumn{4}{r}{H(X \given Y) = \dfrac{11}{8}} \\[0.1in]
\end{array}
\]
Note that whereas $H(X \given y\eq 4) = 0$ is less than $H(X)$, $H(X \given y\eq 3)$ is greater
than $H(X)$.
% _s5A.tex has a solution link already \label{ex.Hcondnal.sol}
% \label{ex.joint.sol}
So in some cases, learning $y$ can
% make us more uncertain
{\em increase\/} our uncertainty
about $x$. Note also that although $P(x \given y\eq 2)$
is a different distribution from $P(x)$, the conditional entropy $H(X \given y\eq 2)$
is equal to $H(X)$. So learning that $y$ is 2 changes our knowledge
about $x$ but does not reduce the uncertainty
of $x$, as measured by the entropy. On average though,
learning $y$ does convey information
about $x$, since $H(X \given Y) < H(X)$.
One may also evaluate $H(Y \specialgiven X) = 13/8$ bits.
The mutual information is
$\I(X;Y) = H(X) - H(X \given Y) = 3/8$ bits.
% INCLUDE ME LATER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% INCLUDE ME LATER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% INCLUDE ME LATER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% \subsection{Solutions to a few other exercises}
% \input{tex/entropy_soln.tex}
%
% INCLUDE ME LATER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% INCLUDE ME LATER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% INCLUDE ME LATER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%\mynewpage MNBV
\section{Noisy channels}
\begin{description}
\item[A discrete memoryless channel $Q$] is\index{channel!discrete memoryless}
characterized by
an input alphabet $\A_X$, an output alphabet $\A_Y$,
and a set of conditional probability distributions $P(y \given x)$, one
for each $x \in \A_X$.
These {\dbf{transition probabilities}} may be written in a matrix\index{transition probability matrix}
\beq
Q_{j|i} = P(y\eq b_j \given x\eq a_i) .
\eeq
\begin{aside}
I\index{notation!conventions of this book}\index{notation!matrices}\index{notation!vectors}\index{conventions!matrices}\index{conventions!vectors}\index{notation!transition probability}
usually orient this matrix with the output variable
$j$ indexing the rows and the input variable $i$
indexing the columns, so that each column of $\bQ$ is a probability
vector. With this convention, we can obtain the probability
of the output, $\bp_Y$, from a probability distribution over the input,
$\bp_X$, by right-multiplication:
\beq
\bp_Y = \bQ \bp_X .
\eeq
%
\end{aside}
%
\end{description}
\noindent
Some useful model channels are:
\begin{description}
% bsc
\item[Binary symmetric channel\puncspace]
\indexs{channel!binary symmetric}\indexs{binary symmetric channel}
$\A_X \eq \{{\tt 0},{\tt 1}\}$. $\A_Y \eq \{{\tt 0},{\tt 1}\}$.
\[
\begin{array}{c}
\setlength{\unitlength}{0.46mm}
\begin{picture}(30,20)(-5,0)
\put(-4,9){{\makebox(0,0)[r]{$x$}}}
\put(5,2){\vector(1,0){10}}
\put(5,16){\vector(1,0){10}}
\put(5,4){\vector(1,1){10}}
\put(5,14){\vector(1,-1){10}}
\put(4,2){\makebox(0,0)[r]{1}}
\put(4,16){\makebox(0,0)[r]{0}}
\put(16,2){\makebox(0,0)[l]{1}}
\put(16,16){\makebox(0,0)[l]{0}}
\put(24,9){{\makebox(0,0)[l]{$y$}}}
\end{picture}
\end{array}
\hspace{1in}
\begin{array}{ccl}
P(y\eq {\tt 0} \given x\eq {\tt 0}) &=& 1 - \q ; \\ P(y\eq {\tt 1} \given x\eq {\tt 0}) &=& \q ;
\end{array}
\begin{array}{ccl}
P(y\eq {\tt 0} \given x\eq {\tt 1}) &=& \q ; \\ P(y\eq {\tt 1} \given x\eq {\tt 1}) &=& 1 - \q .
\end{array}
\hspace{1in}
\begin{array}{c}
\ecfig{bsc15.1}
\end{array}
\]
%
% \BEC bec BEC
%
\item[Binary erasure channel\puncspace] \indexs{channel!binary erasure}\indexs{binary erasure channel}
$\A_X \eq \{{\tt 0},{\tt 1}\}$. $\A_Y \eq \{{\tt 0},\mbox{\tt ?},{\tt 1}\}$.
\[
\begin{array}{c}
\setlength{\unitlength}{0.46mm}
\begin{picture}(30,30)(-5,0)
\put(-4,15){{\makebox(0,0)[r]{$x$}}}
\put(5,5){\vector(1,0){10}}
\put(5,25){\vector(1,0){10}}
\put(5,5){\vector(1,1){10}}
\put(5,25){\vector(1,-1){10}}
\put(4,5){\makebox(0,0)[r]{\tt 1}}
\put(4,25){\makebox(0,0)[r]{\tt 0}}
\put(16,5){\makebox(0,0)[l]{\tt 1}}
\put(16,25){\makebox(0,0)[l]{\tt 0}}
\put(16,15){\makebox(0,0)[l]{\tt ?}}
\put(24,15){{\makebox(0,0)[l]{$y$}}}
\end{picture}
\end{array}
\hspace{1in}
\begin{array}{ccl}
P(y\eq {\tt 0} \given x\eq {\tt 0}) &=& 1 - \q ; \\
P(y\eq \mbox{\tt ?} \given x\eq {\tt 0}) &=& \q ; \\
P(y\eq {\tt 1} \given x\eq {\tt 0}) &=& 0 ;
\end{array}
\begin{array}{ccl}
P(y\eq {\tt 0} \given x\eq {\tt 1}) &=& 0 ; \\
P(y\eq \mbox{\tt ?} \given x\eq {\tt 1}) &=& \q ; \\
P(y\eq {\tt 1} \given x\eq {\tt 1}) &=& 1 - \q .
\end{array}
\hspace{1in}
\begin{array}{c}
\ecfig{bec.1}
\end{array}
\]
\item[Noisy typewriter\puncspace]
\indexs{channel!noisy typewriter}\indexs{noisy typewriter}
$\A_X = \A_Y = \mbox{the 27 letters $\{${\tt A},
{\tt B}, \ldots, {\tt Z}, {\tt -}$\}$}$.
The letters are arranged in a circle, and
when the typist attempts to type {\tt B}, what comes out is
either {\tt A}, {\tt B} or {\tt C}, with probability \dfrac{1}{3} each;
when the input is {\tt C}, the output is {\tt B}, {\tt C} or {\tt D};
and so forth, with the final letter `{\tt -}'
% being
adjacent to the
first letter {\tt A}.
\[
\begin{array}{c}
\setlength{\unitlength}{1pt}
\begin{picture}(48,130)(0,2)
\thinlines
\put(5,5){\vector(3,0){30}}
\put(5,25){\vector(3,0){30}}
\put(5,15){\vector(3,0){30}}
\put(5,5){\vector(3,1){30}}
\put(5,25){\vector(3,-1){30}}
\put(4,5){\makebox(0,0)[r]{{\tt -}}}
\put(4,15){\makebox(0,0)[r]{{\tt Z}}}
\put(4,25){\makebox(0,0)[r]{{\tt Y}}}
\put(36,5){\makebox(0,0)[l]{{\tt -}}}
\put(36,15){\makebox(0,0)[l]{{\tt Z}}}
\put(36,25){\makebox(0,0)[l]{{\tt Y}}}
%
\put(5,15){\vector(3,1){30}}
\put(5,15){\vector(3,-1){30}}
\put(5,25){\vector(3,0){30}}
\put(5,25){\vector(3,1){30}}
\put(20,43){\makebox(0,0){$\vdots$}}
%
%\put(5,35){\vector(3,0){30}}
%\put(5,35){\vector(3,1){30}}
\put(5,35){\vector(3,-1){30}}
%\put(5,45){\vector(3,0){30}}
\put(5,45){\vector(3,1){30}}
%\put(5,45){\vector(3,-1){30}}
\put(5,55){\vector(3,0){30}}
\put(5,55){\vector(3,1){30}}
\put(5,55){\vector(3,-1){30}}
\thicklines
\put(5,65){\vector(3,0){30}}
\put(5,65){\vector(3,1){30}}
\put(5,65){\vector(3,-1){30}}
\thinlines
\put(5,75){\vector(3,0){30}}
\put(5,75){\vector(3,1){30}}
\put(5,75){\vector(3,-1){30}}
\put(5,85){\vector(3,0){30}}
\put(5,85){\vector(3,1){30}}
\put(5,85){\vector(3,-1){30}}
\put(5,95){\vector(3,0){30}}
\put(5,95){\vector(3,1){30}}
\put(5,95){\vector(3,-1){30}}
\put(5,105){\vector(3,0){30}}
\put(5,105){\vector(3,1){30}}
\put(5,105){\vector(3,-1){30}}
\put(5,115){\vector(3,0){30}}
\put(5,115){\vector(3,1){30}}
\put(5,115){\vector(3,-1){30}}
\put(5,125){\vector(3,0){30}}
\put(5,125){\vector(3,-1){30}}
\put(5,5){\vector(1,4){30}}
\put(5,125){\vector(1,-4){30}}
%\put(4,35){\makebox(0,0)[r]{{\tt J}}}
%\put(36,35){\makebox(0,0)[l]{{\tt J}}}
%\put(4,45){\makebox(0,0)[r]{{\tt I}}}
%\put(36,45){\makebox(0,0)[l]{{\tt I}}}
\put(4,55){\makebox(0,0)[r]{{\tt H}}}
\put(36,55){\makebox(0,0)[l]{{\tt H}}}
\put(4,65){\makebox(0,0)[r]{{\tt G}}}
\put(36,65){\makebox(0,0)[l]{{\tt G}}}
\put(4,75){\makebox(0,0)[r]{{\tt F}}}
\put(36,75){\makebox(0,0)[l]{{\tt F}}}
\put(4,85){\makebox(0,0)[r]{{\tt E}}}
\put(36,85){\makebox(0,0)[l]{{\tt E}}}
\put(4,95){\makebox(0,0)[r]{{\tt D}}}
\put(36,95){\makebox(0,0)[l]{{\tt D}}}
\put(4,105){\makebox(0,0)[r]{{\tt C}}}
\put(36,105){\makebox(0,0)[l]{{\tt C}}}
\put(4,115){\makebox(0,0)[r]{{\tt B}}}
\put(36,115){\makebox(0,0)[l]{{\tt B}}}
\put(4,125){\makebox(0,0)[r]{{\tt A}}}
\put(36,125){\makebox(0,0)[l]{{\tt A}}}
\end{picture}
\end{array}
\hspace{1in}
\begin{array}{ccl} & \vdots & \\
P(y\eq {\tt F} \given x\eq {\tt G}) &=& 1/3 ; \\
P(y\eq {\tt G} \given x\eq {\tt G}) &=& 1/3 ; \\
P(y\eq {\tt H} \given x\eq {\tt G}) &=& 1/3 ; \\
& \vdots &
\end{array}
\hspace{1.2in}
\begin{array}{c}
\ecfig{type}
\end{array}
\]
\item[Z channel\puncspace]
\indexs{channel!Z channel}\indexs{Z channel}
$\A_X \eq \{{\tt 0},{\tt 1}\}$. $\A_Y \eq \{{\tt 0},{\tt 1}\}$.
\[
% \begin{array}{c}
% \setlength{\unitlength}{0.46mm}
% \begin{picture}(20,20)(0,0)
% \put(5,5){\vector(1,0){10}}
% \put(5,15){\vector(1,0){10}}
% \put(5,5){\vector(1,1){10}}
% \put(4,5){\makebox(0,0)[r]{1}}
% \put(4,15){\makebox(0,0)[r]{0}}
% \put(16,5){\makebox(0,0)[l]{1}}
% \put(16,15){\makebox(0,0)[l]{0}}
% \end{picture}
% \end{array}
\begin{array}{c}
\setlength{\unitlength}{0.46mm}
\begin{picture}(30,20)(-5,0)
\put(-4,9){{\makebox(0,0)[r]{$x$}}}
\put(5,2){\vector(1,0){10}}
\put(5,16){\vector(1,0){10}}
\put(5,4){\vector(1,1){10}}
% \put(5,14){\vector(1,-1){10}}
\put(4,2){\makebox(0,0)[r]{1}}
\put(4,16){\makebox(0,0)[r]{0}}
\put(16,2){\makebox(0,0)[l]{1}}
\put(16,16){\makebox(0,0)[l]{0}}
\put(24,9){{\makebox(0,0)[l]{$y$}}}
\end{picture}
\end{array}
\hspace{1in}
\begin{array}{ccl}
P(y\eq {\tt 0} \given x\eq {\tt 0}) &=& 1 ; \\
P(y\eq {\tt 1} \given x\eq {\tt 0}) &=& 0 ; \\
\end{array}
\begin{array}{ccl}
P(y\eq {\tt 0} \given x\eq {\tt 1}) &=& \q ; \\
P(y\eq {\tt 1} \given x\eq {\tt 1}) &=& 1- \q .\\
\end{array}
\hspace{1in}
%\:\:\:\:\:\:
\begin{array}{c}
\ecfig{z15.1}
\end{array}
\]
% {\em Check if this orientation of the channel disagrees
% with any demonstrations.}
\end{description}
\section{Inferring the input given the output}
% was a subsection
% a single transmission}
If we assume that the input $x$ to a channel
comes from an ensemble $X$, then
we obtain a joint ensemble $XY$ in which the random variables $x$ and $y$
have the joint distribution:
\beq
P(x,y) = P(y \given x) P(x) .
\eeq
Now if we receive
a particular symbol $y$, what was the input symbol $x$?
We typically won't know for certain. We can write down the posterior
distribution of the input using \Bayes\ theorem:\index{Bayes' theorem}
\beq
P(x \given y) = \frac{ P(y \given x) P(x) }{P(y) }
= \frac{ P(y \given x) P(x) }{\sum_{x'} P(y \given x') P(x') } .
\eeq
\exampla{
%{\sf Example 1:}
Consider a \index{channel!binary symmetric}\ind{binary symmetric channel}
with probability of
error $\q\eq 0.15$. Let the input ensemble be $\P_X: \{p_0 \eq 0.9, p_1 \eq 0.1\}$.
Assume we observe $y\eq 1$.
\beqan
P(x\eq 1 \given y\eq 1) &=&\frac{ P(y\eq 1 \given x\eq 1) P(x\eq 1) }{\sum_{x'} P(y \given x') P(x') } \nonumber \\
&\eq & \frac{ 0.85 \times 0.1 }{ 0.85 \times 0.1 + 0.15 \times 0.9 } \nonumber \\
&=& \frac{ 0.085 }{ 0.22 } \:\:=\:\: 0.39 .
\eeqan
Thus `$x\eq 1$' is still less probable than `$x\eq 0$', although it is not
as improbable as it was before.
}
% Could turn this into an exercise.
% Alternatively, assume we observe $y\eq 0$.
% \beqa
% P(x\eq 1 \given y\eq 0) &=& \frac{ P(y\eq 0 \given x\eq 1) P(x\eq 1)}{\sum_{x'} P(y \given x') P(x')} \\
% &=& \frac{ 0.15 \times 0.1 }{ 0.15 \times 0.1 + 0.85 \times 0.9 } \\
% &=& \frac{ 0.015 }{0.78} = 0.019 .
% \eeqa
\exercissxA{1}{ex.bscy0}{
Now assume we observe $y\eq 0$.
Compute the probability of $x\eq 1$ given $y\eq 0$.
}
\exampla{
%{\sf Example 2:}
Consider a \ind{Z channel}\index{channel!Z channel} with probability of
error $\q\eq 0.15$. Let the input ensemble be $\P_X: \{p_0 \eq 0.9, p_1 \eq 0.1\}$.
Assume we observe $y\eq 1$.
\beqan
P(x\eq 1 \given y\eq 1)
&=& \frac{ 0.85 \times 0.1 }{ 0.85 \times 0.1 + 0 \times 0.9 }
\nonumber \\
&=& \frac{ 0.085}{0.085} \:\:=\:\: 1.0 .
\eeqan
So given the output $y\eq 1$ we become certain of the input.
}
% Alternatively, assume we observe $y\eq 0$.
% \beqa
% P(x\eq 1 \given y\eq 0)
% % &=& \frac{ P(y\eq 0 \given x\eq 1) P(x\eq 1)}{\sum_{x'} P(y \given x') P(x')} \\
% &=& \frac{ 0.15 \times 0.1 }{ 0.15 \times 0.1 + 1.0 \times 0.9 } \\
% &=& \frac{ 0.015}{ 0.915} = 0.016 .
% \eeqa
\exercissxA{1}{ex.zcy0}{
Alternatively, assume we observe $y\eq 0$. Compute $P(x\eq 1 \given y\eq 0)$.
}
\section{Information conveyed by a channel}
We now consider how much information can be communicated through
a channel. In {operational\/} terms, we are interested in finding
ways of using the channel such that all the bits that are communicated
are recovered with negligible probability of error.
In {mathematical\/} terms,
assuming a particular input ensemble $X$, we can measure how
much information the output conveys about the input by the mutual
information:
\beq
\I(X;Y) \equiv H(X) - H(X \given Y) = H(Y) - H(Y \specialgiven X) .
\eeq
Our aim is to establish the connection between these two ideas.
Let us evaluate $\I(X;Y)$ for some of the channels above.
\subsection{Hint for computing mutual information}
\index{hint for computing mutual information}\index{mutual information!how to compute}We
will tend to think of $\I(X;Y)$ as $H(X) - H(X \given Y)$, \ie, how much
the uncertainty of the input $X$ is reduced when we look at the output
$Y$. But for computational
purposes it is often handy to evaluate $H(Y) - H(Y \specialgiven X)$ instead.
%\medskip
% this reproduced from _p5A.tex, figure 9.1 {fig.entropy.breakdown}
\begin{figure}[htbp]
\figuremargin{%
\begin{center}
%
% included by l1.tex
%
\setlength{\unitlength}{1in}
\begin{picture}(3,1.13)(0,-0.2)
\put(0,0.7){\framebox(3,0.20){$H(X,Y)$}}
\put(0,0.4){\framebox(2.2,0.20){$H(X)$}}
\put(1.5,0.1){\framebox(1.5,0.20){$H(Y)$}}
\put(1.5125,-0.2){\framebox(0.675,0.20){$\I(X;Y)$}}
\put(0,-0.2){\framebox(1.475,0.20){$H(X\,|\,Y)$}}
\put(2.225,-0.2){\framebox(0.775,0.20){$H(Y|X)$}}
\end{picture}
\end{center}
}{%
\caption[a]{The relationship between joint information,
marginal entropy, conditional entropy and mutual entropy.
This figure is important, so I'm showing it twice.}
\label{fig.entropy.breakdown.again}
}%
\end{figure}
%\begin{center}
%\input{tex/entropyfig.tex}
%\end{center}
%\noindent
\exampla{
%{\sf Example 1:}
Consider the
\index{channel!binary symmetric}\index{binary symmetric channel}\BSC\
again, with $\q\eq 0.15$ and
$\P_X: \{p_0 \eq 0.9, p_1 \eq 0.1\}$. We already evaluated the
marginal probabilities $P(y)$ implicitly above: $P(y\eq 0) = 0.78$;
$P(y\eq 1) = 0.22$. The mutual information is:
\beqa
\I(X;Y) &=& H(Y) - H(Y \specialgiven X) .
\eeqa
What is $H(Y \specialgiven X)$?
It is defined to be the weighted sum over $x$ of $H(Y \given x)$; but
$H(Y \given x)$ is the same for each value of $x$:
$H(Y \given x\eq{\tt{0}})$ is $H_2(0.15)$,
and $H(Y \given x\eq{\tt{1}})$ is $H_2(0.15)$. So
\beqan
\I(X;Y) &=& H(Y) - H(Y \specialgiven X) \nonumber \\
&=& H_2(0.22) - H_2(0.15) \nonumber \\
& =& 0.76 - 0.61 \:\: = \:\: 0.15 \mbox{ bits}.
\eeqan
% this used to be in error (0.15)
This may be contrasted with the entropy of the source $H(X) = H_2(0.1) =
0.47$ bits.
Note: here we have used the binary entropy function $H_2(p)
\equiv H(p,1\!-\!p)=p \log \frac{1}{p}
+ (1-p)\log \frac{1}{(1-p)}$.\marginpar{\small\raggedright{Throughout this book, $\log$ means $\log_2$.}}
}
%\medskip
% \noindent
\exampla{
% {\sf Example~2:}
And now the \ind{Z channel}\index{channel!Z channel}, with $\P_X$ as above.
% $P(y\eq 0)\eq 0.915;
$P(y\eq 1)\eq 0.085$.
\beqan
\I(X;Y) &=& H(Y) - H(Y \specialgiven X) \nonumber \\
&=& H_2(0.085) - [ 0.9 H_2(0) + 0.1 H_2(0.15) ] \nonumber \\
&=& 0.42 - ( 0.1 \times 0.61 )
= 0.36 \mbox{ bits}.
\eeqan
The entropy of the source, as above, is $H(X) = 0.47$ bits. Notice
that the mutual information $\I(X;Y)$ for the Z channel is bigger than
the mutual information for the binary symmetric channel with the
same $\q$. The Z channel is a more reliable
channel.
% is fits with our intuition that the
}
\exercissxA{1}{ex.bscMI}{Compute the mutual information between $X$ and $Y$
for the \BSC\ with $\q\eq 0.15$ when the input
distribution is $\P_X = \{p_0 \eq 0.5, p_1 \eq 0.5\}$.
}
\exercissxA{2}{ex.zcMI}{Compute the mutual information between $X$ and $Y$
for the Z channel with $\q=0.15$ when the input
distribution is $\P_X: \{p_0 \eq 0.5, p_1 \eq 0.5\}$.
}
\subsection{Maximizing the mutual information}
We have observed in the above examples that
the mutual information between the input and
the output depends on the chosen
{input ensemble}\index{channel!input ensemble}.
Let us assume that we wish to maximize the mutual information
conveyed by the channel by choosing
the best possible input ensemble.
We define the {\dbf\inds{capacity}\/} of the
channel\index{channel!capacity}
to be its maximum \ind{mutual information}.
\begin{description}
\item[The capacity] of a channel $Q$ is:
\beq
C(Q) = \max_{\P_X} \, \I(X;Y) .
\eeq
The distribution $\P_X$ that achieves the maximum is called the
{\dem{\optens}},\indexs{optimal input distribution}
denoted by $\P_X^*$. [There may be multiple
{\optens}s achieving the same value of $\I(X;Y)$.]
\end{description}
%
In \chref{ch6} we will
show that the capacity does indeed measure the maximum amount
of error-free information that can be transmitted
% is transmittable % yes, spell checked
over the channel per unit time.
% \medskip
% Sun 22/8/04 am having problems trying to get fig 9.2 to go at head
% of p 151 - putting it there causes text to move.
%\noindent
\exampla{
%{\sf Example 1:}
Consider the \BSC\ with $\q \eq 0.15$. Above, we considered
$\P_X = \{p_0 \eq 0.9, p_1 \eq 0.1\}$, and found
$\I(X;Y) = 0.15$ bits.
% the page likes to break here
How much better can we do? By symmetry,
the \optens\ is
$\{ 0.5, 0.5\}$ and%
\amarginfig{t}{
\mbox{%
%\begin{figure}[htbp]
\small
%\floatingmargin{%
%\figuremargin{%
\raisebox{0.91in}{$\I(X;Y)$}%
\hspace{-0.42in}%
\begin{tabular}{c}
\mbox{\psfig{figure=figs/IXY.15.ps,%
width=45mm,angle=-90}}\\[-0.1in]
$p_1$
\end{tabular}
}
%}{%
\caption[a]{The mutual information $\I(X;Y)$ for a binary symmetric
channel with $\q=0.15$
as a function of the input distribution.
% (\eqref{eq.IXYBSC}).
}
\label{fig.IXYBSC}
}
%%%
the capacity is
\beq
C(Q_{\rm BSC}) \:=\: H_2(0.5) - H_2(0.15) \:=\: 1.0 - 0.61 \:=\: 0.39
\ubits.
\eeq
We'll justify the \ind{symmetry argument}\index{capacity!symmetry argument}
later.
If there's any doubt about the
% such a
symmetry argument,
we can always resort to explicit maximization of
the \ind{mutual information} $I(X;Y)$,
\beq
I(X;Y) = H_2( (1\!-\!\q)p_1 + (1\!-\!p_1)\q ) - H_2(\q) \ \ \mbox{ (\figref{fig.IXYBSC}). }
\label{eq.IXYBSC}
\eeq
}
% \medskip
% \noindent
% {\sf Example 2:}
\exampl{exa.typewriter}{
The noisy typewriter.
The \optens\ is a uniform distribution over $x$, and gives
$C = \log_2 9$ bits.
}
% \medskip
% \noindent
\exampl{exa.Z.HXY}{
% {\sf Example 3:}
Consider the \ind{Z channel} with $\q \eq 0.15$.
Identifying the \optens\ is not so straightforward. We
evaluate $\I(X;Y)$ explicitly for
$\P_X = \{p_0, p_1\}$. First, we need to compute $P(y)$. The probability
of $y\eq 1$ is easiest to write down:
\beq
P(y\eq 1) \:\:=\:\: p_1 (1-\q) .
\eeq
Then%
\amarginfig{t}{
%\begin{figure}[htbp]
\mbox{%
\small
%\floatingmargin{%
%\figuremargin{%
\raisebox{0.91in}{$\I(X;Y)$}%
\hspace{-0.42in}%
\begin{tabular}{c}
\mbox{\psfig{figure=figs/HXY.ps,%
width=45mm,angle=-90}}\\[-0.1in]
$p_1$
\end{tabular}
}
%}{%
\caption{The mutual information $\I(X;Y)$ for a Z
channel with $\q=0.15$
as a function of the input distribution.}
\label{hxyz}
}
%\end{figure}
%%%%%%%%%%%%% old:
%\begin{figure}[htbp]
%\small
%\begin{center}
%\raisebox{1.3in}{$\I(X;Y)$}%
%\hspace{-0.2in}%
%\begin{tabular}{c}
%\mbox{\psfig{figure=figs/HXY.ps,%
%width=60mm,angle=-90}}\\
%$p_1$
%\end{tabular}
%\end{center}
%\caption[a]{The mutual information $\I(X;Y)$ for a Z channel with $\q=0.15$
% as a function of the input distribution.}
%% (Horizontal axis $=p_1$.)}
%\label{hxyz.old}
%\end{figure}
the mutual information is:
\beqan
\I(X;Y) &=& H(Y) - H(Y \specialgiven X) \nonumber \\
&=& H_2(p_1 (1-\q)) - ( p_0 H_2(0) + p_1 H_2(\q) ) \nonumber \\
&=& H_2(p_1 (1-\q)) - p_1 H_2(\q) .
\eeqan
This is a non-trivial function of $p_1$, shown in \figref{hxyz}.
It is maximized for $\q=0.15$ by
% the \optens\
$p_1^* = 0.445$.
We find $C(Q_{\rm Z}) = 0.685$. Notice
% that
the \optens\ is not
$\{ 0.5,0.5 \}$. We can communicate slightly more information
by using input symbol {\tt{0}} more frequently than {\tt{1}}.
}
%\noindent {\sf Exercise b:}
\exercissxA{1}{ex.bscC}{
What is the capacity of the \ind{binary symmetric channel} for general $\q$?\index{channel!binary symmetric}
}
\exercissxA{2}{ex.becC}{
Show that the capacity of the \ind{binary erasure channel}\index{channel!binary erasure} with $\q=0.15$
is $C_{\rm BEC} = 0.85$. What is its capacity for general $\q$?
Comment.
}
% \bibliography{/home/mackay/bibs/bibs}
%\section{The Noisy Channel Coding Theorem}
\section{The noisy-channel coding theorem}
It seems plausible that the `capacity' we have defined may be
a measure of information conveyed by a channel; what is not obvious,
and what we will prove in the next chapter, is that the \ind{capacity} indeed
measures the rate at which blocks of data can be communicated over the channel
{\em with arbitrarily small probability of error}.
We make the following definitions.\label{sec.whereCWMdefined}
\begin{description}
\item[An $(N,K)$ {block code}] for\indexs{error-correcting code!block code}
a channel $Q$ is a list of $\cwM=2^K$
codewords
$$\{ \bx^{(1)}, \bx^{(2)}, \ldots, \bx^{({2^K)}} \}, \:\:\:\:\:\bx^{(\cwm)} \in \A_X^N ,$$
each of length $N$.
Using this code we can encode a signal $\cwm \in \{ 1,2,3,\ldots, 2^K\}$
% The signal to be encoded is assumed to come from an
% alphabet of size $2^K$; signal $m$ is encoded
as $\bx^{(\cwm)}$. [The number of codewords $\cwM$ is an integer,
but the number of bits specified by choosing a codeword, $K \equiv \log_2 \cwM$,
is not necessarily an integer.]
The {\dbf \inds{rate}\/} of\index{error-correcting code!rate}
the code is $R = K/N$ bits per channel use.
% character.
[We will use this definition of the rate for any channel, not only channels with binary inputs;
note however that it is sometimes conventional to define the rate of a code for a channel
with $q$ input symbols to be $K/(N\log q)$.]
% \item[A linear $(N,K)$ block code] is a block code in which all
% moved into leftovers.tex
\item[A \ind{decoder}] for an $(N,K)$ block code is a mapping from
the set of length-$N$ strings of channel outputs, $\A_Y^N$, to
a codeword label $\hat{\cwm} \in \{ 0 , 1 , 2 , \ldots, 2^K \}$.
The extra symbol $\hat{\cwm} \eq 0$ can be used to indicate a `failure'.
\item[The \ind{probability of block error}\index{error probability!block}]
% $p_B$
of a code and decoder, for a given channel, and for a given probability
distribution over the encoded signal $P(\cwm_{\rm in})$,
is:
\beq
p_{\rm B} = \sum_{\cwm_{\rm in}} P( \cwm_{\rm in} )
P( \cwm_{\rm out} \! \not = \! \cwm_{\rm in} \given \cwm_{\rm in} )
.
\eeq
% the probability
% that the decoded signal $\cwm_{\rm out}$ is not equal to $\cwm_{\rm in}$.
\item[The maximal probability of block error] is
\beq
p_{\rm BM} = \max_{\cwm_{\rm in}} P( \cwm_{\rm out} \! \not = \!
\cwm_{\rm in} \given \cwm_{\rm in} )
.
\eeq
\item[The \ind{optimal decoder}] for a channel code is the one that minimizes
the probability of block error. It decodes an output $\by$ as
the input $\cwm$ that has maximum \ind{posterior probability} $P(\cwm \given \by)$.
\beq
P(\cwm \given \by) =
\frac{ P(\by \given \cwm ) P(\cwm) } { \sum_{\cwm' } P(\by \given \cwm') P(\cwm') }
\eeq
\beq
\hat{\cwm}_{\rm optimal} = \argmax
% _{\cwm} % did not appear underneath
P(\cwm \given \by) .
\eeq
A uniform prior distribution on $\cwm$ is usually assumed, in which case the
optimal decoder is also the {\dem \ind{maximum likelihood decoder}},
\ie, the decoder that maps an output $\by$ to
the input $\cwm$ that has maximum {\dem \ind{likelihood}} $P(\by \given \cwm )$.
\item[The probability of bit error] $p_{\rm b}$ is defined assuming that
the codeword number
$\cwm$ is represented by a binary vector $\bs$ of length $K$ bits;
it is the average probability
that a bit of $\bs_{\rm out}$ is not equal to the corresponding
bit of $\bs_{\rm in}$ (averaging over all $K$ bits).
\item[Shannon's\index{Shannon, Claude}
\ind{noisy-channel coding theorem} (part one)\puncspace]
%\begin{quote}
Associated with each discrete memoryless channel,
\marginfig{
\begin{center}
\setlength{\unitlength}{2pt}
\begin{picture}(60,45)(-2.5,-7)
\thinlines
\put(0,0){\vector(1,0){60}}
\put(0,0){\vector(0,1){40}}
\put(30,-3){\makebox(0,0)[t]{$C$}}
\put(55,-2){\makebox(0,0)[t]{$R$}}
\put(-1,35){\makebox(0,0)[r]{$p_{\rm BM}$}}
\thicklines
\put(0,0){\dashbox{3}(30,30){achievable}}
% \put(0,0){\line(0,1){50}}
%
\end{picture}
\end{center}
\caption[a]{Portion of the $R,p_{\rm BM}$ plane asserted to
be
achievable by the first part of Shannon's noisy
channel coding theorem.}
\label{fig.belowCthm}
}%end marginfig
there is a
non-negative number $C$ (called the channel capacity) with the following
property. For any $\epsilon > 0$ and $R < C$, for large enough $N$,
there exists a block code of length $N$ and rate $\geq R$ and a decoding
algorithm, such that the maximal probability of block error is
$< \epsilon$.
%\end{quote}
% \item[The negative part of the theorem\puncspace] moved to graveyard.tex Sun 3/2/02
\end{description}
\begin{figure}[htbp]
\figuremargin{%
\[
\begin{array}{c}
\setlength{\unitlength}{1pt}
\begin{picture}(48,120)(0,5)
\thinlines
%\put(5,5){\vector(3,0){30}}
%\put(5,25){\vector(3,0){30}}
\put(5,15){\vector(3,0){30}}
%\put(5,5){\vector(3,1){30}}
%\put(5,25){\vector(3,-1){30}}
% \put(4,5){\makebox(0,0)[r]{{\tt -}}}
\put(4,15){\makebox(0,0)[r]{{\tt Z}}}
% \put(4,25){\makebox(0,0)[r]{{\tt Y}}}
\put(36,5){\makebox(0,0)[l]{{\tt -}}}
\put(36,15){\makebox(0,0)[l]{{\tt Z}}}
\put(36,25){\makebox(0,0)[l]{{\tt Y}}}
%
\put(5,15){\vector(3,1){30}}
\put(5,15){\vector(3,-1){30}}
%\put(5,25){\vector(3,0){30}}
%\put(5,25){\vector(3,1){30}}
\put(20,40){\makebox(0,0){$\vdots$}}
%
%\put(5,35){\vector(3,0){30}}
%\put(5,35){\vector(3,1){30}}
% \put(5,35){\vector(3,-1){30}}
%\put(5,45){\vector(3,0){30}}
% \put(5,45){\vector(3,1){30}}
%\put(5,45){\vector(3,-1){30}}
\put(5,55){\vector(3,0){30}}
\put(5,55){\vector(3,1){30}}
\put(5,55){\vector(3,-1){30}}
% \thicklines
% \put(5,65){\vector(3,0){30}}
% \put(5,65){\vector(3,1){30}}
% \put(5,65){\vector(3,-1){30}}
% \thinlines
% \put(5,75){\vector(3,0){30}}
% \put(5,75){\vector(3,1){30}}
% \put(5,75){\vector(3,-1){30}}
\put(5,85){\vector(3,0){30}}
\put(5,85){\vector(3,1){30}}
\put(5,85){\vector(3,-1){30}}
% \put(5,95){\vector(3,0){30}}
% \put(5,95){\vector(3,1){30}}
% \put(5,95){\vector(3,-1){30}}
%\put(5,105){\vector(3,0){30}}
%\put(5,105){\vector(3,1){30}}
%\put(5,105){\vector(3,-1){30}}
\put(5,115){\vector(3,0){30}}
\put(5,115){\vector(3,1){30}}
\put(5,115){\vector(3,-1){30}}
%\put(5,125){\vector(3,0){30}}
%\put(5,125){\vector(3,-1){30}}
%
%\put(5,5){\vector(1,4){30}}
%\put(5,125){\vector(1,-4){30}}
\put(36,45){\makebox(0,0)[l]{{\tt I}}}
\put(4,55){\makebox(0,0)[r]{{\tt H}}}
\put(36,55){\makebox(0,0)[l]{{\tt H}}}
% \put(4,65){\makebox(0,0)[r]{{\tt G}}}
\put(36,65){\makebox(0,0)[l]{{\tt G}}}
% \put(4,75){\makebox(0,0)[r]{{\tt F}}}
\put(36,75){\makebox(0,0)[l]{{\tt F}}}
\put(4,85){\makebox(0,0)[r]{{\tt E}}}
\put(36,85){\makebox(0,0)[l]{{\tt E}}}
% \put(4,95){\makebox(0,0)[r]{{\tt D}}}
\put(36,95){\makebox(0,0)[l]{{\tt D}}}
% \put(4,105){\makebox(0,0)[r]{{\tt C}}}
\put(36,105){\makebox(0,0)[l]{{\tt C}}}
\put(4,115){\makebox(0,0)[r]{{\tt B}}}
\put(36,115){\makebox(0,0)[l]{{\tt B}}}
% \put(4,125){\makebox(0,0)[r]{{\tt A}}}
\put(36,125){\makebox(0,0)[l]{{\tt A}}}
\end{picture}
\end{array}
\hspace{1.5in}
\begin{array}{c}
% roughly 8pts from col to col
\setlength{\unitlength}{1.005pt}% this was 1pt in jan 2000, I tweaked it
\begin{picture}(50,110)(-5,-5)
\thinlines
\put(-5,-5){\ecfig{type}}
\multiput(7.95,-3)(12,0){9}{\framebox(4,126){}}
%\put(2.5,97){\makebox(0,0)[bl]{\small$\bx^{(1)}$}}
%\put(26.5,97){\makebox(0,0)[bl]{\small$\bx^{(2)}$}}
%
\end{picture}
\end{array}
\]
}{%
\caption[a]{A non-confusable subset of inputs for the noisy
typewriter.}
\label{fig.typenine}
}
\end{figure}
\subsection{Confirmation of the theorem for the noisy typewriter channel}
In the case of the \ind{noisy typewriter}\index{channel!noisy typewriter},
we can easily confirm the
% positive part of the
theorem,
% For this channel,
because we can create a
% n {\em error-free\/}
completely error-free
communication strategy using a block code of length $N =1$:
we use only the letters {\tt B}, {\tt E}, {\tt H},
\ldots, {\tt Z},
\ie, every third letter. These letters form a {\dem non-confusable subset\/}\index{non-confusable inputs}
of the input
alphabet (see \figref{fig.typenine}). Any output can be uniquely decoded. The number of
inputs in the non-confusable subset is 9, so the error-free information
rate of this system is $\log_2 9$ bits, which is equal to the capacity $C$,
which we evaluated in \exampleref{exa.typewriter}.
%
How does this translate into the terms of the theorem?
The following table explains.\medskip
%\begin{center}
\begin{raggedright}
\noindent
% THIS TABLE IS DELIBERATELY FULL WIDTH
% for textwidth, use this
% \begin{tabular}{p{2.2in}p{2.5in}}
\begin{tabular}{@{}p{2.7in}p{4.1in}@{}}
\multicolumn{1}{@{}l}{\sf The theorem} &
\multicolumn{1}{l}{\sf How it applies to the noisy typewriter } \\ \midrule
\raggedright\em Associated with each discrete memoryless channel, there is a
non-negative number $C$.
% (called the channel capacity).
&
The capacity $C$ is $\log_2 9$.
\\[0.047in]
\raggedright\em For any $\epsilon > 0$ and $R < C$, for large enough $N$,
&
% Assume we are given an $R0$.
No matter what $\epsilon$ and $R$ are, we set the blocklength $N$ to
1.
\\[0.047in]
\raggedright\em there exists a block code of length $N$ and rate $\geq R$
& The block code is
% can be the following list of nine codewords:
$\{{\tt B,E,\ldots,Z}\}$. The value of
$K$ is given by $2^K = 9$, so $K=\log_2 9$, and this code has rate
$\log_2 9$, which is greater than the requested value of $R$.
\\[0.047in]
\raggedright\em and a decoding
algorithm,
&
The decoding algorithm maps the received
letter to the nearest letter in the code;
\\[0.047in]
\raggedright\em
such that the maximal probability of block error is
$< \epsilon$.
&
the maximal probability of block error is zero, which
is
less than the given $\epsilon$.
\\
\end{tabular}
\end{raggedright}
%\end{center}
% is greater than or equal
% to 1
% source RUNME
\section{Intuitive preview of proof}
\subsection{Extended channels}
To prove the theorem for any given channel, we consider the
{\dem \ind{extended channel}\index{channel!extended}}
corresponding to $N$ uses of the
% original
channel.
The extended channel has
$|\A_X|^N$ possible inputs $\bx$ and
$|\A_Y|^N$ possible outputs.
% {\em add a picture of extended channel here.}
%
\begin{figure}
\figuremargin{%
\small\begin{center}
\begin{tabular}{cccc}
%$\bQ$
& \ecfig{bsc15.1}
& \ecfig{bsc15.2}
& \ecfig{bsc15.4}
\\
& $N=1$
& $N=2$ & $N=4$ \\
\end{tabular}
\end{center}
}{%
\caption{Extended channels obtained from a binary symmetric channel
with transition probability 0.15.}
\label{fig.extended.bsc15}
}
\end{figure}
%
\begin{figure}
\figuremargin{%
\small\begin{center}
\begin{tabular}{cccc}
%$\bQ$
& \ecfig{z15.1}
& \ecfig{z15.2}
& \ecfig{z15.4}
\\
& $N=1$
& $N=2$ & $N=4$ \\
\end{tabular}
\end{center}
}{%
\caption{Extended channels obtained from a Z channel
with transition probability 0.15. Each column corresponds to an input,
and each row is a different output.}
\label{fig.extended.z15}
}
\end{figure}
%
%
% these figures made using
% cd itp/extended
Extended channels obtained from a \BSC\ and from
a Z channel are shown in figures \ref{fig.extended.bsc15}
and \ref{fig.extended.z15}, with $N=2$ and $N=4$.
\exercissxA{2}{ex.extended}{
Find the transition probability matrices $\bQ$ for
the extended channel, with $N=2$, derived from
the binary erasure channel having erasure probability 0.15.
%\item the extended channel with $N=2$ derived from
% the ternary confusion channel,
By selecting two columns of this transition probability matrix,
% that have minimal overlap,
we can define a rate-\dhalf\ code for this channel with blocklength $N=2$.
What is the best choice of two columns? What is the decoding
algorithm?
}
To prove the noisy-channel coding theorem, we
make use of large blocklengths $N$.
The intuitive idea is that, if $N$ is large, {\em
an extended channel looks a lot like the noisy typewriter.}
Any particular input $\bx$ is very likely to produce an output
in a small subspace of the output alphabet -- the typical output
set, given that input.
So we can find a non-confusable subset of the inputs that produce
essentially disjoint output sequences.
%
% add something like:
% Remember what we learnt
% in chapter \ref{ch2}:
%
For a given $N$, let us consider a way of generating such a
non-confusable subset of the inputs, and count up how many distinct
inputs it contains.
Imagine making an input sequence $\bx$ for the extended channel by
drawing it from an ensemble $X^N$, where $X$ is an arbitrary ensemble
over the input alphabet. Recall the source coding theorem of
\chapterref{ch.two}, and consider the number of probable output sequences
$\by$. The total number of typical output sequences $\by$
% , when $\bx$ comes from the ensemble $X^N$,
is $2^{N H(Y)}$, all having similar
probability. For any particular typical input sequence $\bx$, there
are about $2^{N H(Y \specialgiven X)}$ probable sequences. Some of these subsets of
$\A_Y^N$ are depicted by circles in figure \ref{fig.ncct.typs}a.
\begin{figure}%[htbp]
\small
\figuremargin{%
\begin{center}
\hspace*{-1mm}\begin{tabular}{cc}
\framebox{
\setlength{\unitlength}{0.69mm}%was 0.8mm
\begin{picture}(80,80)(0,0)
\put(0,80){\makebox(0,0)[tl]{$\A_Y^N$}}
\thicklines
\put(40,40){\oval(50,50)}
\thinlines
\put(40,67){\makebox(0,0)[b]{Typical $\by$}}
\put(30,50){\circle{12.5}}
\put(50,40){\circle{12.5}}
\put(35,52){\circle{12.5}}
\put(58,33){\circle{12.5}}
\put(33,40){\circle{12.5}}
\put(35,45){\circle{12.5}}
\put(50,30){\circle{12.5}}
\put(40,50){\circle{12.5}}
\put(52,35){\circle{12.5}}
\put(33,58){\circle{12.5}}
\put(40,33){\circle{12.5}}
\put(45,35){\circle{12.5}}
\put(50,50){\circle{12.5}}
\put(23,55){\circle{12.5}}
\put(24,45){\circle{12.5}}
\put(27,57){\circle{12.5}}
\put(25,40){\circle{12.5}}
\put(55,42){\circle{12.5}}
\put(55,52){\circle{12.5}}
\put(58,53){\circle{12.5}}
\put(53,40){\circle{12.5}}
\put(35,22){\circle{12.5}}
\put(27,30){\circle{12.5}}
\put(40,24){\circle{12.5}}
\put(40,39){\circle{12.5}}
\put(46,43){\circle{12.5}}
\put(55,40){\circle{12.5}}
\put(40,55){\circle{12.5}}
\put(52,23){\circle{12.5}}
\put(50,26){\circle{12.5}}
\put(40,54){\circle{12.5}}
\put(52,55){\circle{12.5}}
\put(33,28){\circle{12.5}}
\put(57,33){\circle{12.5}}
\put(25,35){\circle{12.5}}
\put(55,25){\circle{12.5}}
\put(25,26){\circle{12.5}}
\multiput(23,22)(13,0){3}{\circle{12.5}}
\multiput(30,34)(13,0){3}{\circle{12.5}}
\multiput(23,46)(13,0){3}{\circle{12.5}}
\multiput(30,58)(13,0){3}{\circle{12.5}}
\thicklines
\put(23,30){\circle{12.5}}
\put(21,11){\vector(0,1){13}}
\put(8,6){\makebox(0,0)[l]{ Typical $\by$ for a given typical $\bx$}}
\end{picture}
}
&
\framebox{
\setlength{\unitlength}{0.69mm}
\begin{picture}(80,80)(0,0)
\put(0,80){\makebox(0,0)[tl]{$\A_Y^N$}}
\thicklines
\put(40,40){\oval(50,50)}
\thinlines
\put(40,67){\makebox(0,0)[b]{Typical $\by$}}
% \thicklines
\multiput(23,22)(13,0){3}{\circle{12.5}}
\multiput(30,34)(13,0){3}{\circle{12.5}}
\multiput(23,46)(13,0){3}{\circle{12.5}}
\multiput(30,58)(13,0){3}{\circle{12.5}}
%\put(30,34){\circle{12.5}}
%\put(43,34){\circle{12.5}}
%\put(56,34){\circle{12.5}}
%\put(23,45){\circle{12.5}}
%\put(36,45){\circle{12.5}}
%\put(49,45){\circle{12.5}}
%\put(30,56){\circle{12.5}}
%\put(43,56){\circle{12.5}}
%\put(56,56){\circle{12.5}}
\end{picture}
}\\
(a)&(b) \\
\end{tabular}
\end{center}
}{%
\caption[a]{(a) Some typical outputs
in $\A_Y^N$ corresponding
to typical inputs $\bx$.
(b) A subset of the \ind{typical set}s shown in
(a) that do not overlap each other. This picture can be
compared with the
solution to the \ind{noisy typewriter} in \figref{fig.typenine}.}
\label{fig.ncct.typs}
\label{fig.ncct.typs.no.overlap}
}
\end{figure}
We now imagine restricting ourselves to a subset of the typical\index{typical set!for noisy channel}
inputs $\bx$ such that the corresponding typical output sets do not overlap,
as shown in \figref{fig.ncct.typs.no.overlap}b.
We can then bound the number of non-confusable inputs by dividing the size
of the typical $\by$ set,
$2^{N H(Y)}$, by the size of each typical-$\by$-given-typical-$\bx$
set, $2^{N H(Y \specialgiven X)}$. So the number of non-confusable inputs,
if they are selected from the set of typical inputs $\bx \sim X^N$,
is $\leq 2^{N H(Y) - N H(Y \specialgiven X)} = 2^{N \I(X;Y)}$.
% \begin{figure}
% \begin{center}
% \framebox{
% \setlength{\unitlength}{0.8mm}
% }
% \end{center}
% \caption[a]{A subset of the typical sets shown in
% \protect\figref{fig.ncct.typs} that do not overlap.}
% \label{fig.ncct.typs.no.overlap}
% \end{figure}
The maximum value of
this bound is achieved if $X$ is the ensemble that
maximizes $\I(X;Y)$, in which case the number of non-confusable inputs
is $\leq 2^{NC}$. Thus asymptotically
up to $C$ bits per cycle,
and no more, can be communicated with vanishing error probability.\ENDproof
This sketch has not rigorously proved that reliable communication really
is possible --
that's our task for the next chapter.
\section{Further exercises}
% \noindent
%
\exercissxA{3}{ex.zcdiscuss}{
Refer back to the computation of the capacity of the
\ind{Z channel} with $\q=0.15$.
\ben
\item
Why is $p_1^*$ less than 0.5? One could argue that it is good
to favour the {\tt{0}} input, since it is transmitted without error --
and also argue that it is good to favour the {\tt1} input, since it
often gives rise to the highly prized {\tt1} output, which
allows certain identification of the input! Try to make a convincing
argument.
\item
In the case of general $\q$, show that the \optens\ is
\beq
p_1^* = \frac{ 1/(1-\q) }
{ \displaystyle
1 + 2^{ \left( H_2(\q) / ( 1 - \q ) \right)} } .
\eeq
\item
What happens to $p_1^*$ if the noise level $\q$ is very close to 1?
\een
}
% see also ahmed.tex for a nice bound 0.5(1-q) on the capacity of the Z channel
% and related graphs CZ.ps CZ2.ps CZ.gnu
%
\exercissxA{2}{ex.Csketch}{
Sketch graphs of the capacity of the \ind{Z channel}, the \BSC\
and the \BEC\ as a function of $\q$.
% answer in figs/C.ps
% \medskip
}
\exercisaxB{2}{ex.fiveC}{
What is the capacity of the five-input, ten-output channel
% \index{channel!others}
whose
transition probability matrix is
{\small
\beq
\left[ \begin{array}{*{5}{c}}
0.25 & 0 & 0 & 0 & 0.25 \\
0.25 & 0 & 0 & 0 & 0.25 \\
0.25 & 0.25 & 0 & 0 & 0 \\
0.25 & 0.25 & 0 & 0 & 0 \\
0 & 0.25 & 0.25 & 0 & 0 \\
0 & 0.25 & 0.25 & 0 & 0 \\
0 & 0 & 0.25 & 0.25 & 0 \\
0 & 0 & 0.25 & 0.25 & 0 \\
0 & 0 & 0 & 0.25 & 0.25 \\
0 & 0 & 0 & 0.25 & 0.25 \\
\end{array}
\right]
\hspace{0.4in}
\begin{array}{c}\ecfig{five}\end{array}
?
\eeq
}
}
\exercissxA{2}{ex.GC}{
Consider a \ind{Gaussian channel}\index{channel!Gaussian}
with binary input $x \in \{ -1, +1\}$
and {\em real\/} output alphabet $\A_Y$, with transition probability density
\beq
Q(y \given x,\sa,\sigma) = \frac{1}{\sqrt{2 \pi \sigma^2}}
\, e^{-\smallfrac{(y-x \sa)^2}{2 \sigma^2}} ,
\eeq
where $\sa$ is the signal amplitude.
\ben
\item
Compute the posterior probability of $x$ given $y$, assuming that
the two inputs are equiprobable.
Put your answer in the form
\beq
P(x\eq 1 \given y,\sa,\sigma) = \frac{1}{1+e^{-a(y)}} .
\eeq
Sketch the value of $P(x\eq 1 \given y,\sa,\sigma)$
as a function of $y$.
\item
Assume that a single bit is to be
transmitted. What is the optimal decoder,
and what is its probability of error? Express your answer in terms
of the signal-to-noise ratio $\sa^2/\sigma^2$ and
the
\label{sec.erf}\ind{error function}\index{conventions!error function}\index{erf}
(the \ind{cumulative
probability function} of the Gaussian distribution),
\beq
\Phi(z) \equiv \int_{-\infty}^{z} \frac{1}{\sqrt{2 \pi}}
\, e^{-\textstyle\frac{z^2}{2}} \: \d z.
\eeq
%
% P(x \given y,s,sigma) = 1/(1+e^{-a}), a = 2 ( s / \sigma^2 ). y
%
[Note that this definition of the error function $\Phi(z)$ may not
correspond to other people's.]
% definitions of the `error function'.
% Some people
%% and some software libraries
% leave out factors of two in the definition.]
% I think that the
% above definition is the only natural one.
\een
}
% \section{
\subsection*{Pattern recognition as a noisy channel}
We may think of many pattern recognition problems in terms of\index{pattern recognition}
\ind{communication} channels. Consider the case of recognizing handwritten
digits (such as postcodes on envelopes). The author of the digit
wishes to communicate a message from the set $\A_X = \{
0,1,2,3,\ldots, 9 \}$; this selected message is the input to the
channel. What comes out of the channel is a pattern of ink on paper.
If the ink pattern is represented using 256 binary pixels, the channel $Q$
has as its output a random variable $y \in \A_Y = \{0,1\}^{256}$.
% Here is an example of an element from this alphabet.
An example of an element from this alphabet is shown in the margin.
%
% hintond.p zero=0.0 range=1.25 rows=16 background=1.0 pos=0.0 o=/home/mackay/_applications/characters/ex2.ps 16 < /home/mackay/_applications/characters/example2
%
%\[
\marginpar{
{\psfig{figure=/home/mackay/_applications/characters/ex2.ps,width=1.1in}}
}%\end{marginpar}
%\]
\exercisaxA{2}{ex.twos}{
Estimate how many patterns in $\A_Y$ are recognizable as
the character `2'. [The aim of this problem is to
% Try not to underestimate this number ---
try to demonstrate the existence of {\em as many patterns as possible\/}
that are recognizable as 2s.]
\amarginfig{t}{
\begin{center}
\mbox{\psfig{figure=figs/random2.ps}}
\\[0.15in]%\hspace{0.42in}
\mbox{\psfig{figure=figs/2random2.ps}}
\\[0.15in]%\hspace{0.42in}
\mbox{\psfig{figure=figs/6random2.ps}}
\\[0.15in]%\hspace{0.42in}
\mbox{\psfig{figure=figs/7random2.ps}}
\end{center}
\caption[a]{Some more 2s.}
\label{fig.random2s}
%\end{figure}
}%end{marginfig}
% made using figs/random2.ps seed=7
Discuss how one might model the channel $P(y \given x\eq 2)$.\index{2s}\index{twos}\index{handwritten digits}
% in the case of handwritten digit recognition.
Estimate the entropy of the probability distribution $P(y \given x\eq 2)$.
% Recognition of isolated handwritten digits
% Digit 2 -> Q -> y $\in \{0,1\}^{256})$
% 3
% Estimate how many 2's there are.
One strategy for doing \ind{pattern recognition} is to create a model
for $P(y \given x)$ for each value of the input $x= \{ 0,1,2,3,\ldots, 9 \}$,
then use \Bayes\ theorem to
infer $x$ given $y$.
\beq
P(x \given y) = \frac{ P(y \given x) P(x) } { \sum_{x'} P(y \given x') P(x') } .
\eeq
This strategy is known as {\dbf \ind{full probabilistic model}ling\/}
or {\dbf \ind{generative model}ling\/}. This is essentially how
current speech recognition systems work. In addition to the
channel model, $P(y \given x)$, one uses a prior probability distribution
$P(x)$, which in the case of both character recognition and
speech recognition is a language model that specifies the probability of
the next character/word given the context and the known grammar
and statistics of the language.
%
% Alternative, model $P(x \given y)$ directly.
% Discriminative modelling; conditional modelling.
% Feature extraction -- compute some $f(y)$ then model $P(f \given x)$
% - generative modelling in feature space.
% or else model $P(x \given f)$
% which is still discriminative modelling / conditional modelling.
% Notice number of parameters.
%
%
}
\subsection*{Random coding}
\exercissxA{2}{ex.birthday}{
Given
%\index{random coding}
% \index{code!random}
\index{random code}twenty-four people in a room,
% at a party,
what is the probability that
there are at least two people present who
% of them
have the same \ind{birthday} (\ie, day and month of birth)?
What is the expected number of
pairs of people with the same birthday? Which of these
two questions is easiest to solve? Which answer gives most
insight?
You may find it helpful to solve these problems and those that follow
using notation such as $A=$ number of days in year $=365$
and $S=$ number of people $=24$.
}
\exercisaxB{2}{ex.birthdaycode}{
The birthday problem may be related to a coding scheme.
Assume we wish to convey a message
to an outsider identifying one of the twenty-four people.
We could simply communicate a number $\cwm$ from $\A_S = \{ 1,2, \ldots,
24 \}$, having agreed a mapping of people onto numbers;
alternatively, we could convey a number from
$\A_X = \{ 1 ,2 , \ldots, 365\}$, identifying the
day of the year that is the selected person's \ind{birthday}
(with apologies to leapyearians). [The receiver is assumed to know
all the people's birthdays.]
What, roughly, is the probability of error of this communication scheme,
assuming it is used for a single transmission?
What is the capacity of the communication channel, and what is
the rate of communication attempted by this scheme?
}
%
% CHRIS SAYS ``this is not CLEAR''................. :
%
\exercisaxB{2}{ex.birthdaycodeb}{
Now imagine that there are $K$ rooms in a building, each containing
$q$ people. (You might think of $K=2$ and $q=24$ as an example.)
The aim is to communicate a selection of one person
from each room by transmitting an ordered list of $K$ days (from $\A_X$).
Compare the probability of error of the following two schemes.
\ben
\item
As before, where each
room transmits the \ind{birthday} of the selected person.
\item
To each $K$-tuple of people, one drawn from each room,
an ordered $K$-tuple of randomly selected days from $\A_X$ is assigned
(this $K$-tuple has nothing to do with their birthdays).
This enormous list of $S = q^K$ strings is
known to the receiver. When the building has selected
a particular person from each room, the ordered string of days
corresponding to that $K$-tuple of people is transmitted.
\een
What is the probability of error when $q=364$ and $K=1$?
What is the probability of error when $q=364$ and $K$ is large,
\eg\ $K=6000$?
}
% see synchronicity.tex
% for cut example
\dvips
\section{Solutions}% to Chapter \protect\ref{ch5}'s exercises}
%
\fakesection{solns to exercises in l5.tex}
%
\soln{ex.bscy0}{
If we assume we observe $y\eq 0$,
\beqan
P(x\eq 1 \given y\eq 0) &=& \frac{ P(y\eq 0 \given x\eq 1) P(x\eq 1)}{\sum_{x'} P(y \given x') P(x')} \\
&=& \frac{ 0.15 \times 0.1 }{ 0.15 \times 0.1 + 0.85 \times 0.9 } \\
&=& \frac{ 0.015 }{0.78} \:=\: 0.019 .
\eeqan
}
\soln{ex.zcy0}{
If we observe $y=0$,
\beqan
P(x\eq 1 \given y\eq 0)
% &=& \frac{ P(y\eq 0 \given x\eq 1) P(x\eq 1)}{\sum_{x'} P(y \given x') P(x')} \\
&=& \frac{ 0.15 \times 0.1 }{ 0.15 \times 0.1 + 1.0 \times 0.9 } \\
&=& \frac{ 0.015}{ 0.915} \:=\: 0.016 .
\eeqan
}
\soln{ex.bscMI}{
The probability that $y=1$
is $0.5$, so the mutual information is:
\beqan
\I(X;Y) &=& H(Y) - H(Y \given X) \\
&=& H_2(0.5) - H_2(0.15)\\
& =& 1 - 0.61 \:\: = \:\: 0.39 \mbox{ bits}.
\eeqan
}
\soln{ex.zcMI}{
We again compute the mutual information using
$\I(X;Y) = H(Y) - H(Y \given X)$.
% fixed Tue 18/2/03
The probability that $y=0$
is $0.575$, and $H(Y \given X) = \sum_x P(x) H(Y \given x) = P(x\eq1) H(Y \given x\eq1) $
$+$ $P(x\eq0) H(Y \given x\eq0)$ so the mutual information is:
\beqan
\I(X;Y) &=& H(Y) - H(Y \given X) \\
&=& H_2(0.575) - [0.5 \times H_2(0.15)+0.5 \times 0 ] \\
& =& 0.98 - 0.30 \:\: = \:\: 0.679 \mbox{ bits}.
\eeqan
}
\soln{ex.bscC}{
By symmetry, the \optens\ is
$\{0.5,0.5\}$.
Then the capacity is
\beqan
C \:=\: \I(X;Y) &=& H(Y) - H(Y \given X) \\
&=& H_2(0.5) - H_2(\q)\\
& =& 1 - H_2(\q) .
\eeqan
Would you like to find the \optens\ without invoking symmetry?
We can do this by computing the mutual information in the general
case where the input ensemble is $\{p_0,p_1\}$:
\beqan
\I(X;Y) &=& H(Y) - H(Y \given X) \\
&=& H_2(p_0 \q+ p_1(1-\q) ) - H_2(\q) .
\eeqan
The only $p$-dependence is in the first term $H_2(p_0\q+ p_1(1-\q) )$,
which is maximized by setting the argument to 0.5.
This value is given by setting $p_0=1/2$.
}
\soln{ex.becC}{
\noindent {\sf Answer 1}.
By symmetry, the \optens\ is
$\{0.5,0.5\}$. The capacity is
most easily evaluated by
writing the mutual information as $\I(X;Y) = H(X) - H(X \given Y)$.
The conditional entropy $H(X \given Y)$ is $\sum_y P(y) H(X \given y)$;
when $y$ is known, $x$ is only uncertain if $y=\mbox{\tt{?}}$, which
occurs with probability $\q/2+\q/2$,
so the conditional entropy $H(X \given Y)$ is $\q H_2(0.5)$.
\beqan
C \:=\: \I(X;Y) &=& H(X) - H(X \given Y) \\
&=& H_2(0.5) - \q H_2(0.5)\\
& =& 1 - \q .
\eeqan
% The conditional entropy $H(X \given Y)$ is $\q H_2(0.5)$.
%
The binary erasure channel
fails a fraction $\q$ of the
time. Its capacity is precisely
$1-\q$, which is the fraction of
the time that the channel is
reliable.
% functional.
% , even though the sender
% does not know when the channel will
% fail.
This result seems very reasonable, but it is far from obvious
how to encode information so as to communicate {\em reliably\/} over this channel.
\smallskip
\noindent {\sf Answer 2}.
Alternatively, without invoking the symmetry assumed above, we can
start from the input ensemble $\{p_0,p_1\}$. The probability that
$y=\mbox{\tt{?}}$ is $p_0 \q+ p_1 \q = \q$, and when we receive $y=\mbox{\tt{?}}$,
the posterior probability of $x$ is the same as the prior
probability, so:
\beqan
\I(X;Y) &=& H(X) - H(X \given Y) \\
&=& H_2(p_1) - \q H_2(p_1)\\
& =& (1 - \q ) H_2(p_1) .
\eeqan
This mutual information achieves its maximum value of $(1-\q)$ when
$p_1=1/2$.
}
%
%
%
\begin{figure}[htbp]
\figuremargin{%
\begin{center}
\begin{tabular}{ccccc}
$\bQ$
& \ecfig{bec.1}
&{\small{(a)}} \, \ecfig{bec.2}
&{\small{(b)}} \,
% roughly 8pts from col to col
\setlength{\unitlength}{1pt}
\begin{picture}(50,110)(-5,-5)
\put(-5,-5){\ecfig{bec.2}}
\put(3.95,-3){\framebox(8,96){}}
\put(28.5,-3){\framebox(8,96){}}
\put(2.5,97){\makebox(0,0)[bl]{\small$\bx^{(1)}$}}
\put(26.5,97){\makebox(0,0)[bl]{\small$\bx^{(2)}$}}
\end{picture}
&{\small{(c)}} \,
% roughly 8pts from col to col
\setlength{\unitlength}{1pt}
\begin{picture}(50,110)(-5,-5)
\put(-5,-5){\ecfig{bec.2}}
\put(3.95,-3){\framebox(8,96){}}
\put(28.5,-3){\framebox(8,96){}}
\put(2.5,97){\makebox(0,0)[bl]{\small$\bx^{(1)}$}}
\put(26.5,97){\makebox(0,0)[bl]{\small$\bx^{(2)}$}}
% roughly 8pts from col to col
%\setlength{\unitlength}{1pt}
%\begin{picture}(100,110)(-5,-5)
%\put(-5,-5){\ecfig{bec.2}}
%\put(3.95,-3){\framebox(8,96){}}
%\put(28.5,-3){\framebox(8,96){}}
%\put(2.5,97){\makebox(0,0)[bl]{$\bx^{(1)}$}}
%\put(26.5,97){\makebox(0,0)[bl]{$\bx^{(2)}$}}
%
\multiput(-4,3)(0,8){2}{\line(1,0){8}}
\multiput(-4,27)(0,8){3}{\line(1,0){8}}
\multiput(-4,59)(0,8){2}{\line(1,0){8}}
\multiput(37,3)(0,8){2}{\vector(1,0){14}}
\multiput(37,27)(0,8){3}{\vector(1,0){14}}
\multiput(37,59)(0,8){2}{\vector(1,0){14}}
\multiput(57,4)(0,8){2}{\makebox(0,0)[l]{\tiny$\hat{m}=2$}}
\multiput(57,28)(0,8){1}{\makebox(0,0)[l]{\tiny$\hat{m}=2$}}
\multiput(57,44)(0,8){1}{\makebox(0,0)[l]{\tiny$\hat{m}=1$}}
\multiput(57,60)(0,8){2}{\makebox(0,0)[l]{\tiny$\hat{m}=1$}}
\multiput(57,36)(0,8){1}{\makebox(0,0)[l]{\tiny$\hat{m}=0$}}
% the box starts exactly at x=0.
\end{picture}
\\
& $N=1$ & $N=2$ & \\[-0.1in]
\end{tabular}
\end{center}
}{%
\caption[a]{(a) The {\ind{extended channel}} ($N=2$)
obtained from a binary erasure channel
with erasure probability 0.15. (b) A block code
consisting of the two codewords {\tt 00} and {\tt 11}.
(c) The optimal decoder for this code. }
\label{fig.extended.bec}
}
\end{figure}
%
\soln{ex.extended}{
The extended channel is shown in \figref{fig.extended.bec}.
The best code for this channel with $N=2$ is obtained by choosing
two columns that have minimal overlap, for example, columns {\tt 00}
and {\tt 11}. The decoding algorithm returns `{\tt 00}'
if the extended channel output is among the top four
% either output is {\tt 0},
and `{\tt 11}' if
it's among the bottom four,
% if either output is {\tt 1},
and gives up if the output is `{\tt ??}'.
}
%
% end of chapter
%
\soln{ex.zcdiscuss}{
In \exampleref{exa.Z.HXY}
% \exaseven\ of chapter \chfive\
we showed that the mutual information between input and output
of the Z channel is
\beqan
\I(X;Y) &=& H(Y) - H(Y \given X) \nonumber \\
&=& H_2(p_1 (1-\q)) - p_1 H_2(\q) .
\eeqan
We differentiate this expression with respect to $p_1$, taking care not
to confuse $\log_2$ with $\log_e$:
\beq
\frac{\d}{\d p_1} \I(X;Y)
= (1-\q) \log_2 \frac{ 1- p_1 (1-\q) }{ p_1 (1-\q) } - H_2(\q) .
\eeq
Setting this derivative to zero and rearranging using skills developed
in \exthirtyone, we obtain:
\beq
{ p_1^* (1-\q) } = \frac{1}{1 + \displaystyle 2^{H_2(\q)/(1-\q)}} ,
\eeq
so the \optens\ is
\beq
p_1^* = \frac{ 1/(1-\q) }
{ \displaystyle
1 + 2^{ \left( H_2(\q) / ( 1 - \q ) \right)} } .
\eeq
As the noise level $\q$ tends to 1, this expression tends to $1/e$
(as you can prove using L'H\^opital's rule).
For all values of $\q\!$, $p_1^*$ is smaller than $1/2$. A rough
intuition for why input {\tt1} is used less than input {\tt0} is that
when input {\tt1} is used, the noisy channel injects entropy into
the received string; whereas when input {\tt0} is used, the noise has
zero entropy. Thus starting from $p_1=1/2$, a perturbation
towards smaller $p_1$ will reduce the conditional entropy
$H(Y \given X)$ linearly while leaving $H(Y)$ unchanged, to first order.
$H(Y)$ decreases only quadratically in $(p_1-\dhalf)$.
}
\soln{ex.Csketch}{
The capacities of the three channels are shown in \figref{fig.capacities}.
% below.
\amarginfig{b}{
\begin{center}
\mbox{\psfig{figure=figs/C.ps,angle=-90,width=2in}
}
\end{center}
\caption[a]{Capacities of the Z channel, \BSC, and binary erasure channel.}
\label{fig.capacities}
}%end marginpar
For any $\q <0.5$,
% the channels can be ordered with the BEC being the
the BEC is the
channel with highest capacity and the BSC the lowest.
}
\soln{ex.GC}{
The logarithm of the posterior probability ratio, given $y$, is
\beq
a(y) = \ln \frac{P(x\eq 1 \given y,\sa,\sigma)}{P(x\eq -1 \given y,\sa,\sigma)}
= \ln \frac{Q(y \given x\eq 1,\sa,\sigma)}{Q(y \given x\eq -1,\sa,\sigma)}
= 2 \frac{\sa y}{\sigma^2} .
% corrected march 2000
% and corrected log to ln Sun 22/8/04
\eeq
Using our skills picked up from
% in chapter \ref{ch1},
\exerciseref{ex.logit}, we rewrite
% from exercise \label{eq.sigmoid} \label{eq.logistic}
this in the form
\beq
P(x\eq 1 \given y,\sa,\sigma) = \frac{1}{1+e^{-a(y)}} .
\eeq
The optimal decoder selects the most probable hypothesis; this can
be done simply by looking at the sign of $a(y)$. If $a(y)>0$
then decode as $\hat{x}=1$.
The probability of error is
\beq
p_{\rm b} = \int_{-\infty}^{0} \!\! \d y \:
Q(y \given x\eq 1,\sa,\sigma) =
% chris suggests removing the x (=1) from what follows (twice)
\int_{-\infty}^{- x \sa} \! \d y \: \frac{1}{\sqrt{2 \pi \sigma^2}}
e^{-\smallfrac{y^2}{2 \sigma^2}}
= \Phi \left( - \frac{ x\sa }{ \sigma } \right) .
% corrected march 2000
\eeq
% where
%\beq
% \Phi(z) \equiv \int_{z}^{\infty} \frac{1}{\sqrt{2 \pi}}
% e^{-\frac{z^2}{2}} .
%\eeq
%\beq
% \Phi(z) \equiv \int_{-\infty}^{z}{\smallfrac{1}{\sqrt{2 \pi}}}
% e^{-\textstyle\frac{z^2}{2}} .
%\eeq
}
\subsection*{Random coding}
\soln{ex.birthday}{
The probability that $S=24$ people whose birthdays are drawn at random
from $A=365$ days all have {\em distinct\/} birthdays is
\beq
\frac{ A(A-1)(A-2)\ldots(A-S+1) }{ A^q } .
\eeq
The probability that two (or more) people share a \ind{birthday} is one minus
this quantity, which, for $S=24$ and $A=365$,
is about 0.5. This exact way of answering the question
is not very informative
since it is not clear for what
value of $S$ the probability changes from being close to 0 to being
close to 1.
The number of pairs is $S(S-1)/2$, and the probability that a particular
pair shares a birthday is $1/A$, so the {\em expected number\/} of collisions
is
\beq
\frac{ S(S-1)}{2 } \frac{1}{A} .
\eeq
This answer is more instructive. The expected number of collisions
is tiny if $S \ll \sqrt{A}$ and big if $S \gg \sqrt{A}$.
We can also approximate the probability that all birthdays are distinct,
for small $S$, thus:
\beqan
\lefteqn{\hspace*{-0.7in}
\frac{ A(A-1)(A-2)\ldots(A-S+1) }{ A^S }
\:\:=\:\: (1)(1-\dfrac{1}{A})(1-\dfrac{2}{A})\ldots(1-\dfrac{(S\!-\!1)}{A})
\hspace*{1.7in}}
% this hspace{ no good
\nonumber \\
&\simeq&
\exp( 0 ) \exp ( -\linefrac{1}{A}) \exp ( -\linefrac{2}{A}) \ldots \exp ( -\linefrac{(S\!-\!1)}{A})
\\
&\simeq&
\exp \left( - \frac{1}{A} \sum_{i=1}^{S-1} i \right)
= \exp \left( - \frac{S(S-1)/2}{A} \right) .
\eeqan
}
\dvipsb{solutions noisy channel s5}
\prechapter{About Chapter}
\fakesection{prerequisites for chapter 6}
Before reading \chref{ch.six}, you should have read Chapters
\chtwo\ and \chfive. \Exerciseref{ex.extended} is
especially recommended.
% and worked on \exerciseref{ex.dataprocineq}.
%
% \extwentytwo\ from chapter \chone.
% Please note that you {\em don't\/} need to understand
% this proof in order to be able to solve most of the
% problems involving noisy channels.
%\footnote
% {This exposition is based on that of Cover and Thomas (1991).}
\subsection*{Cast of characters}
\noindent%
\begin{tabular}{lp{4in}} \toprule
$Q$ & the noisy channel \\
$C$ & the capacity of the channel \\
$X^N$ & an ensemble used to create a \ind{random code} \\
$\C$ & a random code \\
$N$ & the length of the codewords \\
$\bx^{(\cwm)}$ & a codeword, the $\cwm$th in the code \\
$\cwm$ % $s$
& the number of a chosen codeword
(mnemonic: the {\em source\/} selects $\cwm$) \\
$\cwM = 2^{K}$ % $S$
& the total number of codewords in the code\\
$K=\log_2 \cwM$
& the number of bits conveyed by the choice of one codeword from $\cwM$,
assuming it is chosen with uniform probability \\
$\bs$ & a binary representation of the number $\cwm$ \\
$R = K/N$ & the rate of the code, in bits per channel use
(sometimes called $R'$ instead) \\
% $R'$ & another rate, close to $R$ \\
$\hat{\cwm}$ % $s$
& the decoder's guess of $\cwm$ \\
\bottomrule
\end{tabular} \medskip
%{\sf Typo Warning:}
% the letter $m$ may turn up where it should read $\cwm$.
%%%% !!!!!!!!!!!!!! ok???????????????????????
\ENDprechapter
\chapter{The Noisy-Channel Coding Theorem}
% {The noisy-channel coding theorem}% Proof of
\label{ch.six}
% % \lecturetitle{The noisy-channel coding theorem, part b}
% \chapter{The noisy channel coding theorem}% Proof of
\label{ch6}
\section{The theorem}\index{noisy-channel coding theorem}\index{communication}
The theorem has three parts, two positive and one negative.
The main positive result is the first.
\amarginfig{t}{
\begin{center}\small
\setlength{\unitlength}{2pt}
\begin{picture}(60,45)(-2.5,-7)
\thinlines
\put(0,0){\vector(1,0){60}}
\put(0,0){\vector(0,1){40}}
\put(30,0){\line(0,1){30}}
\put(30,0){\line(1,2){10}}
\put(30,-3){\makebox(0,0)[t]{$C$}}
\put(55,-2){\makebox(0,0)[t]{$R$}}
\put(42,22){\makebox(0,0)[bl]{$R(p_{\rm b})$}}
\put(-1,35){\makebox(0,0)[r]{$p_{\rm b}$}}
\thicklines
\put(0,0){\makebox(30,30){1}}
\put(30,0){\makebox(7.5,35){2}}
\put(35,0){\makebox(30,20){3}}
% \put(0,0){\line(0,1){50}}
%
\end{picture}
\end{center}
\caption[a]{Portion of the $R,p_{\rm b}$ plane
to be proved
achievable (1,$\,$2) and
not achievable (3).
}
\label{fig.belowCcoming}
}%end marginfig
\ben%gin{itemize}
\item
For every discrete memoryless channel, the
channel capacity
\beq
C = \max_{\P_X}\, \I(X;Y)
\eeq
has the following
property. For any $\epsilon > 0$ and $R < C$, for large enough $N$,
there exists a code of length $N$ and rate $\geq R$ and a decoding
algorithm, such that the maximal probability of block error is $<
\epsilon$.
\item
If a probability of bit error $p_{\rm b}$ is acceptable, rates up to $R(p_{\rm b})$
are achievable, where
\beq
R(p_{\rm b}) = \frac{ C }
{1 - H_2(p_{\rm b})} .
\eeq
\item
For any $p_{\rm b}$, rates greater than $R(p_{\rm b})$ are not achievable.
\een%d{itemize}
\section{Jointly-typical sequences}
We formalize the intuitive preview of the last chapter.\index{typicality}
We will define codewords $\bx^{(\cwm )}$ as coming from an ensemble $X^N$,
and consider the random selection of one codeword and a
corresponding channel output $\by$,
thus defining a joint ensemble $(XY)^N$.
%, corresponding to random generation of a codeword and a corresponding channel output.
We will use a {\dem typical-set decoder}, which
decodes
a received signal
$\by$ as $\cwm$ if $\bx^{(\cwm )}$ and $\by$ are {\dem jointly typical},
a term to be defined shortly.
The proof will then centre on determining the probabilities (a) that the true
input codeword is {\em not\/}
jointly \index{typicality}{typical} with the output sequence;
and (b) that a {\em false\/} input codeword {is\/} jointly typical with the output.
We will show that, for large $N$, both probabilities
% $\rightarrow 0$,
go to zero
as long as there are fewer
than $2^{NC}$ codewords, and the ensemble $X$ is the \index{optimal input distribution}{\optens}.
\newcommand{\JNb}{\mbox{$J_{N \beta}$}}
\begin{description}
\item[Joint typicality\puncspace]
A pair of sequences $\bx,\by$ of length $N$ are defined to be
{jointly
typical (to tolerance $\beta$)}\index{joint typicality}
with respect to the distribution
$P(x,y)$ if
\beqan
\mbox{$\bx$ is typical of $P(\bx)$,}
& \mbox{\ie,} &
\left| \frac{1}{N} \log \frac{1}{P(\bx)} - H(X) \right| < \beta ,
\nonumber
\\
\mbox{$\by$ is typical of $P(\by)$,}
& \mbox{\ie,} &
\left| \frac{1}{N} \log \frac{1}{P(\by)} - H(Y) \right| < \beta ,
\nonumber
\\
\mbox{and $\bx,\by$ is typical of $P(\bx,\by)$,}
& \mbox{\ie,} &
\left| \frac{1}{N} \log \frac{1}{P(\bx,\by)} - H(X,Y) \right| < \beta .
\nonumber
\eeqan
\item[The jointly-typical set] $\JNb$ is the set of all jointly
typical sequence pairs of length $N$.
% It has the following three properties,
\end{description}
%\begin{example}
\noindent {\sf Example.}
Here is a jointly-typical pair of length $N=100$
for the ensemble
$P(x,y)$ in which $P(x)$ has $(p_0,p_1) = (0.9,0.1)$
and $P(y \given x)$ corresponds to a binary symmetric channel with
noise level $0.2$.
\[%beq
\mbox{
\begin{tabular}{cc}
$\bx$ &\mbox{\footnotesize\tt 1111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000}\\
$\by$ &\mbox{\footnotesize\tt 0011111111000000000000000000000000000000000000000000000000000000000000000000000000111111111111111111}
\end{tabular}
}
\]%eeq
Notice that $\bx$ has 10 {\tt 1}s, and so is typical of the probability
$P(\bx)$ (at any tolerance $\beta$); and $\by$ has
% 18 + 8 = 26
26 {\tt 1}s, so it is typical of $P(\by)$ (because $P(y\eq 1) = 0.26$);
and $\bx$ and $\by$ differ in
% 18 + 2
20 bits, which is the typical number of flips for
this channel.
%\end{example}
\begin{description}
\item[Joint typicality theorem\puncspace]
Let $\bx,\by$ be drawn from the ensemble $(XY)^N$ defined
by
$$P(\bx,\by)=\prod_{n=1}^N P(x_n,y_n).$$
Then\index{joint typicality theorem}\label{theorem.jtt}
\ben
\item
the probability that $\bx,\by$
are jointly typical (to tolerance $\beta$)
tends to 1 as $N \rightarrow \infty$;
\item
the number of jointly-typical sequences $|\JNb|$
is close to $2^{N H(X,Y) }$. To be precise,
\beq
|\JNb| \leq 2^{N ( H(X,Y) + \beta ) };
\eeq
\item
if $\bx'\sim X^N$ and $\by'\sim Y^N$, \ie, $\bx'$ and $\by'$
are {\em independent\/} samples
with the same marginal distribution as $P(\bx,\by)$, then
the probability that $(\bx' ,\by')$ lands in the
jointly-typical set is about $2^{- N \I(X;Y)}$. To be precise,
\beq
P( (\bx' ,\by') \in \JNb )
\leq 2^{- N ( \I(X;Y) - 3 \beta ) } .
\eeq
% also, for the proof of the converse, we want...
% for sufficiently large N
% P( (\bx' ,\by') \in \JNb
% \geq (1-\beta) 2^{- N ( \I(X;Y) + 3 \beta ) }
\een
\item[{\sf Proof.}] The proof of parts 1 and 2
by the law of large numbers follows that of the source coding theorem in
\chref{ch2}. For part 2, let the pair $x,y$ play the role of
$x$ in the source coding theorem, replacing $P(x)$ there by
the probability distribution $P(x,y)$.
% \marginpar{\footnotesize }
For the third part,
\beqan
% \begin{array}{lll}
% was (thin column) --
% \multicolumn{3}{l}{
% P( (\bx' ,\by') \in \JNb )
% \: = \: \sum_{(\bx ,\by) \in \JNb} P(\bx ) P(\by)}
% \\[0.06in]
% &\leq & |\JNb| \, 2^{-N(H(X)-\beta)} 2^{-N(H(Y)-\beta)}
% \\[0.045in]
% &\leq& 2^{N( H(X,Y) + \b) - N(H(X)+H(Y)-2\b)}
% \\
% & =& 2^{-N ( \I(X;Y) - 3 \beta )}
P( (\bx' ,\by') \in \JNb )
& = & \sum_{(\bx ,\by) \in \JNb} P(\bx ) P(\by)
\\% [0.06in]
&\leq & |\JNb| \, 2^{-N(H(X)-\beta)} \, 2^{-N(H(Y)-\beta)}
\\% [0.045in]
&\leq& 2^{N( H(X,Y) + \b) - N(H(X)+H(Y)-2\b)}
\\
& =& 2^{-N ( \I(X;Y) - 3 \beta )} . \hspace{1in}\epfsymbol
\eeqan
% This quantity is a bound on the probability of confusing
\end{description}
A cartoon of the jointly-typical set is shown in
\figref{fig.joint.typ}.
% The property just proved, that t
Two
independent typical vectors are jointly typical with probability
\beq
P(
(\bx' ,\by') \in \JNb ) \simeq 2^{-N ( \I(X;Y))}
\eeq
% because
%, is readily understood by noticing that
because the {\em total\/} number of independent typical pairs is the
area of the dashed rectangle, $2^{NH(X)} 2^{NH(Y)}$, and the number of
jointly-typical pairs is roughly $2^{NH(X,Y)}$, so the probability of hitting
a jointly-typical pair is roughly
\beq
2^{NH(X,Y)}/2^{NH(X)+NH(Y)} = 2^{-N\I(X;Y)}.
\eeq
%
% the above eq was in-line but it looked ugly
%
\newcommand{\rad}{0.81}
\begin{figure}
\small
\figuremargin{%
\begin{center}\small
\setlength{\unitlength}{1mm}% original picture is 9.75 in by 5.25 in
\begin{picture}(74,105)(-15,-5)
%
\put(-10,-7){\framebox(62,99){}}
% as well as box put Ax and Ay sizes
\put(0,93.5){\vector(-1,0){10}}
\put(0,93.5){\vector(1,0){52}}
\put(-11,8){\vector(0,-1){15}}
\put(-11,8){\vector(0,1){84}}
%\put(0,92){\vector(-1,0){10}}
%\put(0,92){\vector(1,0){52}}
%\put(-10,8){\vector(0,-1){15}}
%\put(-10,8){\vector(0,1){84}}
%
% width indicator
\put(21,90){\vector(1,0){21}}
\put(21,90){\vector(-1,0){21}}
\put(21,88.7){\makebox(0,0)[t]{$2^{NH(X)}$}}
%
% height indicator
\put(-2,45){\vector(0,1){43}}
\put(-2,45){\vector(0,-1){43}}
\put(0,30){\makebox(0,0)[l]{$2^{NH(Y)}$}}% was 45
%
% RECTANGLE
%\put(-1,0){\framebox(45,89){}}
\put(-1,1){\dashbox{1}(44.5,88){}}
%
% strip width indicator
\put(26,35){\vector(1,0){2}}
\put(26,35){\vector(-1,0){2}}
\put(26,15){\vector(1,0){2}}
\put(26,15){\vector(-1,0){2}}
\put(26,14){\makebox(0,0)[t]{$2^{NH(X|Y)}$}}
%
% strip height indicator
\put(21,45){\vector(0,1){5}}
\put(21,45){\vector(0,-1){5}}
\put(28,45){\vector(0,1){5}}% was at 31,32
\put(28,45){\vector(0,-1){5}}
\put(29,45){\makebox(0,0)[l]{$2^{NH(Y|X)}$}}
%
% JT set
\multiput(2,88)(2,-4){21}{\circle*{\rad}}
\multiput(2,86)(2,-4){21}{\circle*{\rad}}
\multiput(0,86)(2,-4){22}{\circle*{\rad}}
\multiput(0,88)(2,-4){22}{\circle*{\rad}}
\multiput(0,82)(2,-4){21}{\circle*{\rad}}
\multiput(0,84)(2,-4){21}{\circle*{\rad}}
%
%\put(38,20){\makebox(0,0)[l]{$2^{NH(X,Y)}$}}
\put(18,64){\makebox(0,0)[l]{$2^{NH(X,Y)}$ dots}}
\put(21,96){\makebox(0,0)[b]{$\A_{X}^N$}}
\put(-12,45){\makebox(0,0)[r]{$\A_{Y}^N$}}
\end{picture}
\end{center}
}{%
\caption[a]{{The jointly-typical set.} The horizontal direction
represents $\A_{X}^N$, the set of all input strings of length $N$.
The vertical direction
represents $\A_{Y}^N$, the set of all output strings of length $N$.
The outer box contains all conceivable input--output pairs.
Each dot represents
a jointly-typical pair of sequences $(\bx,\by)$.
The total number of jointly-typical sequences is about $2^{NH(X,Y)}$.
% [Compare with \protect\figref{fig.extended.bec}a,
% \protect\pref{fig.extended.bec}.]
% page \protect\pageref{fig.extended.bec}.]
}
\label{fig.joint.typ}
}
\end{figure}
\section{Proof of the noisy-channel coding theorem}
\subsection{Analogy}
Imagine that we wish to prove that there is a baby\index{weighing babies} in a class
of one hundred babies who weighs less than 10\kg. Individual babies
are difficult to catch and weigh.%
\amarginfig{c}{
\begin{center}
\mbox{\psfig{figure=figs/babiesscale4.ps,width=53mm}}
\end{center}
\caption[a]{Shannon's method for
proving one baby weighs less than 10\kg.}
}
Shannon's method of\index{Shannon, Claude}
solving the task is to scoop up all the babies and weigh them
all at once on a big weighing machine. If we find that their {\em average\/} weight is
% smaller than 1000\kg\ then the children's average weight
% must be
smaller than 10\kg, there must exist {\em at least one\/}
baby who weighs less than 10\kg\ -- indeed there must be many!
% In the context of weighing children,
Shannon's method isn't guaranteed to reveal the existence of an underweight child,
since it relies on
there being a tiny number of elephants in the class. But if we use his method
and get a total weight smaller than 1000\kg\ then our task is solved.
\subsection{From skinny children to fantastic codes}
We wish to show that there exists a code and a decoder having small
probability of error. Evaluating the probability of error of any
particular coding and decoding
system is not easy. Shannon's innovation was this: instead of
constructing a good coding and decoding
system and evaluating its error probability,
Shannon
calculated the average probability of block error of {\em all\/}
codes, and proved that this average is small. There must then exist
individual codes that have small probability of block error.
% Finally
% to prove that the {\em maximal\/} probability of error is small too,
% we modify one of these good codes by throwing away the worst 50\%
% of its codewords.
\begin{figure}
\small
\figuremargin{%
\begin{center}
\begin{tabular}{cc}
\setlength{\unitlength}{0.81mm}% original picture is 9.75 in by 5.25 in
%\begin{picture}(74,100)(-15,-5)
\begin{picture}(62,100)(-5,-5)
%
%\put(-10,-2){\framebox(62,94){}}
% codewords
\put( 5,0){\framebox(2,91){}}
\put(13,0){\framebox(2,91){}}
\put(31,0){\framebox(2,91){}}
\put(35,0){\framebox(2,91){}}
%
\put(5,94){\makebox(0,2.5)[bl]{$\bx^{(3)}$}}
\put(13,94){\makebox(0,2.5)[bl]{$\bx^{(1)}$}}
\put(29,94){\makebox(0,2.5)[bl]{$\bx^{(2)}$}}
\put(37,94){\makebox(0,2.5)[bl]{$\bx^{(4)}$}}
% JT set
\multiput(2,88)(2,-4){21}{\circle*{\rad}}
\multiput(2,86)(2,-4){21}{\circle*{\rad}}
\multiput(0,86)(2,-4){22}{\circle*{\rad}}
\multiput(0,88)(2,-4){22}{\circle*{\rad}}
\multiput(0,82)(2,-4){21}{\circle*{\rad}}
\multiput(0,84)(2,-4){21}{\circle*{\rad}}
%
%\put(21,96){\makebox(0,0)[b]{$\A_{X}^N$}}
%\put(-12,45){\makebox(0,0)[r]{$\A_{Y}^N$}}
\end{picture}
&
\setlength{\unitlength}{0.81mm}
\begin{picture}(78,100)(-15,-5)
%
%\put(-10,-2){\framebox(62,94){}}
% codewords
\put(5,0){\framebox(2,91){}}
\put(13,0){\framebox(2,91){}}
\put(31,0){\framebox(2,91){}}
\put(35,0){\framebox(2,91){}}
%
\put(5,94){\makebox(0,2.5)[bl]{$\bx^{(3)}$}}
\put(13,94){\makebox(0,2.5)[bl]{$\bx^{(1)}$}}
\put(29,94){\makebox(0,2.5)[bl]{$\bx^{(2)}$}}
\put(37,94){\makebox(0,2.5)[bl]{$\bx^{(4)}$}}
%
% decodings
\put(-13,10){\makebox(0,0)[r]{$\by_c$}}
\put(-13,20){\makebox(0,0)[r]{$\by_d$}}
\put(-13,72){\makebox(0,0)[r]{$\by_b$}}
\put(-13,82){\makebox(0,0)[r]{$\by_a$}}
\put(-11.3,10){\vector(1,0){63}}
\put(-11.3,20){\vector(1,0){63}}
\put(-11.3,72){\vector(1,0){63}}
\put(-11.3,82){\vector(1,0){63}}
\put(54,10){\makebox(0,0)[l]{$\hat{\cwm}(\by_c)\eq 4$}}% was 10,
\put(54,20){\makebox(0,0)[l]{$\hat{\cwm}(\by_d)\eq 0$}}% was 25,
\put(54,72){\makebox(0,0)[l]{$\hat{\cwm}(\by_b)\eq 3$}}
\put(54,82){\makebox(0,0)[l]{$\hat{\cwm}(\by_a)\eq 0$}}
% top end
%
% JT set
\multiput(2,88)(2,-4){21}{\circle*{\rad}}
\multiput(2,86)(2,-4){21}{\circle*{\rad}}
\multiput(0,86)(2,-4){22}{\circle*{\rad}}
\multiput(0,88)(2,-4){22}{\circle*{\rad}}
\multiput(0,82)(2,-4){21}{\circle*{\rad}}
\multiput(0,84)(2,-4){21}{\circle*{\rad}}
%
%\put(21,96){\makebox(0,0)[b]{$\A_{X}^N$}}
%\put(-12,45){\makebox(0,0)[r]{$\A_{Y}^N$}}
\end{picture}
\\
(a) & (b) \\
\end{tabular}
\end{center}
}{%
\caption[a]{(a) {A \ind{random code}.}
% A random code is a selection of input
% sequences $\{ \bx^{(1)}, \ldots, \bx^{(\cwM)}\}$ from the ensemble
% $X^N$. Each codeword
% $\bx^{(\cwm)}$ is likely to be a typical sequence.
% [Compare with \protect\figref{fig.extended.bec}b,
% page \protect\pageref{fig.extended.bec}.]
(b) {Example decodings by the typical set decoder.} A sequence that is not
jointly typical
with any of the codewords, such as $\by_a$, is decoded as $\hat{\cwm}=0$.
A sequence that is jointly typical
with codeword $\bx^{(3)}$ alone, $\by_b$, is decoded as $\hat{\cwm}=3$.
Similarly, $\by_c$ is decoded as $\hat{\cwm}=4$.
A sequence that is jointly typical
with more than one codeword, such as
$\by_d$, is decoded as $\hat{\cwm}=0$.
% [Compare with \protect\figref{fig.extended.bec}c,
% page \protect\pageref{fig.extended.bec}.]
}
\label{fig.rand.code}
\label{fig.typ.set.dec}
}
\end{figure}
\subsection{Random coding and typical-set decoding}
Consider the following encoding--decoding
system, whose rate is $R'$.\index{random code}
\ben
\item
We fix $P(x)$ and generate the $\cwM = 2^{NR'}$
codewords of a $(N,NR')=(N,K)$
code $\C$
at random according to
\beq
P(\bx) = \prod_{n=1}^{N} P(x_n) .
\eeq
A random code is shown schematically in \figref{fig.rand.code}a.
\item
The code is known to both sender and receiver.
\item
A message $\cwm$ is chosen from $\{1,2,\ldots, 2^{NR'}\}$, and $\bx^{(\cwm )}$
is transmitted. The received signal is $\by$, with
\beq
P(\by \given \bx^{(\cwm )} ) = \prod_{n=1}^{N} P(y_n \given x^{(\cwm )}_n) .
\eeq
\item
The signal is decoded by {\dem{typical-set decoding}\index{typical-set decoder}}.
\begin{description}
\item[Typical-set decoding\puncspace] Decode
$\by$ as $\hat{\cwm }$ {\sf if}
$(\bx^{(\hat{\cwm })},\by)$ are jointly typical {\em
and\/} there is no other $\cwm' $ such that $(\bx^{(\cwm')},\by)$ are jointly
typical;\\
{\sf otherwise} declare a failure $(\hat{\cwm }\eq 0)$.
\end{description}
This is not
the optimal decoding algorithm, but it will be good enough, and easier
to \analyze. The typical-set decoder is illustrated in
\figref{fig.typ.set.dec}b.
\item
A decoding error occurs if $\hat{\cwm } \not = \cwm $.
\een
There are three probabilities of error that we can distinguish.
First, there is the probability of block error for a particular
code $\C$, that is,
\beq
p_{\rm B}(\C) \equiv P(\hat{\cwm } \neq \cwm \given \C).
\eeq
This is
a difficult quantity to evaluate for any given code.
Second, there is the average over all codes of this block error probability,
\beq
\langle p_{\rm B} \rangle \equiv \sum_{\C} P(\hat{\cwm } \neq \cwm \given \C)
P(\C) .
\eeq
Fortunately, this quantity is much easier to evaluate than
the first quantity $P(\hat{\cwm } \neq \cwm \given \C)$.%
\marginpar{\small\raggedright{$\langle p_{\rm B} \rangle$
is just the probability that there is a decoding error
at step 5 of the process on the previous page.}}
Third, the maximal block error probability of a code $\C$,
\beq
p_{\rm BM}(\C) \equiv \max_{\cwm } P(\hat{\cwm } \neq \cwm \given \cwm, \C),
\eeq
is the quantity we are most interested in: we wish to show
that there exists a code $\C$ with the required rate
whose maximal block error probability is small.
We will get to this result by first finding the
average block error probability, $\langle p_{\rm B} \rangle$.
Once we have shown that this can be made smaller than
a desired small number, we immediately deduce that
there must exist {\em at least one\/} code $\C$
whose block error probability is also less than this
small number. Finally, we show that this code, whose
block error probability is satisfactorily small but whose
maximal block error probability is unknown (and could
conceivably be enormous), can be
modified to make a code of slightly smaller rate whose
maximal block error probability
is also guaranteed to be small.
We modify the code by throwing away the worst 50\%
of its codewords.
We therefore now embark on finding the average probability of block error.
\subsection{Probability of error of typical-set decoder}
There are two sources of error when we use typical-set
decoding. Either (a) the output $\by$ is not jointly typical with the
transmitted codeword $\bx^{(\cwm )}$, or (b) there is some other codeword
in $\cal{C}$ that is
jointly typical with $\by$.
By the symmetry of the code construction, the average probability of error
averaged over all codes does not depend on the selected value of $\cwm$; we can
assume without loss of generality that $\cwm=1$.
(a) The probability that the input $\bx^{(1)}$ and
the output $\by$ are not jointly typical
vanishes, by the joint typicality theorem's first part
(\pref{theorem.jtt}).
We give a name, $\delta$, to the upper bound on this probability,
% .
satisfying $\delta
\rightarrow 0$ as $N \rightarrow \infty$; for any desired $\delta$,
we can find a blocklength $N(\delta)$ such that the $P( (\bx^{(1)},\by) \not \in
\JNb) \leq \delta$.
(b) The probability that $\bx^{(\cwm')}$ and $\by$
% $(\bx^{(\cwm' )},\by)$
are jointly typical, for
a {\em given\/} $\cwm' \not = 1$
is $\leq 2^{-N(\I(X;Y)-3 \beta)}$, by part 3.
And there are $(2^{NR'}-1)$ rival values of $\cwm'$ to worry about.
Thus the average probability of error $\langle p_{\rm B} \rangle$
satisfies:
\beqan
\langle p_{\rm B} \rangle &\leq &
\delta + \sum_{\cwm' =2}^{2^{NR'}} 2^{-N(\I(X;Y)-3 \beta)}
\label{eq.uniona}
\\
&\leq &
\delta + 2^{-N(\I(X;Y)- R' -3 \beta)} .
\label{eq.unionaa}
\eeqan
% MARGINPAR should align with the eqn if possible (above)
\begin{aside}
{The inequality (\ref{eq.uniona}) that bounds a
total probability of error $P_{\rm TOT}$ by the sum of the probabilities $P_{s'}$ of
all sorts of events $s'$ each of which is sufficient to cause error,
$$P_{\rm TOT} \leq P_1 + P_2 + \cdots, $$
is called a {\dem\ind{union bound}}. It is only an equality if the different events
that cause error never occur at the same time as each other.
}
\end{aside}
The average probability of error (\ref{eq.unionaa})
can be made $< 2 \delta$ by increasing $N$ if
% {\em if\/}
\beq
R' < \I(X;Y) -3 \beta .
\eeq
We are almost there. We make three modifications:
\newcommand{\expurgfig}[1]{%
\hspace*{-0.3in}\raisebox{-0.975in}[2.05in][0pt]{\psfig{figure=figs/expurgate#1.ps,width=3.2in}}\hspace*{-0.3in}}
\begin{figure}
\figuremargin{
%\marginfig{
\begin{center}\small
\begin{tabular}{c@{}c@{}c}
\expurgfig{1}
&$\Rightarrow$ &
\expurgfig{2}
\\
(a) A random code $\ldots$ & &
(b) expurgated \\
\end{tabular}
\end{center}
}{
\caption[a]{How expurgation works.
(a) In a typical random code, a small fraction of the
codewords are involved in collisions -- pairs of codewords are sufficiently
close to each other that the probability of error when either codeword
is transmitted is not tiny.
We obtain a new code from a random code by deleting
all these confusable codewords.
(b) The resulting code has slightly fewer codewords, so
has a slightly lower rate, and its maximal probability of error
is greatly reduced.
}
\label{fig.expurgate}
}
\end{figure}
% \newcommand{\optens}{optimal input distribution}
\ben
\item
We choose $P(x)$ in the proof to be the \optens\ of the channel.
Then the condition $R'<\I(X;Y) -3 \beta$ becomes $R' N C$ is not achievable, so $R > \smallfrac{C}{1-H_2(p_{\rm b})}$
is not achievable.\ENDproof
\exercisxC{3}{ex.m.s.I.aboveC}{
Fill in the details in the preceding argument.
If the bit errors between $\hat{\cwm }$ and $\cwm$ are independent
then we have $\I(\cwm;\hat{\cwm }) = N R ( 1 - H_2(p_{\rm b}))$. What if
we have complex correlations among those bit errors? Why
does the inequality $\I(\cwm;\hat{\cwm }) \geq
N R ( 1 - H_2(p_{\rm b}))$ hold?
}
\section{Computing capacity\nonexaminable}
\label{sec.compcap}
We\marginpar[c]{\small\raggedright{Sections \ref{sec.compcap}--\ref{sec.codthmpractice}
contain advanced material. The first-time reader is encouraged to
skip to section \ref{sec.codthmex} (\pref{sec.codthmex}).}}
have proved that the capacity of a channel is
the maximum rate at which reliable communication can be achieved.
How can we compute the capacity of a given discrete
memoryless channel?
We need to find its \optens. In general we can find
the \optens\ by a
computer search, making use of the derivative of the mutual information
with respect to the input probabilities.
\exercisxB{2}{ex.Iderivative}{
Find the derivative of $\I(X;Y)$ with respect to the
input probability $p_i$, $\partial \I(X;Y)/\partial p_i$, for a channel with
conditional probabilities $Q_{j|i}$.
}
\exercisxC{2}{ex.Iconcave}{
Show that $\I(X;Y)$ is a \concavefrown\ function of
the input probability vector $\bp$.
}
Since $\I(X;Y)$ is \concavefrown\ in the input distribution $\bp$,
any probability distribution $\bp$ at which
% that has $\partial \I(X;Y)/\partial p_i$
$\I(X;Y)$ is stationary
must be a global maximum of $\I(X;Y)$.
%
So it is tempting to put the derivative of $\I(X;Y)$ into a routine that
finds a local maximum of $\I(X;Y)$, that is, an input distribution
$P(x)$ such that
\beq
\frac{\partial \I(X;Y)}{\partial p_i}
= \lambda \:\:\: \mbox{for all $i$},
\label{eq.Imaxer}
\eeq
where $\lambda$ is a Lagrange multiplier associated with the constraint
$\sum_i p_i = 1$.
However, this approach may fail to find the right
answer, because $\I(X;Y)$ might be maximized
by a distribution that has $p_i \eq 0$ for some inputs.
A simple example is given by the ternary confusion channel.
\begin{description}
%
\item[Ternary confusion channel\puncspace] $\A_X \eq \{0,{\query},1\}$. $\A_Y \eq \{0,1\}$.
\[
\begin{array}{c}
\setlength{\unitlength}{0.46mm}
\begin{picture}(20,30)(0,0)
\put(5,5){\vector(1,0){10}}
\put(5,25){\vector(1,0){10}}
\put(5,15){\vector(1,1){10}}
\put(5,15){\vector(1,-1){10}}
\put(4,5){\makebox(0,0)[r]{1}}
\put(4,25){\makebox(0,0)[r]{0}}
\put(16,5){\makebox(0,0)[l]{1}}
\put(16,25){\makebox(0,0)[l]{0}}
\put(4,15){\makebox(0,0)[r]{{\query}}}
\end{picture}
\end{array}
\begin{array}{c@{\:\:\,}c@{\:\:\,}l}
P(y\eq 0 \given x\eq 0) &=& 1 \,; \\
P(y\eq 1 \given x\eq 0) &=& 0 \,;
\end{array}
\begin{array}{c@{\:\:\,}c@{\:\:\,}l}
P(y\eq 0 \given x\eq {\query}) &=& 1/2 \,; \\
P(y\eq 1 \given x\eq {\query}) &=& 1/2 \,;
\end{array}
\begin{array}{c@{\:\:\,}c@{\:\:\,}l}
P(y\eq 0 \given x\eq 1) &=& 0 \,; \\
P(y\eq 1 \given x\eq 1) &=& 1 .
\end{array}
\]
Whenever the input $\mbox{\query}$ is used, the output is random;
the other inputs are reliable inputs. The maximum information
rate of 1 bit is achieved by making no use of the
input $\mbox{\query}$.
\end{description}
\exercissxB{2}{ex.Iternaryconfusion}{
Sketch the mutual information for this channel as a function of
% $$a\in (0,1)$ and $b\in (0,1)$,
the input distribution $\bp$. Pick a convenient two-dimensional
representation of $\bp$.
}
The \ind{optimization} routine must therefore take account
of the possibility that, as we go up hill on $\I(X;Y)$,
we may run into the inequality constraints $p_i \geq 0$.
\exercissxB{2}{ex.Imaximizer}{
Describe the condition, similar to \eqref{eq.Imaxer}, that is satisfied at a
point where $\I(X;Y)$ is maximized, and describe a computer
program for finding the capacity of a channel.
}
\subsection{Results that may help in finding the \optens}
% The following results
\ben
\item
{All outputs must be used}.
\item
{$\I(X,Y)$ is a \convexsmile\ function of the channel parameters.}\marginpar{\small\raggedright {\sf Reminder:} The term `\convexsmile' means `convex',
and the term `\concavefrown' means `concave'; the little
smile and frown symbols are included simply to remind you what
convex and concave mean.}
\item
{There may be several {\optens}s, but they all look the same at the output.}
\een
%\subsubsection{All outputs must be used\subsubpunc}
\exercisxB{2}{ex.Iallused}{
Prove that no output $y$ is unused by an \optens, unless it is unreachable,
that is, has $Q(y \given x)=0$ for all $x$.
}
%\subsubsection{Convexity of $\I(X,Y)$ with respect to the channel parameters\subsubpunc}
\exercisxC{2}{ex.Iconvex}{
Prove that
$\I(X,Y)$ is a \convexsmile\ function of $Q(y \given x)$.
}
%\subsubsection{There may be several {\optens}s, but they all look the same at the output\subsubpunc}
\exercisxC{2}{ex.Imultiple}{
Prove that all {\optens}s of a channel have the same output
probability distribution $P(y) = \sum_x P(x)Q(y \given x)$.
}
These results, along with the fact that
$\I(X;Y)$ is a \concavefrown\ function of
the input probability vector $\bp$, prove the validity of
the symmetry argument that we have used when finding
the capacity of symmetric channels.
If a channel is invariant under a group of symmetry
operations -- for example, interchanging the
input symbols and interchanging the output symbols --
then, given any \optens\ that is not
symmetric, \ie, is not invariant under these operations,
we can create another input distribution
by averaging together this \optens\ and all
%
% WORDY!!!!!!!!!!!
%
its permuted forms that we can make by applying the
symmetry operations to the original \optens.
The permuted distributions must have the same
$\I(X;Y)$ as the original, by symmetry, so the
new input distribution created by averaging must
have $\I(X;Y)$ bigger than or equal to that of the
original distribution, because of the concavity
of $\I$.
% see capacity.p
\subsection{Symmetric channels}
\label{sec.Symmetricchannels}
In order to use symmetry arguments, it will help
to have a definition of a symmetric channel.
I like \quotecite{Gallager68}
% Gallager's
definition.\index{Gallager, Robert}
% page 94
%\subsubsection{Gallager's definition of a symmetric channel}
\begin{description}
\item[A discrete memoryless channel is a symmetric channel]
if the set of outputs can be partitioned into subsets
in such a way that for each subset the matrix of
transition probabilities
% (using inputs as columns and outputs in the subset as rows)
has the property
that each row (if more than 1) is a permutation of each other row
and each column is a permutation of each other column.
\end{description}
\exampl{exSymmetric}{
This channel
\beq
\begin{array}{c@{\:\:\,}c@{\:\:\,}l}
P(y\eq 0 \given x\eq 0) &=& 0.7 \,; \\
P(y\eq {\query} \given x\eq 0) &=& 0.2 \,; \\
P(y\eq 1 \given x\eq 0) &=& 0.1 \,;
\end{array}
\begin{array}{c@{\:\:\,}c@{\:\:\,}l}
P(y\eq 0 \given x\eq 1) &=& 0.1 \,; \\
P(y\eq {\query} \given x\eq 1) &=& 0.2 \,; \\
P(y\eq 1 \given x\eq 1) &=& 0.7.
\end{array}
\eeq
is a symmetric channel because
its outputs can be partitioned into $(0,1)$ and ${\query}$, so that
the matrix can be rewritten:
\beq
\begin{array}{cc} \midrule
\begin{array}{ccl}%{c@{}c@{}l}
P(y\eq 0 \given x\eq 0) &=& 0.7 \,; \\
P(y\eq 1 \given x\eq 0) &=& 0.1 \,;
\end{array}
&
\begin{array}{ccl}%{c@{}c@{}l}
P(y\eq 0 \given x\eq 1) &=& 0.1 \,; \\
P(y\eq 1 \given x\eq 1) &=& 0.7 \,;
\end{array}
\\ \midrule
\begin{array}{ccl}%{c@{}c@{}l}
P(y\eq {\query} \given x\eq 0) &=& 0.2 \,; \\
\end{array}
&
\begin{array}{ccl}%{c@{}c@{}l}
P(y\eq {\query} \given x\eq 1) &=& 0.2 . \\
\end{array}
\\ \midrule
\end{array}
%
\eeq
}
Symmetry is a useful property because, as we will see
in a later chapter,
communication at capacity can be achieved over symmetric channels
by {\em{linear}\/} codes.\index{error-correcting code!linear}\index{linear block code}
% that are good codes
%-- a considerable simplification of the task of finding excellent codes.
\exercisxC{2}{ex.Symmetricoptens}{
Prove that for a \ind{symmetric channel} with any
number of inputs,\index{channel!symmetric}
the uniform distribution over the inputs is an {\optens}.
}
\exercissxB{2}{ex.notSymmetric}{
Are there channels that are not symmetric whose {\optens}s are uniform?
Find one, or prove there are none.
}
\section{Other coding theorems}% this star indicates skippable
\label{sec.othercodthm}
The noisy-channel coding theorem that we proved in this chapter
is quite general, applying to any discrete memoryless channel;
but it is not very specific. The theorem only says that
reliable communication with error probability $\epsilon$ and rate $R$
% can be achieved over a channel
can be achieved
by using codes with {\em sufficiently large\/} blocklength $N$.
The theorem does not say how large $N$ needs to be
% as a function
to achieve given values
of $R$ and $\epsilon$.
Presumably, the smaller $\epsilon$ is
and the closer $R$ is to $C$, the larger $N$ has to be.
% The task of proving explicit results about the blocklength
% is challenging and solutions to this problem are considerably
% more complex than the theorem we proved in this chapter.
%\begin{figure}
\marginfig{
\begin{center}
\mbox{\raisebox{0.5in}{$E_{\rm r}(R)$}\psfig{figure=figs/Er.eps,width=0.97in}}
\end{center}
\caption[a]{A typical random-coding exponent.}
\label{fig.Er}
%\end{figure}
}%\end{marginfig}
%
%
\subsection{Noisy-channel coding theorem -- version with
explicit $N$-dependence}
% explicit blocklength dependence}
\index{noisy-channel coding theorem}
\begin{quote}
For a discrete memoryless channel, a blocklength $N$
and a rate $R$, there exist block codes of length $N$
whose average probability of error satisfies:
\beq
p_{\rm B} \leq \exp \left[ -N E_{\rm r}(R) \right]
\label{eq.pbEr}
\eeq
where $E_{\rm r}(R)$ is the {\dem\ind{random-coding exponent}\/}
of the channel, a \convexsmile, decreasing, positive function of $R$
%which
% satisfies
%\beq
% E_{\rm r}(R) > 0 \:\: \mbox{for all $R$ satisfying $0 \leq R < C$} .
%\eeq
for $0 \leq R < C$. The {random-coding exponent}
is also known as the \ind{reliability function}.
[By an \ind{expurgation} argument it can also be shown that
there exist block codes for which the {\em{maximal\/}} probability of error
$p_{\rm BM}$
% , like $p_{\rm B}$ in \eqref{eq.pbEr},
is also exponentially small in $N$.]
\end{quote}
The definition of $E_{\rm r}(R)$ is
given in \citeasnoun{Gallager68}, p.$\,$139.
$E_{\rm r}(R)$ approaches
zero as $R \rightarrow C$; the typical behaviour of this function
is illustrated in \figref{fig.Er}.
The computation of the {random-coding exponent}
for interesting channels is a challenging task
on which much effort has been expended. Even for simple
channels like the \BSC, there is no simple expression for $E_{\rm r}(R)$.
\subsection{Lower bounds on the error probability as a function of blocklength}
The theorem stated above
% gives an upper bound on the error probability:
asserts that there are codes with $p_{\rm B}$ smaller than $\exp \left[ -N E_{\rm r}(R) \right]$.
But how small can the error probability be? Could it be much smaller?
\begin{quote}
For any code with blocklength $N$ on a discrete memoryless channel,
the probability of error assuming all source messages are
used with equal probability satisfies
\beq
p_{\rm B} \gtrsim \exp[ - N E_{\rm sp}(R) ] ,
\eeq
where the function $E_{\rm sp}(R)$,
the {\dem\ind{sphere-packing exponent}\/} of the channel,
is a \convexsmile, decreasing, positive function of $R$
for $0 \leq R < C$.
\end{quote}
For a precise statement of this result and further references,
see \citeasnoun{Gallager68}, \mbox{p.$\,$157}.\index{Gallager, Robert}
\section{Noisy-channel coding theorems and coding practice}
\label{sec.codthmpractice}
Imagine a customer who wants to buy an error-correcting
code and decoder for a noisy channel.
The results described above allow us to offer
the following service: if he tells us the properties of
his channel, the desired rate $R$ and the desired error probability $p_{\rm B}$,
we can, after working out the relevant functions
$C$, $E_{\rm r}(R)$, and $E_{\rm sp}(R)$, advise him
that there exists a solution to his problem using a particular
blocklength $N$; indeed that almost any randomly
chosen code with that blocklength
should do the job. Unfortunately we have
not found out how to implement these encoders
and decoders in practice; the cost of implementing
the encoder and decoder for a random code with large $N$ would
be exponentially large in $N$.
Furthermore, for practical purposes, the customer is unlikely
to know exactly what channel he is dealing with.
% and might be reluctant to specify a desired rate
So \citeasnoun{Berlekamp80} suggests that\index{Berlekamp, Elwyn}
the sensible way to approach error-correction
is to design encoding-decoding systems
and plot their performance on a {\em variety\/}
of idealized channels
as a function of the channel's noise level. These charts (one of which
is illustrated on page \pageref{fig:GCResults})
can then be shown to the customer, who can choose
among the systems on offer without having to
specify what he really thinks his channel is like.
With this attitude to the practical problem, the importance of the
functions $E_{\rm r}(R)$ and $E_{\rm sp}(R)$ is diminished.
%
% put this back somewhere. :
%
%
%\subsection{Noisy-channel coding theorem with errors allowed:
% rate-distortion theory}
% See Gallager p.466$\pm 20$.
%
%\subsection{Special case of linear codes}
% Give Gallager's p.94 definition of a discrete symmetric channel.
% Give coding theorem for linear codes on any symmetric channel
% (including with memory).
%
%\subsection{More general case of
% channels with memory}
%
%\subsection{Finite state channels}
% Channels with and without intersymbol interference and
% with and without noise. (Is it worth discussing these in any
% individual detail, or shall
% I just have a general channels with memory discussion?)
%
% end detour
\section{Further exercises}
\label{sec.codthmex}
\exercisaxA{2}{ex.exam01}{
A binary erasure channel with input $x$ and output $y$
has transition probability matrix:
\[
\bQ = \left[
\begin{array}{cc}
1-q & 0 \\
q & q \\
0 & 1-q
\end{array}
\right]
\hspace{1in}
\begin{array}{c}
\setlength{\unitlength}{0.13mm}
\begin{picture}(100,100)(0,0)
\put(18,0){\makebox(0,0)[r]{\tt 1}}
%
\put(18,80){\makebox(0,0)[r]{\tt 0}}
\put(20,0){\vector(1,0){38}}
\put(20,80){\vector(1,0){38}}
%
\put(20,0){\vector(1,1){38}}
\put(20,80){\vector(1,-1){38}}
%
\put(62,0){\makebox(0,0)[l]{\tt 1}}
\put(62,40){\makebox(0,0)[l]{\tt ?}}
\put(62,80){\makebox(0,0)[l]{\tt 0}}
\end{picture}
\end{array}
\]
Find the {\em{mutual information}\/} $I(X;Y)$ between the input and output
for general input distribution $\{ p_0,p_1 \}$, and show that the
{\em{capacity}\/} of this channel is $C = 1-q$ bits.
\medskip
\item
%\noindent (c)
A Z channel\index{channel!Z channel}
has transition probability matrix:
\[
\bQ = \left[
\begin{array}{cc}
1 & q \\
0 & 1-q
\end{array}
\right]
\hspace{1in}
\begin{array}{c}
\setlength{\unitlength}{0.1mm}
\begin{picture}(100,100)(0,0)
\put(18,0){\makebox(0,0)[r]{\tt 1}}
%
\put(18,80){\makebox(0,0)[r]{\tt 0}}
\put(20,0){\vector(1,0){38}}
\put(20,80){\vector(1,0){38}}
%
\put(20,0){\vector(1,2){38}}
%
\put(62,0){\makebox(0,0)[l]{\tt 1}}
\put(62,80){\makebox(0,0)[l]{\tt 0}}
\end{picture}
\end{array}
\]
Show that, using
a $(2,1)$ code,
% of blocklength 2,
{\bf{two}} uses of a Z channel
can be made to emulate {\bf{one}} use of an erasure channel,
and state the erasure probability of that erasure channel.
Hence show that the
capacity of the Z channel, $C_{\rm Z}$,
satisfies $C_{\rm Z} \geq \frac{1}{2}(1-q)$ bits.
Explain why the result $C_{\rm Z} \geq \frac{1}{2}(1-q)$
is an inequality rather than an equality.
}
\exercissxC{3}{ex.wirelabelling}{
A \ind{transatlantic} cable contains $N=20$ indistinguishable
electrical wires.\index{puzzle!transatlantic cable}\index{puzzle!cable labelling}
You have the job of figuring out which
wire is which, that is,
% Alice and Bob, located at the opposite ends of the
% cable, wish
to create a consistent labelling of the wires at each end.
Your only tools are the ability to connect wires to each other
in groups of two or more, and to test for connectedness with
a continuity tester.
What is the smallest number of transatlantic trips you need to
make, and how do you do it?
How would you solve the problem for larger $N$ such as $N=1000$?
As an illustration, if $N$ were 3 then the task can be solved
in two steps by labelling one wire at one end $a$, connecting the other two together,
crossing the \ind{Atlantic}, measuring which two wires are connected, labelling them
$b$ and $c$ and the unconnected one $a$, then connecting $b$ to $a$
and returning across the Atlantic, whereupon on disconnecting
$b$ from $c$, the identities of $b$ and $c$ can be deduced.
This problem can be solved by persistent search,
but the reason it is posed in this chapter is that it can
also be solved by a greedy approach based on maximizing the acquired
{\em information}.
Let the unknown permutation of wires be $x$.
% , drawn from an ensemble $X$.
Having chosen a set of connections of wires $\cal C$ at one end,
you can then make measurements at the other end, and
these measurements $y$ convey {\em information\/} about $x$.
How much? And for what set of connections is the information that $y$ conveys
about $x$ maximized?
}
\dvips
\section{Solutions}% to Chapter \protect\ref{ch6}'s exercises} % 80,82,84,85,86
% solutions to _l6.tex
%
%\soln{ex.m.s.I.aboveC}{
%%\input{tex/aboveC.tex}
% {\em [More work needed here.]}
%}
%\soln{ex.Iderivative}{
%% Find derivative of $I$ w.r.t $P(x)$.
% Get a specific mutual information
% like object minus $\log e$.
%}
%\soln{ex.Iconcave}{
% $\I(X,Y) = \sum_{x,y} P(x) Q(y|x) \log \frac{Q(y|x)}{P(x)Q(y|x)}$
% is a \concavefrown\ function of $P(x)$.
% Easy Proof in Gallager p.90, using \verb+z->x->y+, where $z$ chooses
% between the two things we are mixing.
% This satisfies $I(X;Y|Z) = 0$
% (data processing inequality).
%}
\soln{ex.Iternaryconfusion}%
{
\marginpar{\[
\begin{array}{c}
\setlength{\unitlength}{1mm}
\begin{picture}(20,30)(0,0)
\put(5,5){\vector(1,0){8}}
\put(5,25){\vector(1,0){8}}
\put(5,15){\vector(1,1){8}}
\put(5,15){\vector(1,-1){8}}
\put(10,18){\makebox(0,0)[l]{\dhalf}}
\put(10,12){\makebox(0,0)[l]{\dhalf}}
\put(4,5){\makebox(0,0)[r]{\tt1}}
\put(4,25){\makebox(0,0)[r]{\tt0}}
\put(16,5){\makebox(0,0)[l]{\tt1}}
\put(16,25){\makebox(0,0)[l]{\tt0}}
\put(4,15){\makebox(0,0)[r]{\tt{?}}}
\end{picture}
\end{array}
\]
}
If the input distribution is $\bp=(p_0,p_{\tt{?}},p_1)$,
the mutual information is
\beq
I(X;Y) = H(Y) - H(Y|X)
= H_2(p_0 + p_{{\tt{?}}}/2) - p_{{\tt{?}}} .
\eeq
We can build
a good sketch of this function in two ways:
by careful inspection of the function, or
by looking at special cases.
For the plots, the two-dimensional
representation of $\bp$ I will use
has $p_0$ and $p_1$ as the independent
variables, so that $\bp=(p_0,p_{\tt{?}},p_1) = (p_0,(1-p_0-p_1),p_1)$.
\medskip
\noindent {\sf By inspection.}
If we use the quantities $p_* \equiv p_0 + p_{{\tt{?}}}/2$ and $p_{\tt{?}}$
as our two degrees of freedom, the mutual information becomes
very simple: $I(X;Y) = H_2(p_*) - p_{{\tt{?}}}$. Converting back to
$p_0 = p_* - p_{{\tt{?}}}/2$ and $p_1 = 1 - p_* - p_{{\tt{?}}}/2$,
we obtain the sketch shown at the left below.
This function is like a tunnel rising up the direction of
increasing $p_0$ and $p_1$.
To obtain the required plot of $I(X;Y)$ we have to strip
away the parts of this tunnel that live outside
the feasible \ind{simplex} of
probabilities; we do this by redrawing the surface,
showing only the parts where $p_0>0$
and $p_1>0$. A full plot of the function is shown at the right.
\medskip
\begin{center}
\mbox{%
\hspace*{2.3in}%
\makebox[0in][r]{\raisebox{0.3in}{$p_0$}}%
\hspace*{-2.3in}%
\raisebox{0in}[1.9in]{\psfig{figure=figs/confusion.view1.ps,angle=-90,width=3.62in}}%
\hspace{-0.3in}%
\makebox[0in][r]{\raisebox{0.87in}{$p_1$}}%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\hspace*{2.3in}%
\makebox[0in][r]{\raisebox{0.3in}{$p_0$}}%
\hspace*{-2.3in}%
\raisebox{0in}[1.709in]{\psfig{figure=figs/confusion.view2.ps,angle=-90,width=3.62in}}%
\hspace{-0.3in}%
\makebox[0in][r]{\raisebox{0.87in}{$p_1$}}%
}\\[-0.3in]
\end{center}
\medskip
\noindent {\sf Special cases.}
In the special case $p_{{\tt{?}}}=0$, the channel is a noiseless
binary channel, and $I(X;Y) = H_2(p_0)$.
In the special case $p_0=p_1$, the term $H_2(p_0 + p_{{\tt{?}}}/2)$ is equal to 1,
so $I(X;Y) = 1-p_{{\tt{?}}}$.
In the special case $p_0=0$, the channel is a Z channel with error
probability 0.5. We know how to sketch that, from the previous chapter
(\figref{hxyz}).
\amarginfig{c}{\small% skeleton fixed Thu 10/7/03
\begin{center}% was -0.51in until Sat 24/5/03
\hspace*{-0.31in}\mbox{%
\hspace*{1.62in}%
\makebox[0in][r]{\raisebox{0.25in}{$p_0$}}%
\hspace*{-1.62in}%
{\psfig{figure=figs/confusion.skel.ps,angle=-90,width=2.5in}}%was 3in
\hspace{-0.3in}%
\makebox[0in][r]{\raisebox{0.77in}{$p_1$}}}\vspace{-0.2in}%
\end{center}
\caption[a]{Skeleton of the mutual information for the ternary confusion channel.}
\label{fig.skeleton}
}% end marginpar
These special cases allow us to construct the skeleton shown
in \figref{fig.skeleton}.
% below.
}
\soln{ex.Imaximizer}{
Necessary and sufficient conditions for $\bp$ to maximize
$\I(X;Y)$ are
\beq
\left.
\begin{array}{rclcc}
\frac{\partial \I(X;Y)}{\partial p_i} & =& \lambda & \mbox{and} & p_i>0 \\[0.05in]
\frac{\partial \I(X;Y)}{\partial p_i} & \leq & \lambda & \mbox{and} & p_i=0 \\
\end{array} \right\}
\:\:\: \mbox{for all $i$},
\label{eq.IequalsC}
\eeq
where $\lambda$ is a constant related to the capacity by $C = \lambda + \log_2 e$.
This result can be used in a computer program that evaluates the
derivatives, and increments and decrements the probabilities $p_i$
in proportion to the differences between those derivatives.
This result is also useful for lazy human capacity-finders
who are good guessers. Having guessed the \optens, one can
simply confirm that \eqref{eq.IequalsC} holds.
}
%\soln{ex.Iallused}{
% coming
%}
%\soln{ex.Iconvex}{
% Easy Proof, using \verb+(x,z)->y+.
%}
%\soln{ex.Imultiple}{
%% If there are several \optens, they all give the same
%% output probability (theorem). This is a general proof that
%% the `by symmetry' argument is valid.
% coming
%}
%\soln{ex.Symmetricoptens}{
% This can be proved by the symmetry argument given in the chapter.
%
% Alternatively see p.94 of Gallager.
%}
\soln{ex.notSymmetric}{
We certainly expect nonsymmetric channels with uniform {\optens}s to exist, since
when inventing a channel we have $I(J-1)$
degrees of freedom whereas
the \optens\ is just $(I-1)$-dimensional;
so in the $I(J\!-\!1)$-dimensional
space of perturbations around a symmetric channel,
we expect there to be a
subspace of perturbations of dimension
$I(J-1)-(I-1) = I(J-2)+1$
that leave the \optens\ unchanged.
Here is an explicit example, a bit like a Z channel.
\beq \bQ =
\left[
\begin{array}{cccc}
0.9585 & 0.0415 & 0.35 & 0.0 \\
0.0415 & 0.9585 & 0.0 & 0.35 \\
0 & 0 & 0.65 & 0 \\
0 & 0 & 0 & 0.65 \\
\end{array}
\right]
\eeq
}
% removed to cutsolutions.tex
% \soln{ex.exam01}{
\soln{ex.wirelabelling}{
The labelling problem can be solved for any $N>2$ with
just two trips, one each way across the Atlantic.
The key step in the information-theoretic approach to
this problem is to write down
the information content of
one {\dem\ind{partition}}, the combinatorial object that is the connecting
together of subsets of wires.
If $N$ wires are grouped together into
$g_1$ subsets of size $1$,
$g_2$ subsets of size $2$, $\ldots,$
% $g_r$ groups of size $r$ $\ldots,$
then the number of such partitions is
\beq
\Omega = \frac{ N! }{\displaystyle \prod_r \left( r! \right)^{g_r} g_r! } ,
\eeq
and the information content of one such \ind{partition} is the $\log$ of this quantity.
In a greedy strategy we choose the first partition to maximize this information
content.
One game we can play is to maximize this information content
with respect to the quantities $g_r$, treated as real numbers, subject to the
constraint $\sum_r g_r r = N$.
Introducing a \ind{Lagrange multiplier} $\l$ for the constraint,
the derivative is
\beq
\frac{ \partial }{\partial g_r} \left( \log \Omega + \l \sum_r g_r r \right)
= - \log r! - \log g_r + \l r ,
\eeq
which, when set to zero, leads to the rather nice expression
\beq
g_r = \frac{ e^{\l r} }{ r! } ;
% \:\:(r \geq 1)
\eeq
the optimal $g_r$ is
proportional to a \ind{Poisson distribution}\index{distribution!Poisson}!
We can solve for the Lagrange multiplier by plugging $g_r$ into the
constraint $\sum_r g_r r = N$, which gives the implicit
equation
\beq
N = \mu \, e^{\mu},
\eeq
where $\mu \equiv e^{\l}$ is a convenient reparameterization of the
Lagrange multiplier.
\Figref{fig.atlantic}a shows a graph of $\mu(N)$;
\figref{fig.atlantic}b
shows the deduced non-integer assignments $g_r$ when $\mu=2.2$,
and nearby integers $g_r = \{1,2,2,1,1\}$
that motivate setting the first partition to
(a)(bc)(de)(fgh)(ijk)(lmno)(pqrst).
\marginfig{\footnotesize
\begin{center}\hspace*{-0.2in}
\begin{tabular}{r@{\hspace{0.2in}}l}
(a)&\mbox{\psfig{figure=figs/atlanticmuN.ps,width=1.5in,angle=-90}}\\[0.2in]
(b)&\mbox{\psfig{figure=figs/atlanticpoi.ps,width=1.5in,angle=-90}}\\
\end{tabular}
\end{center}
\caption[a]{Approximate solution of the \index{cable labelling}{cable-labelling} problem
using Lagrange multipliers.
(a) The parameter $\mu$ as a function of $N$; the value $\mu(20) = 2.2$ is highlighted.
(b) Non-integer values of the function $g_r = \dfrac{ \mu^{r} }{ r! }$
are shown by lines and
integer values of $g_r$ motivated by those non-integer values are
shown by crosses.
}
\label{fig.atlantic}
}
This partition produces a random partition at the other
end, which has an information content of $\log \Omega =40.4\ubits$,
% pr log(20!*1.0/( (2!)**2 * 2 * (3!)**2 * 2 * (4!) * (5!) ) )/log(2.0)
% pr log(20!*1.0/( (2!)**10 * 10! ))/log(2.0)
which is a lot more than half the total information content
we need to acquire to infer the transatlantic permutation, $\log 20! \simeq 61\ubits$.
[In contrast, if all the wires are joined together in pairs,
the information content generated
is only about 29$\ubits$.]
How to choose the second partition is left
to the reader. A Shannonesque approach is appropriate, picking a
random partition at the other end, using the same $\{g_r\}$; you
need to ensure the two partitions are as unlike each other as possible.
If $N \neq 2$, 5 or 9, then the labelling problem
has solutions
that are particularly simple to implement,
called \ind{Knowlton--Graham partitions}:
partition $\{1,\ldots,N\}$ into disjoint sets in two ways $A$
% $A_1,\ldots,A_p$ and
and $B$,
% $B_1,\ldots,B_q$,
subject to the condition that at most one element appears
both in an $A$~set of cardinality~$j$ and in a $B$~set of cardinality~$k$, for
each $j$ and~$k$ \cite{Graham66,GrahamKnowlton68}.\index{Graham, Ronald L.}
% (R. L. Graham, ``On partitions of a finite set,'' {\sl Journal of Combinatorial
% Theory\/ \bf 1} (1966), 215--223;\index{Graham, Ronald L.}
% Ronald L. Graham and Kenneth C. Knowlton, ``Method of identifying conductors in
% a cable by establishing conductor connection groupings at both ends of the
% cable,'' U.S. Patent 3,369,177 (13~Feb 1968).)
}
%%%%%%%%%%%%%%%%%%%%%%%%
%
% end of chapter
%
%%%%%%%%%%%%%%%%%%%%%%%%
%
\dvipsb{solutions noisy channel s6}
%
%
% CHAPTER 12 (formerly 7)
%
\prechapter{About Chapter}
\fakesection{prerequisites for chapter 7}
Before reading \chref{ch.ecc}, you should have read Chapters
\ref{ch.five} and \ref{ch.six}.
You will also need to be familiar with the {\dem\inds{Gaussian distribution}}.
\label{sec.gaussian.props}
\begin{description}
\item[One-dimensional Gaussian distribution\puncspace]
If a
random variable $y$ is Gaussian and has mean $\mu$ and variance $\sigma^2$,
which we write:
\beq
y \sim \Normal(\mu,\sigma^2) ,\mbox{ or } P(y) = \Normal(y;\mu,\sigma^2) ,
\eeq
then the distribution of $y$ is:
% a Gaussian distribution:
\beq
P(y\given \mu,\sigma^2) = \frac{1}{\sqrt{2 \pi \sigma^2}}
\exp \left[ - ( y - \mu )^2 / 2 \sigma^2 \right] .
\eeq
[I use the symbol $P$ for both probability densities and
probabilities.]
The inverse-variance $\tau \equiv \dfrac{1}{\sigma^2}$ is sometimes
called the {\dem\inds{precision}\/} of the Gaussian distribution.
\item[Multi-dimensional Gaussian distribution\puncspace]
If $\by = (y_1,y_2,\ldots,y_N)$ has a \ind{multivariate Gaussian} {distribution}, then
\beq
P( \by \given \bx, \bA ) = \frac{1}{Z(\bA)} \exp \left( - \frac{1}{2}
(\by -\bx)^{\T} \bA (\by -\bx) \right) ,
\eeq
where $\bx$ is the mean of the distribution,
$\bA$ is the inverse of the \ind{variance--covariance matrix}\index{covariance matrix}, and
the normalizing constant is ${Z(\bA)} = \left( { {\det}\! \left( \linefrac{\bA}{2 \pi}
\right) } \right)^{-1/2}$.
This distribution has the property that
the variance $\Sigma_{ii}$ of $y_i$, and the covariance $\Sigma_{ij}$ of
$y_i$ and $y_j$
are given by
\beq
\Sigma_{ij} \equiv \Exp \left[ ( y_i - \bar{y}_i ) ( y_j - \bar{y}_j ) \right]
= A^{-1}_{ij} ,
\eeq
where $\bA^{-1}$ is the inverse of the matrix $\bA$.
The marginal distribution $P(y_i)$ of one component $y_i$ is Gaussian;
the joint marginal distribution of any subset of the
components is multivariate-Gaussian;
and the conditional density of any subset, given the values of
another subset, for example, $P(y_i\given y_j)$, is also Gaussian.
\end{description}
%\chapter{Error correcting codes \& real channels}
% ampersand used to keep the title on one line on the chapter's opening page
\ENDprechapter
\chapter[Error-Correcting Codes and Real Channels]{Error-Correcting Codes \& Real Channels}
\label{ch.ecc}\label{ch7}
% % : l7.tex -- was l78.tex
% \setcounter{chapter}{6}% set to previous value
% \setcounter{page}{70} % set to current value
% \setcounter{exercise_number}{89} % set to imminent value
% %
% \chapter{Error correcting codes \& real channels}
% \label{ch7}
The noisy-channel coding theorem that we have proved shows that there
exist reliable
% `very good'
error-correcting codes for any noisy channel.
In this chapter we address two questions.
First, many practical channels have real, rather than discrete,
inputs and outputs. What can Shannon tell us about
these continuous channels? And how should digital signals be
mapped into analogue waveforms, and {\em vice versa}?
Second, how are practical error-correcting codes
made, and what is achieved in practice, relative to the
possibilities proved by Shannon?
\section{The Gaussian channel}
The most popular model of a real-input, real-output
channel is the \inds{Gaussian channel}.\index{channel!Gaussian}
\begin{description}
\item[The Gaussian channel] has a real input $x$ and a real output $y$.
The conditional distribution of $y$ given $x$ is a Gaussian distribution:
\beq
P(y\given x) = \frac{1}{\sqrt{2 \pi \sigma^2}}
\exp \left[ - ( y - x )^2 / 2 \sigma^2 \right] .
\label{eq.gaussian.channel.def}
\eeq
%
This channel has a continuous input and output but is discrete
in time.
We will show below that certain continuous-time channels
are equivalent to the discrete-time Gaussian channel.
This channel is sometimes called the additive white Gaussian noise (AWGN)
channel.\index{channel!AWGN}\index{channel!Gaussian}\index{AWGN}
\end{description}
% Why is this a useful channel model? And w
As with discrete channels, we will discuss
what rate of error-free
information communication can be achieved over this channel.
\subsection{Motivation
% for the Gaussian channel
in terms of a continuous-time channel \nonexaminable}
Consider a physical (electrical, say) channel with inputs and outputs that
are continuous in time. We put in $x(t)$,
% which is a
%% some sort of
% band-limited signal,
and out comes $y(t) = x(t) + n(t)$.
Our transmission has a power cost. The average power of
a transmission of length $T$ may be constrained thus:
\beq
\int_0^T \d t \: [x(t)]^2 / T \leq P .
\eeq
The received signal is assumed to differ from $x(t)$ by additive
noise $n(t)$ (for example \ind{Johnson noise}), which we will model as
white\index{white noise}\index{noise!white}
Gaussian noise. The magnitude of this noise is quantified by the
{\dem noise spectral
density}, $N_0$.\index{noise!spectral density}\index{E$_{\rm b}/N_0$}\index{signal-to-noise ratio}
% , which might depend on the effective temperature of the system.
How could such a channel be used to communicate information?
\amarginfig{t}{
\begin{tabular}{r@{}l}
$\phi_1(t)$&\raisebox{-0.8cm}{\psfig{figure=figs/realchannel/phi1.ps,angle=-90,width=1in}}\\
$\phi_2(t)$&\raisebox{-0.8cm}{\psfig{figure=figs/realchannel/phi2.ps,angle=-90,width=1in}}\\
$\phi_3(t)$&\raisebox{-0.8cm}{\psfig{figure=figs/realchannel/phi3.ps,angle=-90,width=1in}}\\
$x(t)$ &\raisebox{-0.8cm}{\psfig{figure=figs/realchannel/xt.ps,angle=-90,width=1in}}\\
\end{tabular}
%
\caption[a]{Three basis functions, and a
weighted combination of them,
$
x(t) = \sum_{n=1}^N x_n \phi_n(t) ,
$
with $x_1 \eq 0.4$,
$x_2 \eq -0.2$, and $x_3 \eq 0.1$.
% see figs/realchannel.gnu
}
\label{fig.continuousfunctionexample}
}
Consider transmitting a set of $N$ real numbers $\{ x_n \}_{n=1}^N$
in a signal of duration $T$ made up of a weighted combination
of orthonormal basis functions $\phi_n(t)$,
\beq
x(t) = \sum_{n=1}^N x_n \phi_n(t) ,
\eeq
where $\int_0^T \: \d t \: \phi_n(t) \phi_m(t) = \delta_{nm}$.
The receiver can then compute the scalars:
\beqan
y_n \:\: \equiv \:\: \int_0^T \: \d t \: \phi_n(t) y(t)
&=&
x_n + \int_0^T \: \d t \: \phi_n(t) n(t)
\\
&\equiv& x_n + n_n
\eeqan
for $n=1 \ldots N$.
If there were no noise, then $y_n$ would equal $x_n$. The white Gaussian
noise $n(t)$ adds scalar noise $n_n$ to the estimate $y_n$. This noise
is Gaussian:
\beq
n_n \sim \Normal(0,N_0/2),
\eeq
where $N_0$ is the spectral
density introduced above.
% [This is the definition of $N_0$.]
Thus a continuous channel used in this way
is equivalent to the Gaussian channel
defined at \eqref{eq.gaussian.channel.def}.
The power constraint $\int_0^T \d t \, [x(t)]^2 \leq P T$
defines a constraint on the signal amplitudes $x_n$,
\beq
\sum_n x_n^2 \leq PT \hspace{0.5in} \Rightarrow
\hspace{0.5in}
\overline{x_n^2} \leq \frac{PT}{N} .
\eeq
Before returning to the Gaussian channel, we define the {\dbf\ind{bandwidth}}
(measured in \ind{Hertz})
of the \ind{continuous channel}\index{channel!continuous} to be:
\beq
W = \frac{N^{\max}}{2 T},
\eeq
where $N^{\max}$ is the maximum number of orthonormal functions that can be
produced in an interval of length $T$.
This definition can be motivated by imagining creating a
\ind{band-limited signal} of duration $T$ from orthonormal cosine and sine
curves of maximum frequency $W$. The number of orthonormal functions
is $N^{\max} = 2 W T$. This definition relates to the
\ind{Nyquist sampling theorem}: if the highest frequency present in a signal
is $W$, then the signal can
be fully determined from its values at a series of discrete
sample points separated by the Nyquist interval
$\Delta t = \dfrac{1}{2W}$ seconds.
So the use of a real continuous channel with bandwidth $W$, noise spectral
density $N_0$ and power $P$ is equivalent to $N/T = 2 W$ uses per second
of a Gaussian channel with noise level
$\sigma^2 = N_0/2$ and subject to the signal power
constraint $\overline{x_n^2} \leq\dfrac{P}{2W}$.
\subsection{Definition of $E_{\rm b}/N_0$\nonexaminable}
Imagine\index{E$_{\rm b}/N_0$}
that the Gaussian channel $y_n = x_n + n_n$ is used {with
% an
% error-correcting code
an encoding system} to transmit {\em binary\/}
source bits at a rate of $R$ bits per channel use.
% , where a rate of 1 corresponds to the uncoded case.
How can we compare two encoding systems that have different
rates of \ind{communication} $R$ and that use different powers $\overline{x_n^2}$?
Transmitting at a large rate $R$ is good; using small power is
good too.
It is conventional to measure the rate-compensated
\ind{signal-to-noise ratio}
% \marginpar{\footnotesize{I'm using signal to noise ratio in two different ways. Elsewhere it is defined to be $\frac{\overline{x_n^2}}{\sigma^2}$. Should I modify this phrase?}}
by the ratio of the power per source bit $E_{\rm b} = \overline{x_n^2}/R$
to the noise spectral density $N_0$:\marginpar[t]{\small\raggedright
{$E_{\rm b}/N_0$ is dimensionless, but it is usually reported in the units
of \ind{decibels}; the value given is $10 \log_{10} E_{\rm b}/N_0$.}}
\beq
E_{\rm b}/N_0 = \frac{\overline{x_n^2}}{2 \sigma^2 R} .
\eeq
% This signal-to-noise measure equates low rate, low power
% cf ebno.p
% The difference in
$E_{\rm b}/N_0$ is one of the
measures used to compare coding schemes
for Gaussian channels.
\section{Inferring the input to a real channel}
\subsection{`The best detection of pulses'}
\label{sec.pulse}
In 1944
Shannon
wrote a memorandum \cite{shannon44} on the
problem of best differentiating between two types of pulses of known shape,
represented by vectors $\bx_0$ and $\bx_1$, given that one of them has
been transmitted over a noisy channel. This is a
\ind{pattern recognition} problem.%
\amarginfig{t}{
\begin{tabular}{r@{}l}
$\bx_0$&\raisebox{-0.8cm}{\psfig{figure=figs/realchannel/x0.ps,angle=-90,width=1in}}\\
$\bx_1$&\raisebox{-0.8cm}{\psfig{figure=figs/realchannel/x1.ps,angle=-90,width=1in}}\\
$\by$&\raisebox{-0.8cm}{\psfig{figure=figs/realchannel/xn1.ps,angle=-90,width=1in}}\\
\end{tabular}
%
\caption[a]{Two pulses $\bx_0$ and $\bx_1$, represented
as 31-dimensional vectors, and
a noisy version of one of them, $\by$.
% see figs/realchannel.gnu
}
\label{fig.detectionofpulses}
}
It is assumed that the noise is Gaussian with probability density
\beq
P( \bn ) = \left[ {\det}\left( \frac{\bA}{2 \pi}
\right) \right]^{1/2} \exp \left( - \frac{1}{2}
\bn^{\T} \bA \bn \right) ,
\eeq
where $\bA$ is the inverse of the variance--covariance matrix of the
noise, a symmetric and positive-definite matrix.
(If $\bA$ is a multiple of the identity matrix, $\bI/\sigma^2$,
then the noise is `white'.\index{noise!white}\index{white noise}
For more general $\bA$, the
noise is \index{noise!coloured}\index{coloured noise}`{coloured}'.) The probability of the received vector $\by$ given that the
source signal was $s$ (either zero or one) is then
\beq
P( \by \given s ) = \left[ { {\det} \left( \frac{\bA}{2 \pi}
\right) }\right]^{1/2} \exp \left( - \frac{1}{2}
(\by -\bx_s)^{\T} \bA (\by -\bx_s) \right) .
\eeq
The optimal detector is based on the posterior probability ratio:
\beqan
\hspace{-0.6cm}
\lefteqn{\frac{ P( s \eq 1\given \by )}{P(s \eq 0\given \by )} =
\frac{ P( \by \given s \eq 1 ) }{ P( \by \given s \eq 0)}
\frac{ P( s \eq 1 )}{P(s \eq 0 )} }
\\
&=& \exp \left( - \frac{1}{2}
(\by -\bx_1)^{\T} \bA (\by -\bx_1) + \frac{1}{2}
(\by -\bx_0)^{\T} \bA (\by -\bx_0) + \ln \frac{ P( s \eq 1 )}{P(s \eq 0 )} \right)
\nonumber
\\ &=& \exp \left( \by^{\T} \bA ( \bx_1 -\bx_0) + \theta \right),
\eeqan
where $\theta$ is a constant independent of the received vector $\by$,
\beq
\theta = - \frac{1}{2}
\bx_1^{\T} \bA \bx_1 + \frac{1}{2}
\bx_0^{\T} \bA \bx_0 + \ln \frac{ P( s \eq 1 )}{P(s \eq 0 )} .
\eeq
If the detector is forced to make a decision (\ie, guess either
$s \eq 1$ or $s \eq 0$) then the
decision that minimizes the probability of error is
to guess the most probable hypothesis. We can write the
optimal decision in terms of a {\dem\ind{discriminant function}}:
\beq
a(\by) \equiv \by^{\T} \bA ( \bx_1 -\bx_0) + \theta
\eeq
with the decisions
\marginfig{
\begin{tabular}{r@{}l}
$\bw$&\raisebox{-0.8cm}{\psfig{figure=figs/realchannel/w.ps,angle=-90,width=1in}}\\
\end{tabular}
%
\caption[a]{The weight vector $\bw \propto \bx_1 -\bx_0$
that is used to discriminate between $\bx_0$ and $\bx_1$.
% see figs/realchannel.gnu
}
\label{fig.detectionofpulses.w}
}
\beq
\begin{array}{ccl} a(\by) > 0& \rightarrow & \mbox{guess $s \eq 1$} \\
a(\by) < 0& \rightarrow & \mbox{guess $s \eq 0$} \\
a(\by)=0 & \rightarrow & \mbox{guess either.}
\end{array}
\eeq
Notice
% It should be noted
that $a(\by)$ is a linear function of the
received vector,
\beq
a(\by) = \bw^{\T} \by + \theta ,
\eeq
where $\bw \equiv \bA ( \bx_1 -\bx_0)$.
\section{Capacity of Gaussian channel}
\label{sec.entropy.continuous}
Until now we have only measured the joint, marginal, and conditional
entropy of discrete variables. In order to define the information conveyed
by continuous variables, there are two issues we must
address -- the infinite length of the real line, and the infinite
precision of real numbers.
\subsection{Infinite inputs}
How much information can we convey in one use of a Gaussian
channel? If we are allowed to put {\em any\/} real number $x$ into the
Gaussian channel, we could communicate an enormous
string of $N$ digits $d_1d_2d_3\ldots d_N$
by setting $x = d_1d_2d_3\ldots d_N 000\ldots 000$.
The amount of
error-free information conveyed in just a single transmission could
be made arbitrarily large by increasing $N$,
and the communication could be made arbitrarily reliable
by increasing the number of zeroes at the end of $x$.
There is usually some \ind{power cost} associated
with large inputs, however, not to mention practical limits
in the dynamic range acceptable to a receiver.
It is therefore conventional to introduce a {\dem\ind{cost
function}\/} $v(x)$ for every input $x$, and constrain codes to have
an average cost $\bar{v}$ less than or equal to some maximum value.
% a maximum average cost $\bar{v}$.
A generalized channel coding theorem, including a cost
function for the inputs, can be proved
% for the discrete channels discussed previously
-- see McEliece (1977).\nocite{McEliece77}
The result is a channel
capacity $C(\bar{v})$ that is a function of the permitted cost. For
the Gaussian channel we will assume a cost
\beq
v(x) = x^2
\eeq
such that the `average power' $\overline{x^2}$ of the input is
constrained. We motivated this cost function
above in the case of real electrical channels in
which the physical power consumption is indeed quadratic in $x$.
The constraint $\overline{x^2}=\bar{v}$ makes it impossible to
communicate infinite information in one use of the Gaussian channel.
\subsection{Infinite precision}
\amarginfig{b}{
{\footnotesize\setlength{\unitlength}{1mm}
\begin{tabular}{lc}
(a)&{\psfig{figure=gnu/grainI.ps,angle=-90,width=1.3in}}\\
(b)&\makebox[0in]{\hspace*{4mm}\begin{picture}(20,10)%
\put(17.65,6){\vector(1,0){1.42}}
\put(17.65,6){\vector(-1,0){1.42}}
\put(17.5,8){\makebox(0,0){$g$}}
%
\end{picture}}%
{\psfig{figure=gnu/grain10.ps,angle=-90,width=1.3in}}\\
&{\psfig{figure=gnu/grain18.ps,angle=-90,width=1.3in}}\\
&{\psfig{figure=gnu/grain34.ps,angle=-90,width=1.3in}}\\
& $\vdots$ \\
\end{tabular}
}
%
\caption[a]{(a) A probability density $P(x)$. {\sf Question:}
can we define the `entropy' of this density?
(b) We could evaluate the entropies of
a sequence of probability distributions with
decreasing grain-size $g$, but these entropies tend to
$\displaystyle \int P(x) \log \frac{1}{ P(x) g } \, \d x$,
which is not independent of $g$:
% increases as $g$ decreases:
the entropy goes up by
one bit for every halving of $g$.
$\displaystyle \int P(x) \log \frac{1}{ P(x) } \, \d x$
is an\index{sermon!illegal integral}
% \\ \hspace
illegal integral.}
% see gnu/grain.gnu
\label{fig.grain}
}
It is tempting to define joint, marginal, and conditional entropies\index{entropy!of continuous variable}\index{grain size}
for real variables simply by replacing summations by integrals, but
this is not a well defined operation. As we discretize an interval
into smaller and smaller divisions, the entropy of the discrete
distribution diverges (as the logarithm of the granularity) (\figref{fig.grain}).
Also, it is not permissible
to take the logarithm of a dimensional quantity such as
a probability density $P(x)$ (whose dimensions are $[x]^{-1}$).\index{sermon!dimensions}\index{dimensions}
There is one information measure, however, that has a well-behaved
limit, namely the mutual information -- and this is the one that
really matters, since it measures how much information one variable
conveys about another. In the discrete case,
\beq
\I(X;Y) = \sum_{x,y}
P(x,y) \log \frac{P(x,y)}{P(x)P(y)} .
\eeq
Now because the argument of the log is a ratio of two probabilities
over the same space, it is
OK to have $P(x,y)$, $P(x)$ and $P(y)$ be
probability densities
% (as long as they are not pathological)
% densities)
and replace the sum by an integral:
\beqan
\I(X;Y)& =& \int \! \d x \: \d y \:
P(x,y) \log \frac{P(x,y)}{P(x)P(y)}
\\ &=&
\int \! \d x \: \d y \:
P(x)P(y\given x) \log \frac{P(y\given x)}{P(y)} .
\eeqan
We can now ask these questions for the Gaussian channel:
(a) what probability distribution
$P(x)$ maximizes the mutual information (subject to the constraint
$\overline{x^2}={v}$)? and (b) does the maximal
mutual information still measure the maximum
error-free communication rate of this real channel,
as it did for the discrete channel?
\exercissxD{3}{ex.gcoptens}{
Prove that the probability distribution
$P(x)$ that maximizes the mutual information (subject to the constraint
$\overline{x^2}={v}$) is a Gaussian distribution of mean zero
and variance $v$.
}
% solution is in tex/sol_gc.tex
\exercissxB{2}{ex.gcC}{
%
Show that the
mutual information $\I(X;Y)$,
in the case of this optimized distribution, is
\beq
C = \frac{1}{2} \log \left( 1 + \frac{v}{\sigma^2} \right) .
\eeq
}
This is an important result. We see that the capacity of the Gaussian
channel is a function of the {\dem signal-to-noise ratio} $v/\sigma^2$.
\subsection{Inferences given a Gaussian input distribution}
If
$
P(x) = \Normal(x;0,v) \mbox{ and } P(y\given x) = \Normal(y;x,\sigma^2)
$
then the marginal distribution of $y$ is
$
P(y) = \Normal(y;0,v\!+\!\sigma^2)
$
and the posterior distribution of the input, given that the output is $y$,
is:
\beqan
P(x\given y) &\!\!\propto\!\!&
P(y\given x)P(x)
\\
&\!\!\propto\!\!& \exp( -(y-x)^2/2 \sigma^2)
\exp( -x^2/2 v)
\label{eq.two.gaussians}
\\
&\!\! =\!\! &
\Normal\left( x ; \frac{ v}{v+\sigma^2} \, y \, , \,
\left({\frac{1}{v}+\frac{1}{\sigma^2}}\right)^{\! -1} \right) .
\label{eq.infer.mean.gaussian}
\eeqan
%
% label this bit for reference when we get to Gaussian land
[The step from (\ref{eq.two.gaussians}) to (\ref{eq.infer.mean.gaussian})
is made by completing the square in the exponent.]
This
\label{sec.infer.mean.gaussian}
formula deserves careful study. The mean of the posterior
distribution, $\frac{ v}{v+\sigma^2} \, y $, can be viewed
as a weighted combination of the value that best fits the
output, $x=y$, and the value that best fits the prior, $x=0$:
\beq
\frac{ v}{v+\sigma^2} \, y =
\frac{1/\sigma^2 }{1/v+1/\sigma^2} \, y + \frac{1/v}{1/v+1/\sigma^2} \, 0 .
\eeq
The weights $1/\sigma^2$ and $1/v$ are the {\dem\ind{precision}s\/}
% parameters'
of the two Gaussians that we multiplied together in \eqref{eq.two.gaussians}:
the prior and the likelihood.
%-- the probability of the output given the input,
% and the prior probability of the input.
The precision of the posterior distribution is
the sum of these two precisions. This is a general property:
whenever two independent sources contribute information, via
Gaussian distributions, about an unknown variable, the\index{precisions add}
precisions add. [This is the dual to the better-known
relationship `when independent variables are added,
their variances add'.]\index{variances add}
% inverse-variances add to define the inverse-variance of the
% posterior distribution.
\subsection{Noisy-channel coding theorem for the Gaussian channel}
We\index{noisy-channel coding theorem!Gaussian channel}
have evaluated a maximal mutual information. Does it correspond
to a maximum possible rate of error-free information transmission?
One way of proving that this is so
is to define a sequence
of discrete channels, all derived from the Gaussian channel, with
increasing numbers of inputs and outputs, and prove that the maximum
mutual information of these channels tends to the asserted $C$.
The noisy-channel coding theorem for discrete channels applies
to each of these derived channels, thus we obtain a coding theorem for
the continuous channel.
% coding theorem is then proved.
% (with discrete inputs and
% discrete outputs) by chopping the output into bins and using a
% finite set of inputs, and then defining a sequence of such channels with
% increasing numbers of inputs and outputs. A proof that the maximum
% mutual information
% of these channels tends to $C$ then completes the job, as we have already
% proved the noisy channel coding theorem for discrete channels.
%
% A more intuitive argument for the coding theorem may be preferred.
Alternatively, we can make an intuitive argument for the coding theorem
specific for the Gaussian channel.
\subsection{Geometrical view of the noisy-channel coding theorem: sphere packing}
\index{sphere packing}Consider a sequence $\bx = (x_1,\ldots, x_N)$ of inputs, and the
corresponding output $\by$, as defining two points in an $N$ dimensional
space. For large $N$, the noise power is very likely to be close
(fractionally) to $N \sigma^2$. The output $\by$ is therefore very likely
to be close to the surface of a sphere of radius $\sqrt{ N \sigma^2}$
centred on $\bx$. Similarly, if the original signal $\bx$ is generated
at random subject to an average power constraint $\overline{x^2} = v$,
then $\bx$ is likely to lie close to a sphere, centred on the
origin, of radius $\sqrt{N v}$; and because the total average power of $\by$
is $v+\sigma^2$, the received signal $\by$ is likely to lie on the surface
of a sphere of radius $\sqrt{N (v+\sigma^2)}$, centred on the origin.
The volume of an $N$-dimensional sphere of radius $r$ is
%
% this also appeared in _s1.tex
%
\beq
\textstyle
V(r,N) = \smallfrac{ \pi^{N/2} }{ \Gamma( N/2 + 1 ) } r^N .
\eeq
Now consider making a communication system based on non-confusable
inputs $\bx$, that is, inputs whose spheres do not overlap significantly.
The maximum number $S$ of non-confusable inputs is given by dividing
the volume of the sphere of probable $\by$s by the volume of
the sphere for $\by$ given $\bx$:
%
% An upper bound for the number $S$ of non-confusable inputs is:
\beq
S \leq \left( \frac{ \sqrt{N (v+\sigma^2)} }{ \sqrt{ N \sigma^2} }
\right)^{\! N}
\eeq
Thus the capacity is bounded by:\index{capacity!Gaussian channel}
\beq
C = \frac{1}{N} \log M \leq \frac{1}{2} \log \left( 1 + \frac{v}{\sigma^2}
\right) .
\eeq
A more detailed argument
% using the law of large numbers
like the one used in the previous chapter
can establish equality.
\subsection{Back to the continuous channel}
Recall that
the use of a real continuous channel with bandwidth $W$, noise spectral
density $N_0$ and power $P$ is equivalent to $N/T = 2 W$ uses per second
of a Gaussian channel with $\sigma^2 = N_0/2$ and subject to the
constraint $\overline{x_n^2} \leq P/2W$.
Substituting the result for the capacity of the Gaussian channel, we find the
capacity of the continuous channel to be:
\beq
C = W \log \left( 1 + \frac{P}{N_0 W} \right) \: \mbox{ bits per second.}
\eeq
This formula gives insight into the tradeoffs of practical
\ind{communication}. Imagine that we have a fixed power constraint. What
is the best \ind{bandwidth} to make use of that power? Introducing
$W_0=P/N_0$, \ie, the bandwidth for which the signal-to-noise ratio
is 1, figure \ref{fig.wideband} shows $C/W_0 = W/W_0 \log \! \left( 1 + W_0/W
\right)$ as a function of $W/W_0$. The capacity increases to an
asymptote of $W_0 \log e$. It is dramatically better (in terms of capacity
for fixed power) to transmit at a
low signal-to-noise ratio over a large bandwidth, than with high
signal-to-noise in a narrow bandwidth; this is one motivation for wideband
communication methods such as the `direct sequence spread-spectrum'\index{spread spectrum}
approach used
in {3G} \ind{mobile phone}s. Of course, you are not alone,
and your electromagnetic neighbours
may not be pleased if you use a large bandwidth, so for social reasons,
engineers often have to make do with higher-power, narrow-bandwidth
transmitters.
%\begin{figure}
%\figuremargin{%
\marginfig{
% figs: load 'wideband.com'
\begin{center}
\mbox{\psfig{figure=figs/wideband.ps,%
width=1.75in,angle=-90}}
\end{center}
%}{%
\caption[a]{Capacity versus bandwidth for a real channel:
$C/W_0 = W/W_0 \log \left( 1 + W_0/W
\right)$ as a function of $W/W_0$.}
\label{fig.wideband}
}%
%\end{figure}
\section{What are the capabilities of practical error-correcting codes?\nonexaminable}
\label{sec.bad.code.def}% see also {sec.good.codes}!
% cf also \ref{sec.bad.dist.def}
% in _linear.tex
% Description of Established Codes}
%
Nearly all codes are good, but nearly all codes require exponential look-up
tables for practical
implementation of the encoder and decoder -- exponential in the
blocklength $N$. And the coding theorem required $N$ to be large.
By a {\dem\ind{practical}\/} error-correcting code, we mean one that
can be encoded and decoded in a reasonable amount of time,
for example, a time that scales as a polynomial function
of the blocklength $N$ -- preferably linearly.
\subsection{The Shannon limit is not achieved in practice}
The non-constructive proof of the noisy-channel coding theorem showed
that good block codes exist for any noisy channel, and indeed that nearly
all block codes are good. But writing down an explicit and {practical\/}
encoder
and decoder that are as good as promised by Shannon is still an unsolved
problem.
% Most of the explicit families of codes that have been written down have the
% property that they can achieve a vanishing error probability $p_{\rm b}$
% as $N \rightarrow \infty$ only if the rate $R$ also goes to zero.
%
% There is one exception to this statement:
% , given by a family of codes based on
% {\dbf concatentation}.
\label{sec.good.codes}
\begin{description}
\item[Very good codes\puncspace]
Given a channel, a family of block\index{error-correcting code!very good}
codes that achieve arbitrarily small
probability of error
at any communication rate
up to the capacity
of the channel are called `very good' codes
for that channel.
\item[Good codes]
are code families that
achieve arbitrarily small probability of error
at non-zero communication rates
up to some maximum rate
that may be {\em less than\/} the \ind{capacity}
of the given channel.\index{error-correcting code!good}
\item[Bad codes] are code families that cannot achieve arbitrarily small
probability of error, or that
can only achieve arbitrarily small
probability of error\index{error-correcting code!bad}
% $\epsilon$ `bad'
by decreasing the information rate
% $R$
to zero.
Repetition codes\index{error-correcting code!repetition}\index{repetition code}%
\index{error-correcting code!bad}
are an example of a bad code family.
(Bad codes are not necessarily useless for practical
purposes.)
\item[Practical codes] are code families that can be\index{error-correcting code!practical}
encoded and decoded in time and space polynomial in the blocklength.
\end{description}
\subsection{Most established codes are linear codes}
Let us review the definition of a block code, and then add
the definition of a linear block
code.\index{error-correcting code!block code}\index{error-correcting code!linear}\index{linear block code}
\begin{description}
\item[An $(N,K)$ block code] for a channel $Q$ is a list of $\cwM=2^K$
codewords
$\{ \bx^{(1)}, \bx^{(2)}, \ldots, \bx^{({2^K)}} \}$, each of length $N$:
$\bx^{(\cwm)} \in \A_X^N$.
The signal to be encoded, $\cwm$, which comes from an
alphabet of size $2^K$, is encoded as $\bx^{(\cwm)}$.
% The {\dbf\ind{rate}} of the code\index{error-correcting code!rate} is $R = K/N$ bits.
%
% [This definition holds for any channels, not only binary channels.]
\item[A linear $(N,K)$ block code] is a block code in which
the codewords $\{ \bx^{(\cwm)} \}$ make up a $K$-dimensional subspace of
$\A_X^N$. The encoding operation can be represented by an $N \times K$
binary matrix\index{generator matrix}
$\bG^{\T}$ such that if the signal to be encoded,
in binary notation, is $\bs$ (a vector of length $K$ bits), then the
encoded signal is $\bt = \bG^{\T} \bs \mbox{ modulo } 2$.
The codewords $\{ \bt \}$ can be defined as the set of vectors
satisfying $\bH \bt = {\bf 0} \mod 2$, where $\bH$ is the
{\dem\ind{parity-check matrix}\/}
of the code.
\end{description}
\marginpar[c]{\[%beq
\bG^{\T} = {\small \left[ \begin{array}{@{\,}*{4}{c@{\,}}}
1 & \cdot & \cdot & \cdot \\[-0.05in]
\cdot & 1 & \cdot & \cdot \\[-0.05in]
\cdot & \cdot & 1 & \cdot \\[-0.05in]
\cdot & \cdot & \cdot & 1 \\[-0.05in]
1 & 1 & 1 & \cdot \\[-0.05in]
\cdot & 1 & 1 & 1 \\[-0.05in]
1 & \cdot & 1 & 1 \end{array} \right] } % nb different from l1.tex, no longer
\]%eeq
}
For example the
$(7,4)$ \ind{Hamming code} of section \ref{sec.ham74}
takes $K=4$ signal bits, $\bs$, and transmits
them followed by three parity-check bits. The $N=7$ transmitted
symbols are given by $\bG^{\T} \bs \mod 2$.
% , where:
Coding theory was born with the work of Hamming, who invented a
family of practical
error-correcting codes, each able to correct one error in a
block of length $N$, of which the repetition code $R_3$ and the
$(7,4)$ code are the simplest.
Since then most established codes have been
generalizations of Hamming's codes:
% `BCH' (Bose, Chaudhury and Hocquenhem)
Bose--Chaudhury--Hocquenhem
% The search for decodeable codes has produced the following families.
codes, Reed--M\"uller codes, Reed--Solomon codes, and
Goppa codes, to name a few.
\subsection{Convolutional codes}
Another family of linear codes are {\dem\ind{convolutional code}s}, which
do not divide the source stream into blocks, but instead read
and\index{error-correcting code!convolutional}
transmit bits continuously. The transmitted bits
are a linear function of the past source bits.
% both bits and parity checks in some fixed proportion.
Usually the rule for generating the transmitted bits
% parity checks
involves feeding the present source bit
into a \lfsr\index{linear feedback shift register} of length $k$,
and transmitting one or more
linear functions of the state of the shift register
at each iteration.
The resulting transmitted bit stream
is
%can be thought of as
the convolution
of the source stream with a linear filter.
The impulse-response function of this filter may have finite
or infinite duration, depending on the choice of feedback shift-register.
% it is
We will discuss convolutional codes in \chapterref{ch.convol}.
\subsection{Are linear codes `good'?}
One might ask, is the reason that the Shannon limit is not achieved
in practice
because linear codes are inherently not\index{error-correcting code!linear}\index{error-correcting code!good}\index{error-correcting code!random}
as good as random codes?\index{random code} The answer is no, the noisy-channel coding theorem
can still be proved for linear codes, at least for some channels
(see \chapterref{ch.linear.good}),
though the proofs, like Shannon's
proof for random codes, are non-constructive.
%(We will prove that
% there exist linear codes that are very good codes
% in chapter \ref{ch.linear.good}.
% and in particular for `cyclic codes',
% a class to which BCH and Reed--Solomon codes belong.
Linear codes are easy to implement at the encoding end. Is decoding a
linear code also easy? Not necessarily. The general decoding problem\index{error-correcting code!decoding}\index{linear block code!decoding}
(find the maximum likelihood $\bs$ in the equation $\bG^{\T} \bs + \bn =
\br$) is in fact \inds{NP-complete} \cite{BMT78}. [NP-complete problems are
computational problems that are all equally difficult and which
are widely believed to require exponential
computer time to solve in general.] So attention focuses on families of codes
% (such as those listed above)
for which there is a fast decoding algorithm.
\subsection{Concatenation}
One trick for building codes with practical decoders
is the idea of {concatenation}.\index{error-correcting code!concatenated}\index{concatenation!error-correcting codes}
An\amarginfignocaption{t}{
\begin{center}
\setlength{\unitlength}{1mm}
\begin{picture}(25,10)%
\put(17.5,8){\makebox(0,0){$\C' \rightarrow \underbrace{\C \rightarrow Q \rightarrow \D}
\rightarrow \D'$}}
\put(17.5,3){\makebox(0,0){$Q'$}}
%
\end{picture}%
\end{center}
%\caption[a]{none}
}
encoder--channel--decoder system $\C \rightarrow Q \rightarrow \D$
can be viewed as defining a \ind{super-channel} $Q'$
with a smaller probability of error, and with complex\index{channel!complex}
correlations among
its errors. We can create an encoder $\C'$ and decoder $\D'$
for this super-channel $Q'$.
The code consisting of the outer code $\C'$ followed by the inner code $\C$ is known
as a {\dem{concatenated code}}.\index{concatenation!error-correcting codes}
Some concatenated codes make use of the idea of {\dbf
\ind{interleaving}}. We read
% Interleaving involves encoding
the data in blocks, the size of each block being larger than the
blocklengths of the constituent codes $\C$ and $\C'$.
After encoding the data of one block using code $\C'$, the bits
are reordered within the block in such a way that
nearby bits are separated from each other once the block is fed to
the second code $\C$. A simple example of an interleaver
is a {\dbf\ind{rectangular code}\/}
or\index{error-correcting code!rectangular}\index{error-correcting code!product code}
{\dem\ind{product code}\/} in which the data are arranged in a
$K_2 \times K_1$ block, and encoded horizontally using an
$(N_1,K_1)$
linear code, then vertically using a $(N_2,K_2)$ linear code.
\exercisaxB{3}{ex.productorder}{
Show that either of the two codes can be viewed as the \ind{inner code} or the
\ind{outer code}.
}
%\subsection{}
% see also _concat2.tex
As an example, \figref{fig.concath1}
shows a product code in which we
% encode horizontally
% For example, if we
encode first with the repetition code $\Rthree$ (also known
as the \ind{Hamming code} $H(3,1)$)
horizontally then with $H(7,4)$
vertically.
The blocklength of the
concatenated\index{concatenation} code is 27. The number of source bits per codeword is
four, shown by the small rectangle.
% The code would be equivalent if we
% encoded first with $H(7,4)$ and second with $\Rthree$.
\begin{figure}
\figuremargin{%
\setlength{\unitlength}{0.4mm}
\begin{center}
\begin{tabular}{rrrrr}
(a)
\begin{picture}(30,70)(0,0)
\put(0,0){\framebox(30,70)}
\put(0,30){\framebox(10,40)}
\put(5,65){\makebox(0,0){1}}
\put(5,55){\makebox(0,0){0}}
\put(5,45){\makebox(0,0){1}}
\put(5,35){\makebox(0,0){1}}
\put(5,25){\makebox(0,0){0}}
\put(5,15){\makebox(0,0){0}}
\put(5,5){\makebox(0,0){1}}
\put(15,65){\makebox(0,0){1}}
\put(15,55){\makebox(0,0){0}}
\put(15,45){\makebox(0,0){1}}
\put(15,35){\makebox(0,0){1}}
\put(15,25){\makebox(0,0){0}}
\put(15,15){\makebox(0,0){0}}
\put(15,5){\makebox(0,0){1}}
\put(25,65){\makebox(0,0){1}}
\put(25,55){\makebox(0,0){0}}
\put(25,45){\makebox(0,0){1}}
\put(25,35){\makebox(0,0){1}}
\put(25,25){\makebox(0,0){0}}
\put(25,15){\makebox(0,0){0}}
\put(25,5){\makebox(0,0){1}}
\end{picture}&
%
% noise picture
%
(b)
\begin{picture}(30,70)(0,0)
\put(0,0){\framebox(30,70)}
\put(0,30){\framebox(10,40)}
\put(5,55){\makebox(0,0){$\star$}}%
\put(5,15){\makebox(0,0){$\star$}}%
%
\put(15,55){\makebox(0,0){$\star$}}%
\put(15,35){\makebox(0,0){$\star$}}%
%
\put(25,25){\makebox(0,0){$\star$}}%
\end{picture}&
%
% received vector picture
%
(c)
\begin{picture}(30,70)(0,0)
\put(0,0){\framebox(30,70)}
\put(0,30){\framebox(10,40)}
\put(5,65){\makebox(0,0){1}}
\put(5,55){\makebox(0,0){1}}%
\put(5,45){\makebox(0,0){1}}
\put(5,35){\makebox(0,0){1}}
\put(5,25){\makebox(0,0){0}}
\put(5,15){\makebox(0,0){1}}%
\put(5,5){\makebox(0,0){1}}
%
\put(15,65){\makebox(0,0){1}}
\put(15,55){\makebox(0,0){1}}%
\put(15,45){\makebox(0,0){1}}
\put(15,35){\makebox(0,0){0}}%
\put(15,25){\makebox(0,0){0}}
\put(15,15){\makebox(0,0){0}}
\put(15,5){\makebox(0,0){1}}
%
\put(25,65){\makebox(0,0){1}}
\put(25,55){\makebox(0,0){0}}
\put(25,45){\makebox(0,0){1}}
\put(25,35){\makebox(0,0){1}}
\put(25,25){\makebox(0,0){1}}%
\put(25,15){\makebox(0,0){0}}
\put(25,5){\makebox(0,0){1}}
\end{picture} &
% after R3 correction
(d)
\begin{picture}(30,70)(0,0)
\put(0,0){\framebox(30,70)}
\put(0,30){\framebox(10,40)}
\put(5,65){\makebox(0,0){1}}
\put(5,55){\makebox(0,0){1}}%
\put(5,45){\makebox(0,0){1}}
\put(5,35){\makebox(0,0){1}}
\put(5,25){\makebox(0,0){0}}
\put(5,15){\makebox(0,0){{\bf 0}}}%
\put(5,5){\makebox(0,0){1}}
%
\put(15,65){\makebox(0,0){1}}
\put(15,55){\makebox(0,0){1}}%
\put(15,45){\makebox(0,0){1}}
\put(15,35){\makebox(0,0){{\bf 1}}}%
\put(15,25){\makebox(0,0){0}}
\put(15,15){\makebox(0,0){0}}
\put(15,5){\makebox(0,0){1}}
%
\put(25,65){\makebox(0,0){1}}
\put(25,55){\makebox(0,0){{\bf 1}}}
\put(25,45){\makebox(0,0){1}}
\put(25,35){\makebox(0,0){1}}
\put(25,25){\makebox(0,0){{\bf 0}}}%
\put(25,15){\makebox(0,0){0}}
\put(25,5){\makebox(0,0){1}}
\end{picture}&
% after 74 correction
(e)
\begin{picture}(30,70)(0,0)
\put(0,0){\framebox(30,70)}
\put(0,30){\framebox(10,40)}
\put(5,65){\makebox(0,0){1}}
\put(5,55){\makebox(0,0){{\bf 0}}}%
\put(5,45){\makebox(0,0){1}}
\put(5,35){\makebox(0,0){1}}
\put(5,25){\makebox(0,0){0}}
\put(5,15){\makebox(0,0){{0}}}%
\put(5,5){\makebox(0,0){1}}
%
\put(15,65){\makebox(0,0){1}}
\put(15,55){\makebox(0,0){{\bf 0}}}%
\put(15,45){\makebox(0,0){1}}
\put(15,35){\makebox(0,0){{1}}}%
\put(15,25){\makebox(0,0){0}}
\put(15,15){\makebox(0,0){0}}
\put(15,5){\makebox(0,0){1}}
%
\put(25,65){\makebox(0,0){1}}
\put(25,55){\makebox(0,0){{\bf 0}}}
\put(25,45){\makebox(0,0){1}}
\put(25,35){\makebox(0,0){1}}
\put(25,25){\makebox(0,0){{0}}}%
\put(25,15){\makebox(0,0){0}}
\put(25,5){\makebox(0,0){1}}
\end{picture}\\
&
%
% noise picture
%
&
&
% after 74 correction
(d$^{\prime}$)
\begin{picture}(30,70)(0,0)
\put(0,0){\framebox(30,70)}
\put(0,30){\framebox(10,40)}
\put(5,65){\makebox(0,0){1}}
\put(5,55){\makebox(0,0){1}}%
\put(5,45){\makebox(0,0){1}}
\put(5,35){\makebox(0,0){1}}
\put(5,25){\makebox(0,0){{\bf 1}}}%
\put(5,15){\makebox(0,0){1}}%
\put(5,5){\makebox(0,0){1}}
%
\put(15,65){\makebox(0,0){{\bf 0}}}%
\put(15,55){\makebox(0,0){1}}%
\put(15,45){\makebox(0,0){1}}
\put(15,35){\makebox(0,0){0}}%
\put(15,25){\makebox(0,0){0}}
\put(15,15){\makebox(0,0){0}}
\put(15,5){\makebox(0,0){1}}
%
\put(25,65){\makebox(0,0){1}}
\put(25,55){\makebox(0,0){0}}
\put(25,45){\makebox(0,0){1}}
\put(25,35){\makebox(0,0){1}}
\put(25,25){\makebox(0,0){{\bf 0}}}%
\put(25,15){\makebox(0,0){0}}
\put(25,5){\makebox(0,0){1}}
\end{picture} &
% after R3 correction
(e$^{\prime}$)
\begin{picture}(30,70)(0,0)
\put(0,0){\framebox(30,70)}
\put(0,30){\framebox(10,40)}
\put(5,65){\makebox(0,0){1}}
\put(5,55){\makebox(0,0){(1)}}
\put(5,45){\makebox(0,0){1}}
\put(5,35){\makebox(0,0){1}}
\put(5,25){\makebox(0,0){{\bf 0}}}
\put(5,15){\makebox(0,0){{\bf 0}}}%
\put(5,5){\makebox(0,0){1}}
%
\put(15,65){\makebox(0,0){{\bf 1}}}
\put(15,55){\makebox(0,0){(1)}}%
\put(15,45){\makebox(0,0){1}}
\put(15,35){\makebox(0,0){{\bf 1}}}%
\put(15,25){\makebox(0,0){0}}
\put(15,15){\makebox(0,0){0}}
\put(15,5){\makebox(0,0){1}}
%
\put(25,65){\makebox(0,0){1}}
\put(25,55){\makebox(0,0){(1)}}
\put(25,45){\makebox(0,0){1}}
\put(25,35){\makebox(0,0){1}}
\put(25,25){\makebox(0,0){{0}}}%
\put(25,15){\makebox(0,0){0}}
\put(25,5){\makebox(0,0){1}}
\end{picture}\\
\end{tabular}
\end{center}
}{%
\caption[a]{A product code.
(a) A string {\tt{1011}} encoded using a concatenated code
consisting of two Hamming codes, $H(3,1)$ and $H(7,4)$.
(b) a noise pattern that flips 5 bits. (c) The received vector.
(d) After decoding using the horizontal $(3,1)$ decoder,
and (e) after subsequently using the vertical $(7,4)$ decoder. The decoded
vector matches the original.
(d$^{\prime}$, e$^{\prime}$) After decoding in the other order, three errors
still remain.}
\label{fig.concath1}
}%
\end{figure}
\label{sec.concatdecode}We
can decode conveniently (though not optimally) by using the
individual decoders for each of the subcodes in some sequence.
It makes most sense to first decode the code which has the
lowest rate and hence the greatest error-correcting ability.
\Figref{fig.concath1}(c--e) shows what happens if we receive the
codeword of \figref{fig.concath1}a with some errors (five bits flipped,
as shown) and
apply the decoder for $H(3,1)$ first, and then the
decoder for $H(7,4)$. The first decoder corrects three of the errors,
but erroneously modifies the third bit in the second row where there
are two bit errors. The $(7,4)$ decoder can then correct all three of
these errors.
\Figref{fig.concath1}(d$^{\prime}$--$\,$e$^{\prime}$) shows what happens if we decode the two codes
in the other order.
In columns one and two there are two errors, so the $(7,4)$ decoder
introduces two extra errors. It corrects the one error in column 3.
The $(3,1)$ decoder then cleans up four of the errors, but erroneously
infers the second bit.
% To make simple decoding possible,
% we split up bits that are in a single codeword at the first level,
% grouping them with other bits. Rectangular arrangement makes this easiest
% to see.
\subsection{Interleaving}
The motivation for interleaving is that by spreading out bits that
are nearby in one code, we make it possible to ignore
% forget about
the complex correlations among the errors that are produced by the
inner code. Maybe the inner code will mess up an entire codeword;
but that codeword is spread out one bit at a time over several codewords
of the outer code. So we can treat the errors introduced by the
inner code as if they are independent.\index{approximation!of complex distribution}
% by a simpler one}
%
% By iterating this process, with each successive
% code adding a small amount of redundancy to a geometrically increasing block,
% we can define an explicit sequence of codes with the property that
% $p_{\rm b} \rightarrow 0$ for some rate $R > 0$ (but not any $R$ up to the
% capacity $C$).
%
% There is also a proof by Forney that better concatenations
% exist, which achieve rates up to capacity and have encoding and decoding
% complexity of order $O(N^4)$. But the proof is non-constructive.
%
% gf.tex could be included here
%
% \subsection{Coding theory sells you short}
% At this point could discuss the universalist `this code corrects
% all errors up to $t$' with the Shannonist `the prob of error
% is tiny'. The latter attitude allows you to communicate at far
% greater rates. The former attitude is happy with something that
% is only halfway.
%
% Distance
% Show Prob of error of ideal decoder (Schematic) as function of noise level.
% Show that you can cope with double the noise.
\subsection{Other channel models}
% Most of the codes mentioned above are designed in terms of
In addition to the binary
symmetric channel and the Gaussian channel,
% or in terms of the number of errors they can correct, but
coding theorists keep more complex
channels in mind also.
%\index{burst-error channels}
{\dem Burst-error channels\/}\index{channel!bursty}\index{burst errors}
are important models in
practice. \ind{Reed--Solomon code}s use \ind{Galois field}s
(see \appendixref{app.GF})
with large numbers of
elements (\eg\ $2^{16}$) as their input alphabets,
and thereby automatically achieve a degree
of burst-error tolerance in that even if 17 successive bits are
corrupted, only 2 successive symbols in the Galois field representation are
corrupted. Concatenation and interleaving can give further
% fortuitous
protection against
% \index{concatenated code}
burst errors. The concatenated\index{concatenation!error-correcting codes}\index{error-correcting code!concatenated}
Reed--Solomon codes used on digital compact discs
% DISKS?
are able to correct bursts of errors of length 4000 bits.
\exercissxB{2}{ex.interleaving.dumb}{
The technique of \ind{interleaving},\index{implicit assumptions}
which allows bursts of\index{error-correcting code!interleaving}
errors to be treated as independent, is widely used, but is theoretically
a poor way to protect data against
\ind{burst errors}, in terms of the amount of redundancy required.
Explain why interleaving is a poor method, using the following
burst-error channel as an example. Time is divided into chunks
of length
$N=100$ clock cycles; during each chunk, there is a burst with
probability $b=0.2$; during a burst, the channel is a binary symmetric channel
with $f=0.5$. If there is no burst, the channel is an error-free binary
channel. Compute the capacity of this channel and compare it with the
maximum communication rate that could conceivably be achieved if one
used interleaving and treated the errors as independent.
}
% The BSC is an inadequate channel model for a second reason: many
% channels have {\em real outputs}. For example, a
% binary input $x$ may give rise to a
% probability distribution over a real output $y$. Codes whose decoders
% can handle real outputs (log likelihood ratios) are therefore
% important. `Convolutional codes' are such codes, as are some block codes.
{\dem\index{fading channel}{Fading channels}\/} are real\index{channel!fading}
channels like Gaussian\index{channel!Gaussian}
channels except that the received
power is assumed to vary with time.
A moving
\ind{mobile phone}\index{cellphone|see{mobile phone}}\index{phone!cellular|see{mobile phone}}
is an important example.
The incoming \ind{radio} signal is reflected
off nearby objects so that there are interference patterns and the
intensity of the
signal received by the phone varies with its location. The received power
can easily vary by 10 decibels\index{decibel}
(a factor of ten) as the phone's antenna
moves through a distance similar to the wavelength of the radio signal
(a few centimetres).
%Fading channels are used as models
% of the radio channel of mobile phones, in which the received power
% varies rapidly
\section{The state of the art}
What are the best known codes for communicating over Gaussian channels?
All the practical codes are linear codes, and are either
based on convolutional codes or block codes.\index{linear block code}
\subsection{Convolutional codes, and codes based on them}
\begin{description}
\item[Textbook convolutional codes\puncspace] The `de facto standard'
% cite golomb?
error-correcting code for\index{communication}
\ind{satellite communications} is a
convolutional code with constraint length 7.
Convolutional codes are discussed in \chref{ch.convol}.
\item[Concatenated convolutional codes\puncspace]
The above \ind{convolutional code}
can be used as the inner code of a\index{error-correcting code!concatenated}
concatenated code whose
outer code
is a \ind{Reed--Solomon code} with eight-bit symbols. This code
was used in deep space communication systems such as the
Voyager spacecraft.
For further reading about Reed--Solomon codes,
see \citeasnoun{lincostello83}.
\item[The code for \index{Galileo code}{Galileo}\puncspace]
A code using the same format but using a longer
constraint length -- 15 -- for its convolutional code and a larger
Reed--Solomon code was developed by the \ind{Jet Propulsion Laboratory} \cite{JPLcode}.
The details of this code are unpublished outside JPL,
and the decoding is only possible using
a room full of special-purpose hardware.
In 1992, this
was the best code known of rate \dfrac{1}{4}.
\item[Turbo codes\puncspace]
In 1993, \index{Berrou, C.}{Berrou}, \index{Glavieux, A.}{Glavieux}
and \index{Thitimajshima, P.}{Thitimajshima}
\nocite{Berrou93:Turbo}reported
work on {\dem\ind{turbo code}s}. The encoder of a turbo code is based on
the encoders of two
% or more constituent codes. In
% the original paper the two constituent codes were
convolutional codes.
The source bits are fed into each encoder, the order of the
source bits being permuted in a random way, and the resulting
parity bits from each constituent code are transmitted.
The decoding algorithm
% invented by Berrou {\em et al\/}
involves iteratively decoding each constituent code%
\amarginfig{b}{
\begin{center}
\setlength{\unitlength}{1mm}
\begin{picture}(25,30)(0,8)%
\put(15,18){\framebox(8,8){$C_1$}}
\put(15, 8){\framebox(8,8){$C_2$}}
\put( 9,12){\circle{6}}
\put( 5, 8){\framebox(8,8){$\pi$}}
\put(9.7,14.875){\vector(1,0){0.1}}% right pointing circle vector % was 975
\put(23,22){\vector(1,0){3}}
\put(23,12){\vector(1,0){3}}
\put(13,12){\line(1,0){2}}
\put( 2,12){\vector(1,0){3}}
\put( 0,22){\vector(1,0){15}}
\put( 2,22){\line(0,-1){10}}
%
\end{picture}%
\end{center}
\caption[a]{The encoder of a turbo code.
Each box $C_1$, $C_2$, contains a convolutional code.
The source bits are reordered using a permutation $\pi$ before
they are fed to $C_2$. The transmitted codeword is obtained
by concatenating or interleaving the outputs of the two
convolutional codes.
The random
permutation is chosen when the code is designed, and fixed
thereafter.
}
}
using its
standard decoding algorithm, then using the
output of the decoder as the input to the other decoder.
This decoding algorithm is an instance of a
{\dbf{message-passing}}\index{message passing}
algorithm called the {\dbf\ind{sum--product algorithm}}.
Turbo codes are discussed in \chref{ch.turbo}, and message passing in Chapters \ref{ch.message},
\ref{ch.noiseless}, \ref{ch.exact}, and \ref{ch.sumproduct}.
\end{description}
\subsection{Block codes}
\begin{description}
\item[Gallager's low-density parity-check codes\puncspace]
The%
\amarginfig{c}{
\[
\raisebox{0.425in}{ \bH \hspace{0.02in} =}\hspace{-0.1in}
\psfig{figure=MNCfigs/12.4.3.111/A.ps,angle=-90,width=1.5in,height=1in}
\]
\begin{center}
\mbox{
\psfig{figure=/home/mackay/itp/figs/gallager/16.12.ps,width=2in,angle=-90}
}\end{center}
\caption[a]{A low-density parity-check matrix
and the corresponding graph of a rate-\dfrac{1}{4}
low-density parity-check code with
% $(j,k) = (3,4)$,
blocklength $N \eq 16$, and $M \eq 12$ constraints.
Each white circle represents a transmitted bit. Each bit
participates in $j=3$ constraints, represented by
\plusnode\ squares. Each
% \plusnode\
constraint forces the
sum of the $k=4$ bits to which it is connected to
be even.
This code is a $(16,4)$
code. Outstanding performance is obtained when
the blocklength is increased to
$N \simeq 10\,000$.
}
\label{fig.ldpccIntro}
}
best block codes known for Gaussian channels
were invented by Gallager\index{Gallager, Robert} in
1962 but were promptly forgotten by most of the coding theory community.
% by MacKay and Neal,
They were rediscovered in 1995\nocite{mncEL,wiberg:phd}\index{Wiberg, Niclas}\index{MacKay, David}\index{error-correcting code!low-density parity-check}\index{Neal, Radford}
and shown to have outstanding theoretical and practical properties.\index{error-correcting code!practical}
Like turbo codes, they are decoded by message-passing algorithms.
We will discuss these beautifully simple codes in Chapter
% \ref{ch.belief.propagation} and
\ref{ch.gallager}.
\end{description}
The performances of the above codes are compared for Gaussian
channels in \figref{fig:GCResults}, \pref{fig:GCResults}.%{fig.gl.gc}.
% the Galileo code and
% Only the Galileo code and turbo codes outperform the original
% regular, binary Gallager codes.
% The best known Gallager codes, which are irregular,
%% and non-binary,
% outperform the Galileo code and turbo codes too \cite{DaveyMacKay96,Richardson2001b}.
\section{Summary}
\begin{description}
\item[Random codes] are good, but they require exponential resources to encode
and decode them.
\item[Non-random codes] tend for the most part not to be as good as
random codes. For a non-random code, encoding may be easy, but even for
simply-defined linear codes, the decoding problem remains very difficult.
\item[The best practical codes]
%\ben
%\item
(a)
employ very large block sizes; (b)
% \item
are based on semi-random code constructions;
and (c)
%\item
make use of probability-based decoding algorithms.
% \een
\end{description}
\section{Nonlinear codes}
Most practically used codes are linear, but not all.\index{error-correcting code!nonlinear}\index{nonlinear code}
Digital soundtracks are encoded onto cinema film as
a binary pattern. The likely errors affecting the
film involve
dirt and scratches, which produce large numbers of {\tt{1}}s
and {\tt{0}}s respectively. We want none
of the codewords to look like all-{\tt{1}}s or all-{\tt{0}}s,
so that it will be
easy to detect errors caused by dirt and scratches.
One of the codes used in \ind{digital cinema}\index{cinema} \ind{sound} systems is
a nonlinear $(8,6)$ code consisting of 64 of the ${{8}\choose{4}}$
binary patterns of weight 4.
% That's 70 patterns. Pick 64.
\section{Errors other than noise}
Another source of uncertainty for the receiver
is uncertainty about the {\em{\ind{timing}}\/} of the transmitted signal $x(t)$.
In ordinary coding theory and information theory,
the transmitter's time $t$ and the receiver's time
$u$ are assumed to be perfectly synchronized.
% If a bit sequence is encoded by a simple signal
% $x(t) \in \pm 1$, information is easily conveyed if
% the transmitter and the receiver both know the same
% time $t$;
But if the receiver receives a signal $y(u)$,
where the receiver's time, $u$, is an imperfectly
known function $u(t)$
of the transmitter's time
$t$, then the capacity of this channel for communication
is reduced. The theory of
such channels is incomplete, compared
with the
% ordinary
% `normal'
synchronized channels\index{insertions}\index{deletions}
we have discussed thus far. Not even
the {\em capacity\/} of channels with \ind{synchronization errors}\index{capacity!channel with synchronization errors}
is known \cite{Levenshtein66,Ferreira97};
%
% ear recommends citing zigangirov69 ullman67
%
codes for reliable communication over channels
with synchronization errors remain an active research area
\cite{DaveyMacKay99b}.
% ear recommends citing ratzer2003
\subsection*{Further reading}
For a review of the history of spread-spectrum\index{spread spectrum} methods, see
\citeasnoun{Scholtz82}.
\section{Exercises}
\subsection{The Gaussian channel}
\exercissxB{2}{ex.gcCb}{
Consider a Gaussian channel with a real input $x$, and signal to
noise ratio $v/\sigma^2$.
\ben
\item
What is its capacity $C$?
\item
If the input is constrained to be binary, $x \in \{ \pm \sqrt{v} \}$,
what is the capacity $C'$ of this constrained channel?
\item
If in addition the output of the channel is thresholded using the
mapping
\beq
y \rightarrow y' = \left\{ \begin{array}{cc} 1 & y > 0 \\
0 & y \leq 0, \end{array}
\right.
\eeq
what is the capacity $C''$ of the resulting channel?
\item
Plot the three capacities above as a function of $v/\sigma^2$
from 0.1 to 2. [You'll need to do a numerical integral to
evaluate $C'$.]
\een
}
\exercisaxB{3}{ex.codeslinear}{
For large integers $K$ and $N$,
what fraction of all binary error-correcting codes of length $N$
and rate $R=K/N$ are {\em{linear}\/} codes?
[The answer will depend on whether you choose to define the
code to be an {\em{ordered}\/} list of $2^K$ codewords,
that is, a mapping from $s \in \{1,2,\ldots,2^K\}$ to $\bx^{(s)}$,
or to define the code to be an unordered list, so that
two codes consisting of the same codewords are identical.
Use the latter definition: a code\index{error-correcting code} is a set of
codewords; how the encoder operates is not part of the
definition of the code.]
}
% that have not already been covered.
\subsection{Erasure channels}
\exercisxB{4}{ex.beccode}{
Design a code for the binary erasure channel, and a decoding
algorithm, and evaluate their probability of error.
[The design of good codes for erasure channels\index{erasure-correction}\index{channel!erasure}
is an active research area
\cite{spielman-96,LubyDF}; see also \chref{chdfountain}.]
% Have fun!]
%
}
\exercisaxB{5}{ex.qeccode}{
Design a code for the $q$-ary erasure channel,\index{erasure-correction}
whose input $x$ is drawn from $0,1,2,3,\ldots,(q-1)$,
and whose output $y$ is equal to $x$ with probability $(1-f)$
and equal to {\tt{?}} otherwise.
[This erasure channel is a good model for \ind{packet}s
transmitted over the \ind{internet}, which are either received reliably
or are lost.]
}
\exercissxC{3}{ex.raid}{
How do redundant arrays of independent disks (RAID) work?\marginpar{%
\small\raggedright{%
% aside
[Some people say RAID stands for `redundant array of inexpensive disks',
but I think that's silly -- RAID would still be a good idea\index{RAID}\index{redundant array of independent disks}
even if the disks were expensive!]
% end aside
}}
These are information storage systems consisting of about\index{erasure-correction}
ten \disc{} drives,\index{disk drive} of which any two or three can be disabled and the others
are able to still able to reconstruct any requested file.\index{file storage}
What codes are used, and how far are these systems from the Shannon
limit for the problem they are solving? How would {\em you\/} design
a better RAID system?
%
Some information is provided in the solution section.
See {\tt http://{\breakhere}www.{\breakhere}acnc.{\breakhere}com/{\breakhere}raid2.html}; see also \chref{chdfountain}.
% and {\tt http://www.digitalfountain.com/} for more.
}
%%%%\input{tex/_e7.tex}
\dvips
\section{Solutions}% to Chapter \protect\ref{ch.ecc}'s exercises} %
% ex 89
\soln{ex.gcoptens}{
% \subsection{Maximization}
Introduce a Lagrange multiplier $\l$ for the power constraint and another,
$\mu$, for the constraint of normalization of $P(x)$.
\beqan
F &\eq & \I(X;Y) -
{ \l \textstyle \int \d x \, P(x) x^2 - \mu \textstyle \int \d x \, P(x) }
\\ &\eq &
\int \! \d x \,
P(x) \left[ \int \! \d y \, P(y\given x) \ln \frac{P(y\given x)}{P(y)}
- \l x^2 - \mu \right] .
\eeqan
Make the functional derivative with respect to
$P(x^*)$.
\beqan
\frac{\delta F}{\delta P(x^*)} &=&
\int \! \d y \, P(y\given x^*) \ln \frac{P(y\given x^*)}{P(y)}
- \l {x^*}^2 - \mu
\nonumber \\ &&
- \int \! \d x \: P(x)
\int \! \d y \: P(y\given x) \frac{1}{P(y)} \frac{\delta P(y)}{\delta P(x^*)} . \hspace{0.5cm}
\eeqan
The final factor $\delta P(y)/\delta P(x^*)$ is found, using $P(y) =
\int \! \d x \, P(x) P(y\given x)$, to be $P(y\given x^*)$, and the whole of the
last term collapses in a puff of smoke to 1, which can be absorbed into the
$\mu$ term.
% We now substitute
Substitute
$P(y\given x) = \exp( -(y-x)^2/2 \sigma^2) / \sqrt{2 \pi \sigma^2}$
and set the derivative to zero:
\beq
\int \! \d y \, P(y\given x) \ln \frac{P(y\given x)}{P(y)} - \l x^2 - \mu' = 0
\eeq
\beq
\Rightarrow
\int \! \d y \,
\frac{\exp( -(y-x)^2/2 \sigma^2)}{\sqrt{2 \pi \sigma^2} }
\ln \left[ P(y) \sigma \right] = - \l x^2 - \mu' - \frac{1}{2} .
\label{eq.theconstr}
\eeq
This condition must
be satisfied by $\ln \! \left[ P(y) \sigma \right]$ for all $x$.
Writing a Taylor expansion of $\ln \! \left[ P(y) \sigma \right]
= a + b y + c y^2 + \cdots$, only a quadratic function
$\ln \! \left[ P(y) \sigma \right]
= a + c y^2$ would satisfy the constraint (\ref{eq.theconstr}).
(Any higher order terms $y^p$, $p>2$, would produce
terms in $x^p$ that are not present on the right-hand side.)
Therefore $P(y)$ is Gaussian. We can obtain this optimal output distribution
by using a Gaussian input distribution $P(x)$.
% \footnote{Note in passing that
% the Gaussian is the probability distribution that has maximum
% pseudo-entropy
}
\soln{ex.gcC}{
Given a Gaussian input distribution of variance $v$, the
output distribution is $\Normal(0,v\!+\!\sigma^2)$, since
$x$ and the noise are independent random variables,
and variances add for independent random variables.
The mutual information is:
\beqan
\!\!\!\!\!\!\!\!\!\! \I(X;Y)& =& \!\! \int \! \d x \, \d y \:
P(x)P(y\given x) \log {P(y\given x)}
- \int \! \d y \:
P(y) \log {P(y)} \\
&=&
\frac{1}{2} \log \frac{1}{\sigma^2} - \frac{1}{2} \log \frac{1}{v+\sigma^2} \\
&=&
% \frac{1}{2} \log \frac{v+\sigma^2}{\sigma^2} =
\frac{1}{2} \log \left( 1 + \frac{v}{\sigma^2}
\right) .
\eeqan
}
\soln{ex.interleaving.dumb}{
The capacity of the channel is one minus the information content
of the noise that it adds. That information content is, per chunk,
the entropy of the selection of whether the chunk is bursty,
$H_2(b)$, plus, with probability $b$, the entropy of the flipped bits, $N$,
which adds up to $H_2(b) + Nb$ per chunk (roughly; accurate if $N$ is large).
So, per bit, the capacity is, for $N=100$,
\beq
C = 1 - \left( \frac{1}{N} H_2(b) + b \right) = 1 - 0.207 = 0.793 .
\eeq
In contrast, interleaving, which treats bursts of\index{sermon!interleaving}
errors as independent, causes the channel to be treated as
a binary symmetric channel with $f= 0.2 \times 0.5 = 0.1$, whose capacity is about 0.53.
Interleaving throws away the useful information about the
correlatedness of the errors.
Theoretically, we should be able to communicate about
$(0.79/0.53) \simeq 1.6$ times
faster using a code and decoder that explicitly treat bursts as bursts.
}
% ex 91
\soln{ex.gcCb}{
\ben
\item
Putting together the results of exercises \ref{ex.gcoptens} and \ref{ex.gcC},
we deduce that
a Gaussian channel with real input $x$, and signal to
noise ratio $v/\sigma^2$ has capacity
\beq
C = \frac{1}{2} \log \left( 1 + \frac{v}{\sigma^2}
\right) .
\label{eq.unconstrained.cap}
\eeq
\item
If the input is constrained to be binary, $x \in \{ \pm \sqrt{v} \}$,
the capacity is achieved by using these two inputs
with equal probability.
The capacity is reduced to a somewhat messy integral,
\beq
C'' =
\int_{-\infty}^{\infty}
\d y \, N(y;0) \log N(y;0)
%\nonumber \\
%& &
-
\int_{-\infty}^{\infty}
\d y \,
P(y) \log P(y) ,
\eeq
where $N(y;x) \equiv (1/\sqrt{2 \pi}) \exp [ ( y-x)^2/2 ]$,
$x\equiv \sqrt{v}/ \sigma$,
and $P(y) \equiv [ N(y;x)+N(y;-x) ]/2$.
This capacity is smaller than
the unconstrained capacity (\ref{eq.unconstrained.cap}),
but for small signal-to-noise ratio, the two capacities are close
in value.
\item
If the output is thresholded, then the Gaussian channel is turned
into a binary symmetric channel whose transition probability
is given by the error function $\erf$
defined on page \pageref{sec.erf}. The capacity is
%%%%%%%
\marginfig{%
\begin{center}
\psfig{figure=/home/mackay/_doc/code/brendan/gc.ps,width=1.85in,angle=-90}
\mbox{\psfig{figure=/home/mackay/_doc/code/brendan/gc.l.ps,width=1.85in,angle=-90}}\\[-0.05in]
\end{center}
%
\caption[a]{Capacities (from top to bottom in each graph)
$C$, $C'$, and $C''$,
versus the signal-to-noise ratio $(\sqrt{v}/\sigma)$.
The lower graph is a log--log plot.}
}
%%%%%%%%
\beq
C'' = 1 - H_2( f ), \mbox{ where $f= \erf(\sqrt{v}/\sigma)$} .
\eeq
%\item
% The capacities are plotted in the margin.
\een
}
%\soln{ex.beccode}{
% The design of good codes for erasure channels\index{erasure-correction}
% is an active research area
% \cite{spielman-96,LubyDF}. Have fun!
%}
% RAID
\soln{ex.raid}{
There are several RAID systems. One of the easiest
to understand consists of 7 \disc{} drives which store data\index{erasure-correction}
at rate $4/7$ using a $(7,4)$ \ind{Hamming code}: each successive\index{RAID}\index{redundant array of independent disks}
four bits are encoded with the code and the seven codeword
bits are written one to each disk. Two or perhaps
three disk drives
can go down and the others can recover the data. The
effective channel model here is a binary erasure channel,
because it is assumed that we can tell when a disk is
dead.
It is not
possible to recover the data for {\em some\/} choices
of the three dead disk drives; can you see why?
}
\exercissxB{2}{ex.raid3}{
Give an example of three \disc{} drives that, if lost, lead
to failure of the above RAID system, and three that can
be lost without failure.
}
\soln{ex.raid3}{
The $(7,4)$ Hamming code has codewords of weight 3. If any set of
three \disc{} drives\index{erasure-correction} corresponding to one of those codewords
is lost, then the other four disks can only recover 3 bits
of information about the four source bits; a fourth bit is lost.
[\cf\ \exerciseref{ex.qeccodeperfect} with $q=2$: there are
no binary MDS codes. This deficit is discussed further in
\secref{sec.RAIDII}.]
Any other set of three disk drives can be lost without
problems because the corresponding four by four submatrix
of the generator matrix is invertible.
% The simplest
% example of a recoverable failure is when the three parity
% drives (5,6,7) go down.
A better code would be the digital fountain
-- see \chref{chdfountain}.
% \cite{LubyDF},\footnote{{\tt http://www.digitalfountain.com/}}
}
\dvipsb{solutions real channels s7}
%%%%%%% was a chapter on further exercises here once!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%% PART %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\renewcommand{\partfigure}{\poincare{8.2}}
\part{Further Topics in Information Theory}
\prechapter{About Chapter}
In Chapters \ref{ch1}--\ref{ch7}, we
concentrated on two aspects of information theory
and coding theory: source coding -- the compression
of information so as to make efficient use of data transmission
and storage channels; and channel coding -- the redundant
encoding of information so as to be able to detect and correct
\ind{communication} errors.
In both these areas we started by ignoring practical
considerations, concentrating on the question
of the theoretical limitations and possibilities of coding.
We then discussed practical source-coding and channel-coding
schemes, shifting the emphasis towards computational
feasibility. But the prime criterion for comparing encoding
schemes remained the efficiency of the code in terms of
the channel resources it required: the best source codes
were those that achieved the greatest compression; the best channel
codes were those that communicated at the highest rate with a given
probability of error.
In this chapter we now shift our viewpoint a little, thinking of
{\em ease of information retrieval\/} as a primary goal. It turns out that
the random codes\index{random code} which were theoretically useful in our
study of channel coding are also useful for rapid information
retrieval.
Efficient information retrieval is one of the problems that brains seem
to solve effortlessly, and
\ind{content-addressable memory}\index{memory!content-addressable} is one of the
topics we will study when we look at neural networks.
\medskip
%\chapter{Hash codes: codes for efficient information retrieval}
\ENDprechapter
\chapter{Hash Codes: Codes for Efficient Information Retrieval \nonexaminable}
% 9
\label{ch.hash}
% \chapter{Hash codes: codes for efficient information retrieval}
% \input{tex/_lhash.tex}
%
% prerequisites -- the birthday problem questions
% postreqs: hopfield nets
%
% exercises also in _e8.tex AND _e7.tex, solns in _shash and _se8
% _e8 has ones relevant to hashes
%
% \label{ch.hash}
%
% SUGGESTION:
%
% include an illustrative example at start.
% add a diagram showing buckets, memory....
\newcommand{\hashS}{S}
\newcommand{\hashs}{s}
\newcommand{\hashN}{N}
\newcommand{\hashT}{T}
% \newcommand{\hashn}{n}
\section{The information-retrieval problem}
A simple example of an
\index{information retrieval}{information-retrieval}\index{hash code}\
%\index{code!hash}
problem is the task of
implementing a \ind{phone directory}\index{telephone directory} service, which, in response to a
person's {\dem name}, returns (a) a confirmation that that person
is listed in the directory; and (b) the person's {phone number} and other
details.
We could formalize this problem as follows, with $\hashS$ being the
number of names that must be stored in the \ind{directory}.
\marginfig{\small
\begin{tabular}{@{}p{1.20in}l} \toprule
\parbox[t]{1.2in}{\small string length} & $N \simeq 200$ \\
\parbox[t]{1.2in}{\small\raggedright number of strings} & $S \simeq 2^{23}$ \\
\parbox[t]{1.2in}{\small\raggedright number of possible} & $2^N \simeq 2^{200}$ \\
\parbox[t]{1.2in}{\small\raggedright \hspace{0.2in} strings} & \\
\bottomrule
% WOULD love this paragraph to be indented differently
% HELP
\end{tabular}
\caption[a]{Cast of characters.}
}
% Imagine that y
You are given a list of $\hashS$ binary strings of length
$\hashN$ bits, $\{\bx^{(1)}, \ldots, \bx^{(\hashS)}\}$, where
$\hashS$ is considerably
smaller than the total number of possible strings, $2^\hashN$. We will call
the superscript `$\hashs$' in $\bx^{(\hashs)}$ the {\dem record number\/} of the string.
The idea is that $\hashs$ runs over customers in the order in which they are
added to the directory and $\bx^{(\hashs)}$ is the name of customer $\hashs$. We assume
for simplicity that all people have names of the same length.
The name length might be, say,
$\hashN = 200$ bits, and
we might want to store the details of
ten million customers, so $\hashS \simeq 10^7 \simeq 2^{23}$. We will ignore the possibility that two
customers have identical names.
The task is to construct the inverse of the mapping from $s$ to
$\bx^{(\hashs)}$, \ie, to make a system that, given a string $\bx$,
% with an unknown record number, will
returns the value of $\hashs$ such that
$\bx = \bx^{(\hashs)}$ if one exists, and otherwise reports that no such
$\hashs$ exists. (Once we have the record number, we can go and look in
memory location $\hashs$ in a separate memory full of
phone numbers to find the required
number.)
The aim, when solving this task, is to
% is system should
use minimal computational resources
in terms of the amount of memory used to store the inverse
mapping from $\bx$ to
$\hashs$ and the amount of time to compute the inverse
mapping. And, preferably, the inverse mapping should be implemented
in such a way that
further new strings can be added to the directory
in a small amount of computer time too.\index{content-addressable memory}
%
% add picture to show lookup table
%
\subsection{Some standard solutions}
\label{sec.simplehash}
The simplest and dumbest solutions to the information-retrieval problem
are a look-up table and a raw list.
\begin{description}
\item[The look-up table] is a piece of memory of size $2^N \log_2 \hashS$,
$\log_2 \hashS$ being the amount of memory required to store an integer
between 1 and $\hashS$. In each of the $2^N$ locations, we put a zero, except
for the locations $\bx$ that correspond to strings $\bx^{(\hashs)}$,
into which we write the value of $\hashs$.
The look-up table is a simple and quick solution, but only if there
is sufficient memory for the table, and if the cost of
looking up entries in memory is independent of the memory size.
But in our definition of the task, we assumed that $N$ is
% sufficiently large
about 200 bits or more, so the amount of memory required would be
of size $2^{200}$;
this solution is completely out of the question. Bear in mind that
the number of particles in the solar system is only about $2^{190}$.
% particles in the known universe is
\item[The raw list]
is a simple list of ordered pairs $(\hashs, \bx^{(\hashs)} )$ ordered by the value of
$\hashs$. The mapping from $\bx$ to $\hashs$ is achieved by searching through
the list of strings, starting from the top, and comparing the incoming
string $\bx$
with each record $\bx^{(\hashs)}$ until a
match is found. This system is very easy to
maintain, and uses a small amount of memory, about $\hashS \hashN$ bits,
but is rather slow to use, since on average five million pairwise
comparisons will be made.
\end{description}
\exercissxB{2}{ex.meanhash}{
Show that the average time taken
to find the required string in a raw list, assuming that the original names
were chosen at random, is about $\hashS + N$ binary comparisons.
(Note
that you don't have to compare the whole string of length $N$,
since a comparison can be terminated as soon as a mismatch occurs;
show that you need on average two binary comparisons per incorrect
string match.)
Compare this with the worst-case search time
-- assuming that the devil chooses
the set of strings and the search key.
}
The standard way in which phone directories are made improves
on the look-up table and the raw list by using
an {\dem{{alphabetically-ordered list}}}\index{alphabetical ordering}.
\begin{description}
\item[Alphabetical list\puncspace]
The strings $\{ \bx^{(\hashs)} \}$
% $...$
are sorted into alphabetical order. Searching for
an entry now usually takes less time than was needed for the raw list because
we can take advantage of the sortedness; for example, we can open
the phonebook at its middle page, and compare the
name we find there with the target string; if the target is `greater'
than the middle string then we know that the required string, if
it exists, will be found in the second half of the alphabetical directory.
Otherwise, we look in the first half.
By iterating this splitting-in-the-middle procedure,
we can identify the target string, or establish that the string is not
listed, in $\lceil \log_2 \hashS \rceil$ string comparisons. The expected
number of binary comparisons per string comparison
will tend to increase as the search
progresses,
%, because the leading bits of the two strings involved
% in the comparison are expected to become similar; but by being smart
% and keeping track of which leading bits we have looked at
% already in previous searches, it seems plausible that
% we can reduce the number of binary
% operations to about $\lceil \log_2 \hashS \rceil + N$ binary comparisons.
but the total number of binary comparisons required will be
no greater than $\lceil \log_2 \hashS \rceil N$.
The amount of memory required is the same as that required for the raw list.
Adding new strings to the database requires that we insert them in the
correct location in the list. To find that location takes about
$\lceil \log_2 \hashS \rceil$ binary comparisons.
%Then shuffling along all
% of the subsequent entries in the directory to make space for the
% new entry may take some computer time, depending on how the memory works.
\end{description}
Can we improve on the well-established alphabetized list?
Let us consider our task from some new viewpoints.
% for a moment and think of other ways of viewing it.
The task is to construct a mapping $\bx \rightarrow \hashs$ from $N$ bits
% ($\bx$)
to $\log_2 \hashS$ bits.
% ($\hashs$).
%
% what does this mean?
%
This is a pseudo-invertible mapping, since for any $\bx$
that maps to a non-zero $\hashs$, the customer database contains the
pair $(\hashs , \bx^{(\hashs)})$ that takes us back. Where have we come
across the idea of mapping from $N$ bits to $M$ bits before?
We encountered this idea twice: first,
in source coding, we studied block codes which were mappings
from strings of $N$ symbols to a selection of one label in a list.
% $...$.
The task of information retrieval is similar
% pretty much identical
to the task
(which we never actually solved) of making an encoder for a
typical-set compression code.
The second time that we mapped bit strings to bit strings of another dimensionality
was when we studied channel codes. There, we considered codes
that mapped from $K$ bits to $N$ bits, with $N$ greater than $K$,
and we made theoretical progress using {\em random\/} codes.
In hash codes, we put together these two notions.
We will study {random codes that map from $N$ bits to $M$ bits where
$M$ is {\em smaller\/} than $N$}.\index{random code}
% Another strand: the dumb look-up table would be really nice, very quick,
% the only problem is it requires too much memory. But there are so
% few vectors, what if we project them down into a lower-dimensional
% space? A few will collide, but if they are mainly distinct then
% we can just implement the look-up table in a lower dimensional
% space.
The idea is that we will map the original high-dimensional space
down into a lower-dimensional space, one in which it is feasible
to implement the dumb look-up table method which we rejected a
moment ago.
\marginfig{\small
\begin{tabular}{@{}p{1.2in}l} \toprule
\parbox[t]{1.2in}{\small string length} & $N \simeq 200$ \\
\parbox[t]{1.2in}{\small number of strings} & $S \,\simeq 2^{23}$ \\
\parbox[t]{1.2in}{\small size of hash function} & $M \simeq 30\ubits$ \\[0.01in]
\parbox[t]{1.2in}{\small size of hash table} & $T = 2^M $\\
& $\:\:\:\:\: \simeq 2^{30}$ \\ \bottomrule
% HELP the spacing between successive rows
% is smaller than the spacing between lines!! :-(
% HELP
\end{tabular}
\caption[a]{Revised cast of characters.}
}
\section{Hash codes}
First we will describe how a hash code works, then we will study the
properties of idealized hash codes.
A hash code implements a solution to the information-retrieval problem,
that is, a mapping from $\bx$ to $s$, with the help of a pseudo-random
function called a {\dem\ind{hash function}},
which maps the $N$-bit string $\bx$ to an $M$-bit string $\bh(\bx)$,
where $M$ is smaller than $N$. $M$ is typically chosen
% to be sufficiently small
such that the `table size' $\hashT \simeq
2^M$ is a little bigger than $S$ -- say,
ten times
% one or two orders of magnitude
bigger.
For example,
if we were expecting
% $S$ a million values for $\bx$
$S$ to be about a million,
we might map
%a 200-bit
$\bx$ into a 30-bit hash $\bh$ (regardless of the size $N$ of each
item $\bx$).
The hash function is some fixed deterministic function which should
ideally be indistinguishable from a fixed random code. For practical
purposes, the hash function must be quick to compute.
Two simple examples of \ind{hash function}s are:
\begin{description}
\item[Division method\puncspace]
The table size $\hashT$ is a prime number, preferably
one that is not close to a power of 2. The hash value is the remainder
when the integer $\bx$ is divided by $\hashT$.
\item[Variable string addition method\puncspace]
This method assumes that $\bx$ is a string of
bytes and that the table size $\hashT$ is 256.
The characters of $\bx$ are added, modulo 256.
%
%
% http://members.xoom.com/thomasn/s_man.htm
%
%
%
% This hash function does not distinguish anagrams.
This hash function has the defect that it maps strings
that are anagrams of each other onto the same hash.
It may be improved by putting the running total through
a fixed pseudorandom permutation
after each character is added.
%
%\item[
In the\index{hash function}
{\dem variable string exclusive-or method\/} with table size $\leq 65\,536$,
the string is hashed twice in this way, with the initial running total
being set to 0 and 1 respectively (\algref{alg.hashxor}).
The result is a 16-bit hash.
\end{description}
%
% probably a good idea to include this code stolen from Thomas Niemann
% typedef unsigned short int HashIndexType; (changed to int)
%
\begin{algorithm}% figure}
\begin{framedalgorithmwithcaption}%
{
\caption[a]{{\tt C} code implementing the variable string exclusive-or method
to create
a hash {\tt h} in the range $0\ldots 65\,535$
from a string {\tt x}.
Author: Thomas Niemann.}
\label{alg.hashxor}
}
\small
\begin{verbatim}
unsigned char Rand8[256]; // This array contains a random
permutation from 0..255 to 0..255
int Hash(char *x) { // x is a pointer to the first char;
int h; // *x is the first character
unsigned char h1, h2;
if (*x == 0) return 0; // Special handling of empty string
h1 = *x; h2 = *x + 1; // Initialize two hashes
x++; // Proceed to the next character
while (*x) {
h1 = Rand8[h1 ^ *x]; // Exclusive-or with the two hashes
h2 = Rand8[h2 ^ *x]; // and put through the randomizer
x++;
} // End of string is reached when *x=0
h = ((int)(h1)<<8) | // Shift h1 left 8 bits and add h2
(int) h2 ;
return h ; // Hash is concatenation of h1 and h2
}
\end{verbatim}
% original code stored in tex/_hash.code
\end{framedalgorithmwithcaption}
\end{algorithm}% figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}
\figuremargin{\footnotesize
\setlength{\unitlength}{1mm}
\thinlines
\begin{picture}(100,100)(-20,-40)
\put(65,-40){\line(0,1){90}}
\put(75,-40){\line(0,1){90}}
\multiput(65,-40)(0,3){31}{\line(1,0){10}}
\newcommand{\xvector}[2]{\put(-10,#1){\framebox(40,4){$\bx^{(#2)}$}}}
\newcommand{\hvector}[2]{\put(53,#1){\makebox(5,0){$\bh(\bx^{(#2)})\rightarrow$}}}
\newcommand{\svector}[2]{\put(74.3,#1){\makebox(0,0)[r]{$#2$}}}
\newcommand{\slvector}[2]{\put(35,#1){\vector#2{10}}}
\newcommand{\xhs}[4]{\xvector{#1}{#2}\hvector{#3}{#2}\svector{#3}{#2}\slvector{#1}{#4}}
\xhs{30}{1}{18.7}{(1,-1)}
\xhs{24}{2}{45.536}{(1,2)}
\xhs{18}{3}{6.7}{(1,-1)}
\xhs{0}{s}{-20.5}{(1,-2)}
% labels
\put(39,65){\makebox(0,0){Hash}}
\put(39,62){\makebox(0,0){function}}
\put(34,59){\vector(1,0){11}}
\put(10,60){\makebox(0,0){Strings}}
\put(48,58.60){\makebox(0,0)[l]{hashes}}
\put(70,62){\makebox(0,0){Hash table}}
%
\put(10,12){\makebox(0,0){$\vdots$}}
\put(10,-8){\makebox(0,0){$\vdots$}}
% N range indication
\put(10,40){\vector(-1,0){20}}
\put(10,40){\vector(1,0){20}}
\put(10,43){\makebox(0,0){$N$ bits}}
% M range indication
\put(70,54){\vector(-1,0){5}}
\put(70,54){\vector(1,0){5}}
\put(70,57){\makebox(0,0){$M$ bits}}
% 2^M range
\put(82,5){\vector(0,-1){45}}
\put(82,5){\vector(0,1){45}}
\put(84,5){\makebox(0,0)[l]{$2^M$}}
% S range
\put(-15,10){\vector(0,1){23}}
\put(-15,10){\vector(0,-1){30}}
\put(-17,10){\makebox(0,0)[r]{$S$}}
%
\end{picture}
}{
\caption[a]{Use of hash functions for information retrieval.
For each string $\bx^{(s)}$,
the hash $\bh= \bh(\bx^{(s)})$ is computed,
and the value of $s$ is written into the
$\bh$th row of the hash table. Blank rows in the hash table
contain the value zero.
The table size is $T = 2^M$.}
\label{fig.hashtable}
}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Having picked a hash function $\bh(\bx)$,
we implement an
% efficient
information retriever
as follows. (See \figref{fig.hashtable}.)
\begin{description}
\item[Encoding\puncspace]
A piece of memory called the {\em hash table\/}
is created of size $2^Mb$ memory units, where $b$ is the amount of memory needed to represent
an integer between $0$ and $\hashS$. This table is initially set to zero
throughout. Each memory $\bx^{(\hashs)}$ is put through
the hash function, and at the location in the hash table corresponding
to the resulting vector $\bh^{(\hashs)} = \bh( \bx^{(\hashs)} )$, the integer $\hashs$ is written --
unless that entry in the hash table is already occupied, in which case
we have a {\em collision\/} between $\bx^{(\hashs)}$ and some earlier
$\bx^{(\hashs')}$ which both happen to have the same hash code.
Collisions can be handled in various ways -- we will discuss some
in a moment -- but first let us complete the basic picture.
\item[Decoding\puncspace]
To retrieve a piece of information corresponding to a
target vector $\bx$, we compute the hash $\bh$ of
$\bx$ and look at the corresponding location in the hash table.
If there is a zero, then we know immediately that the string $\bx$ is
not in the database. The cost of this answer is the cost of one hash-function
evaluation and one look-up in the table of size $2^M$.
If, on the other hand, there is a non-zero entry $\hashs$ in the table,
there are two possibilities: either the vector $\bx$ is
indeed equal to $\bx^{(\hashs)}$; or the vector $\bx^{(\hashs)}$ is another
vector that happens to have the same hash code as the target $\bx$. (A third
possibility is that this
non-zero entry might have something to do with our
yet-to-be-discussed collision-resolution system.)
To check whether $\bx$ is
indeed equal to $\bx^{(\hashs)}$, we take the tentative answer $\hashs$,
look up $\bx^{(\hashs)}$ in the original forward database, and compare it
bit by bit with $\bx$; if it matches then we report $\hashs$ as the
desired answer. This successful retrieval has an overall cost of
one hash-function
evaluation, one look-up in the table of size $2^M$,
another look-up in a table of size $\hashS$, and
% up to
$N$ binary comparisons -- which may be much cheaper
than the simple solutions presented in section \ref{sec.simplehash}.
\end{description}
\exercissxB{2}{ex.hash.retrieval}{
If we have checked the first few bits of $\bx^{(\hashs)}$ with
$\bx$
and found them to be equal, what is the probability that
the correct entry has been retrieved, if the alternative hypothesis
is that $\bx$ is actually not in the database? Assume that
the original source strings are random, and the hash function is a random hash function.
How many
% Could have an exercise here on the number of
binary evaluations are
needed to be sure with odds of a billion to one that
the correct entry has been retrieved?
% [Note we are not assuming that the
% original strings $\{ \bx^{(\hashs)} \}$ are random; they may be
% very similar to each other. We are just assuming that the hash function
% is random.]
}
%
% view as a kind of source
% encoding - reduces huge redundancy, where the redundancy
% has the form P(x) = sum_x pi_c delta(x_c)
%
% does so using random coding.
The hashing method of information retrieval
can be used for strings $\bx$ of arbitrary length,
if the hash function $\bh(\bx)$ can be applied to strings of
any length.
\section{Collision resolution}
We will study two ways of resolving collisions: appending in the
table, and storing elsewhere.
\subsection{Appending in table}
When encoding, if a collision occurs, we continue
down the hash table and write the value of $s$ into the next available
location in memory that currently contains a zero. If we reach the bottom
of the table before encountering a zero, we continue from the top.
When decoding, if we compute the hash code for $\bx$ and find that
the $s$ contained in the table doesn't point to an $\bx^{(s)}$ that
matches the cue $\bx$, we continue down the hash table until we
either find an $s$ whose $\bx^{(s)}$ does match the cue
% key
$\bx$, in which case we are done, or else encounter a zero, in which
case we know that the cue $\bx$ is not in the database.
For this method, it is essential that the table be substantially
bigger in size than $\hashS$. If $2^M < \hashS$ then the encoding
rule will become stuck with nowhere to put the last strings.
\subsection{Storing elsewhere}
A more robust and flexible method is to use {\dem pointers\/}
to additional pieces of memory in which collided strings are stored.
There are many ways of doing this. As an example, we could store
in location $\bh$ in
the hash table a pointer (which must be distinguishable from
a valid record number $s$) to a `bucket' where all the
strings that have hash code $\bh$ are stored in a
{\dem sorted list}.
The encoder sorts the strings in each bucket alphabetically as the hash table and buckets
are created.
The decoder simply has to go and look in the relevant bucket
and then check the short list of strings that are there by a
brief alphabetical search.
% of strings that have this encoding.
This method of storing the strings in buckets allows the option of
making the hash table quite small, which may have practical benefits. We
may make it so small that almost all strings are involved in collisions,
so all buckets contain a small number of strings.
It only takes a small number of binary comparisons to identify which
of the strings in the bucket matches the cue $\bx$.
\section{Planning for collisions: a birthday problem}
\index{birthday}
\exercissxA{2}{ex.hash.collision}{
If we wish to store $S$ entries using a hash function
whose output has $M$ bits, how many collisions should we expect
to happen, assuming that our hash function is an ideal random function?
What size $M$ of hash table is needed if we would like
the expected number of collisions to be smaller than 1?
What size $M$ of hash table is needed if we would like
the expected number of collisions to be a small fraction, say 1\%,
of $S$?
}
[Notice the similarity of this problem to
\exerciseref{ex.birthday}.]
\section{Other roles for hash codes}
\subsection{Checking arithmetic}
\index{error detection}If you wish to check an addition that was done by hand,
you may find useful the method of {\dem{\ind{casting out nines}}}.\index{nines}
In casting out nines, one finds the sum, modulo nine, of
all the {\em digits\/} of the numbers
to be summed and compares it with the
sum, modulo nine, of the digits of the putative answer.
[With a little practice, these sums can be computed
much more rapidly than the full original addition.]
% calculation proper.]
\exampla{%???????????
% want this to have reference: {ex.nines}{
In the calculation shown in the margin
\marginpar{\begin{center}
\begin{tabular}[t]{r}
{\tt 189} \\
{\tt +1254} \\
{\tt + 238} \\
\hline
{\tt 1681} \\
\end{tabular}
\end{center}}
the sum, modulo nine, of the digits in {\tt 189+1254+238}
is {\tt 7}, and the sum, modulo nine, of {\tt 1+6+8+1} is {\tt 7}.
The calculation thus passes the casting-out-nines test.
}
Casting out nines gives a simple example of a hash function.
For any addition expression of the form $a+b+c+\cdots$,
where $a, b, c, \ldots$ are decimal numbers
we define $h \in \{0,1,2,3,4,5,6,7,8\}$ by
\beq
h(a+b+c+\cdots) = \mbox{ sum modulo nine of all digits in $a,b,c$ } ;
\eeq
then it is nice property of decimal arithmetic that if
\beq
a+b+c+\cdots = m+n+o+\cdots
\eeq
then the hashes $h(a+b+c+\cdots)$ and $h(m+n+o+\cdots)$ are equal.
\exercissxB{1}{ex.nines.p}{
What evidence\index{model comparison} does a correct casting-out-nines
match give in favour of the
hypothesis that the addition has been done correctly?
}
\subsection{Error detection among friends}
\index{error detection}Are two files the same? If the files are on the same computer,
we could just compare them bit by bit. But if the two files
are on separate machines,
it would be nice to have a way of confirming that two
files are identical without having to transfer one of the
files from A to B. [And even if we did transfer one of the files,
we would still like a way to confirm whether it has been
received without modifications!]
This problem can be solved using hash codes.
% Alice sends a file to Bob, and wants to do error detection.
Let Alice and Bob be the holders of the two files; Alice sent
the file to Bob, and they wish to confirm it has been received
without error.
If Alice computes the hash
% function
of her file and sends it to Bob,
and Bob computes the hash
% function
of his file, using the
same $M$-bit hash function, and the two hashes match, then
Bob can deduce that the two files are almost surely the
same.
% should have some sort of reference to digest?
% The hash of the file is often called the {\dem\ind{digest}}.
\exampl{example.hash.II}{
What is the probability of a false negative, \ie,
the probability, given that
the two files do differ, that the two hashes
% Bob concludes
are nevertheless identical?
}
% Solution::::::::
If we assume that the hash function is random and that
the
% unrelated
process that causes the files to differ knows nothing about the hash function,
then the probability of a false negative is $2^{-M}$.\ENDsolution
A 32-bit hash gives a probability of false negative of about $10^{-10}$.
% 2.3283064365387e-10
It is common practice to use a linear hash function called
a 32-bit cyclic redundancy check to detect errors in files.
(A cyclic redundancy check is a set of 32 parity-check bits
similar to the 3 parity-check bits of the $(7,4)$ Hamming code.)
%%%%%%%%% end solution
\begin{conclusionbox}
To have a false-negative rate smaller than one in a billion,
$M = 32$ bits is plenty, if the errors are produced by noise.
\end{conclusionbox}
\exercissxB{2}{ex.whyonlyCRC}{
Such a simple parity-check code only detects errors; it doesn't help correct
them. Since error-{\em{correcting\/}} codes exist, why not use
one of them to get some error-correcting
capability too?
}
%
% more maths requested here
%
\subsection{Tamper detection}
\index{security}\index{tamper detection}\index{detection of forgery}\index{forgery}What
if the differences between the two files are not
simply `noise', but are introduced by an adversary,
a clever {\dem forger\/} called
Fiona, who modifies the original file to make
a \ind{forgery}\index{cryptography!digital signatures}\index{cryptography!tamper detection}
that purports to be \ind{Alice}'s file?
How can Alice make a \ind{digital signature}
for the file so that \ind{Bob} can confirm that
no-one has tampered with the file?
And how can we prevent Fiona from listening
in on Alice's signature and attaching it to other
files?
Let's assume that Alice computes a hash function for the
file and sends it securely to Bob.
% , in the same way as for error-detection above.
If Alice computes a simple hash function for the file
like the linear cyclic redundancy check, and Fiona knows
that this is the method of verifying the file's
integrity, Fiona can make her chosen modifications
to the file and then easily identify (by linear
algebra) a further 32-or-so single bits
that, when flipped, restore the hash function
of the file to its original value.
{\em Linear hash functions give no security against
forgers.}
We must
therefore require that the hash function\index{inversion of hash function}
be {\em hard to invert\/} so that no-one can construct a
tampering that leaves the hash function unaffected.
We would still like the hash function to be easy
to compute, however, so that Bob doesn't have to
do hours of work to verify every file he received.
Such a hash function -- easy to compute, but hard to invert --
is called
a {\dem\ind{one-way hash function}}.\index{hash function!one-way}
Finding such functions is one of the active research areas of
\ind{cryptography}.
% Don't want to use an ecc, because with a linear ecc it is easy to construct
% a pair of tamperings which have the same syndrome and
% so leave the hash unaffected.
%How can we invent a function that has the
%property that h(x) is easy to compute, but
%it is very hard to find an x
%suxh that h(x) has a chosen value h?
%A lot of research is being done on this question
%still, and the sort of functions people use
%to make a one-way hash function are functions like:
%
% exponentiation-modulo-M
%
%Definition:
% take x, and think of it as a number.
% compute 1023^(x) modulo M,
% where "^" means "1023 to the power x",
% and M is some other integer, eg 97.
%
%Apparently it is hard to invert this sort of
% function (i.e. to take the "discrete logarithm").
%
%Real one-way hash functions are more complicated than
%this, but I hope this gives the idea.
%
A hash function that is widely used in the free software\index{software!hash function}
community to confirm that two files do not differ
is {\tt\ind{MD5}}, which produces a 128-bit
hash. The details of how it works
are quite complicated, involving convoluted exclusive-or-ing
and if-ing and and-ing.\footnote{{\tt http://www.freesoft.org/CIE/RFC/1321/3.htm}}
%
% of bits with each other
%
% Cryptography is the topic of the next chapter.
%
% rsync uses MD4 with a 128-bit checksum (for files with a matching size
% and date) initially. But (from the man entry):
% Current versions of rsync actually use an adaptive
% algorithm for the checksum length by default, using
% a 16 byte file checksum to determine if a 2nd pass
% is required with a longer block checksum. Only use
% this option if you have read the source code and
% know what you are doing.
% The `md5sum' program also uses 128 bits.
Even with a good one-way hash function, the digital signatures
described above
are still vulnerable to attack, if Fiona has access to the
hash function. Fiona could take the tampered file
and hunt for a further tiny modification to it such that its hash
matches the original hash of Alice's file. This would take
some time -- on average,
about $2^{32}$ attempts, if the hash function has
32 bits -- but eventually Fiona would find a tampered file that
matches the given hash. To be secure against
forgery, \ind{digital signature}s must either have
enough bits for such a random search to take too long,
or the \ind{hash function} itself must be kept
\ind{secret}.
\begin{conclusionbox}
Fiona has to hash $2^M$ files to cheat.
$2^{32}$ file modifications is not very many, so a 32-bit hash function
is not large enough for \ind{forgery} prevention.
\end{conclusionbox}
% If Fiona works as
Another person who might have a motivation for forgery is
Alice herself.
For example, she might be making a bet on the outcome
of a race, without wishing to broadcast her prediction
publicly; a method for placing bets would be for her
to send to Bob the bookie the hash of her bet.
Later on, she could send Bob the details of her bet.
Everyone can confirm that her bet is consistent with
the previously publicized hash. [This method
of secret publication
% shing ideas
was used by Isaac Newton and Robert Hooke\index{Newton, Isaac}\index{Hooke, Robert}
% (1635-1703)
when they
wished to establish priority for scientific ideas
without revealing them. Hooke's hash function was alphabetization
% ed latin statements,
as illustrated by the
conversion of {\em UT TENSIO, SIC VIS\/} into the \ind{anagram} {\tt{CEIIINOSSSTTUV}}.]
% http://www.microscopy-uk.org.uk/mag/artmar00/hooke2.html
% http://www.rod.beavon.clara.net/leonardo.htm
% It was in his Helioscopes in 1676 that Hooke followed the popular seventeenth-century conceit of announcing a discovery in an anagram: cediinnoopsssttuu. He published its key two years later, in his most complete treatment of elasticity, in De Potentia Bestitutiva, or Of Spring. Here Hooke enunciated the original formulation of the law that bears his name: Ut Pondus sic Tensia, or 'the weight is equal to the tension'. [33] As the tension was seen as the product of an increasing series of weights in pans suspended on coiled springs, it is easy in this pre-Newtoniangravitation age to understand how Hooke spoke of the pondus, or weight, as acting on the spring. The formulation of 'Hooke's Law' with which we are more familiar today is Ut Tensia, sic Vis, or 'the tension is equal to the force'.
%
% http://www.aero.ufl.edu/~uhk/strength/strength.htm ??? CEIIOSSOTTUU ??? CEIINOSSITTUV
% ??? ceiiinosssttvv
% all accounts differ!
% http://arc-gen1.life.uiuc.edu/Bioph354/lect19.html
Such a protocol relies on the assumption that Alice cannot
change her bet after the event without the hash coming out wrong.
How big a hash function do we need to use to ensure that
Alice cannot cheat?
The answer is different from the size of the hash we needed
in order to defeat Fiona above, because Alice is the author of {\em
both\/} files. Alice could \ind{cheat} by searching for
two files that have identical
hashes to each other. For example, if
she'd like to cheat by placing two bets for the price of
one, she could make a large number $N_1$ of
versions of bet one (differing from each other
in minor details only), and a large number $N_2$ of versions of bet two,
and hash them all. If there's a \ind{collision} between
the hashes of two bets of different types,
then she can submit the common hash and thus buy herself the
option of placing either \ind{bet}.
\exampl{example.hashN1N2}{
If the hash has $M$ bits, how big do $N_1$
and $N_2$ need to be for Alice to have a good chance of finding
two different bets with the same hash?
}
% solution
This is a \ind{birthday} problem like \exerciseref{ex.birthday}.
If there are $N_1$ Montagues and $N_2$ Capulets at a party,
and each is assigned a `birthday' of $M$ bits,
the expected number of \ind{collision}s between a Montague and a Capulet
is
\beq
N_1 N_2 2^{-M} ,
\eeq
so to minimize the number of files hashed, $N_1+N_2$, Alice
should make $N_1$ and $N_2$ equal, and will need to hash about
$2^{M/2}$ files until she finds two that match.\ENDsolution
\begin{conclusionbox}
Alice has to hash $2^{M/2}$ files to cheat.
[This is the square root of the number of hashes Fiona
had to make.]
\end{conclusionbox}
If Alice has the use of $C=10^6$ computers for $T=10$\,years, each computer
taking $t=1\,$ns to evaluate a hash,
the bet-communication system is\index{security}
secure against Alice's dishonesty only if $M \gg 2 \log_2 CT/t \simeq
160$ bits.
% end solution
\section*{Further reading}
The Bible for hash codes is volume 3 of \citeasnoun{KnuthAll}.
I highly recommend the story of Doug McIlroy's {\tt{\ind{spell}}}
program, as told in section 13.8 of {\em{Programming Pearls}} \cite{Bentley2}.
This astonishing piece of software makes use of a 64-\kilobyte\ data structure
to store the spellings of all the words of $75\,000$-word dictionary.
% also has some hash functions for strings on p 161, chapter 15.
% and random text generator.
\section{Further exercises} % removed and returned (maybe should transfer some of these?)
% solutions in _se8.tex
% oct 97
%
% info theory and the real world
%
\fakesection{Information theory and the real world (questions relating to hash functions)}
\exercisaxA{1}{ex.address}{
What is the shortest the \ind{address} on a typical international {letter}
could be, if it is to get to a unique human recipient? (Assume the permitted
characters are {\tt{[A-Z,0-9]}}.)
How long are typical \ind{email} addresses?
}
\exercissxA{2}{ex.uniquestring}{
How long does a piece of text need to be for you to be pretty
sure that no human has written that string of characters
before?
How many notes are there in a new \ind{melody}\index{music} that
has not been composed before?
}
\exercissxB{3}{ex.proteinmatch}{
{\sf Pattern recognition by \ind{molecules}}.\index{pattern recognition}
Some proteins produced in a cell have a regulatory role.
A regulatory \ind{protein} controls
the transcription of specific \ind{genes} in the \ind{genome}.
% that might code for other proteins or sometimes the protein itself.
This control often involves the protein's binding to a particular \ind{DNA}
sequence in the vicinity of the regulated gene. The presence of the
bound protein either promotes or inhibits transcription of the gene.
\ben
\item
Use information-theoretic arguments to obtain a lower bound on the size of a
typical protein that acts as a regulator specific to one gene in the
whole human genome. Assume that the genome is
a sequence of
$3 \times 10^{9}$
nucleotides drawn from a four letter alphabet $\{{\tt A},{\tt C},{\tt G},{\tt T}\}$;\index{amino acid}\index{nucleotide}\index{binding DNA}
a protein is a sequence of amino acids drawn from a twenty letter alphabet.
[Hint: establish how long the recognized DNA sequence has to be
in order for that sequence to be
unique to the vicinity of one
gene, treating the rest of the genome as a random sequence. Then
discuss how big the protein must be to recognize a
sequence of that length uniquely.]
\item
Some of the sequences recognized by \ind{DNA}-binding regulatory\index{protein!regulatory}
proteins consist of a subsequence that is repeated twice or
more, for example
the sequence
\beq
\mbox{{\tt{\underline{GCCCCC}CACCCCT\underline{GCCCCC}}}}
\eeq
is a binding site found upstream of the alpha-actin gene in humans.
%; this is a binding site for a transcription factor called Sp1.
Does the fact that some binding sites consist of
a {repeated\/} subsequence influence your answer to part (a)?
\een
}
%
% stole information acquisition exercises from here to move to gene chapter
%
\dvips
\section{Solutions}% to Chapter \protect\ref{ch.hash}'s exercises} %
\soln{ex.meanhash}{
First imagine comparing the string $\bx$ with
another random string $\bx^{(s)}$.
The probability that the first bits of the two strings match
is $1/2$. The probability that the second bits match
is $1/2$. Assuming we stop comparing once we hit the
first mismatch, the expected number of matches is 1,
so the expected number of comparisons is 2
\exercisebref{ex.waithead}.
% errors corrected in draft 2.0.7 on Sun 31/12/00
Assuming the correct string is located at random in the
raw list, we will have to compare with an average
of $\hashS/2$ strings before we find it, which costs
$2 \hashS/2$ binary comparisons; and comparing
the correct strings takes $N$ binary comparisons,
giving a total expectation of $\hashS + N$ binary comparisons,
if the strings are chosen at random.
In the worst case (which may indeed happen in practice),
the other strings are very similar
to the search key, so that a lengthy sequence of comparisons
is needed to find each mismatch. The worst case is when the correct
string is last in the list, and all the other strings
differ in the last bit only, giving a requirement of $\hashS N$
binary comparisons.
}
\soln{ex.hash.retrieval}{
The likelihood ratio for the two hypotheses,
$\H_0$: $\bx^{(\hashs)} = \bx$, and
$\H_1$: $\bx^{(\hashs)} \neq \bx$,
contributed by the datum `the first bits of $\bx^{(\hashs)}$ and $\bx$ are equal'
is
\beq
\frac{ P( \mbox{Datum} \given \H_0 ) }
{ P( \mbox{Datum} \given \H_1 ) }
= \frac{1}{1/2} = 2.
\eeq
If the first $r$ bits all match, the likelihood ratio is $2^r$ to one.
On finding that 30 bits match, the odds are a billion to one
in favour of $\H_0$, assuming we start from even odds.
[For a complete answer, we should compute the evidence
% prior probability of $\H_0$ and $\H_1$
given by the prior information that the hash entry $s$
has been found in the table at $\bh(\bx)$. This fact gives further evidence
in favour of $\H_0$.]
}
\soln{ex.hash.collision}{
Let the hash function have an output alphabet of size $T = 2^M$.
If $M$ were equal to $\log_2 S$ then we would have exactly enough bits
for each entry to have its own unique hash.
The probability that one particular pair of entries collide under a random
hash function is $1/T$.
The number of pairs is $S(S-1)/2$. So the expected number
of collisions between pairs is exactly
\beq
S(S-1)/(2T).
\eeq
If we would like this to be smaller than 1, then we need
$
T > S(S-1)/2
% S(S-1) < 2A \:\: \Rightarrow \:\: S < \sqrt{2A}
$
so
\beq
M > 2 \log_2 S.
\label{eq.M2Shash}
\eeq
We need {\em twice as many\/} bits as the number of bits,
$\log_2 S$,
that would be sufficient to give each entry a unique name.
% fS = S(S-1)/(2A)
% A = (S-1) / (2 f )
If we are happy to have occasional collisions, involving a fraction
$f$ of the names $S$, then
we need $T > S/f$ (since the probability that one particular name
is collided-with is $f \simeq S/T$) so
\beq
M > \log_2 S + \log_2 [1/f] ,
\label{eq.MShash}
\eeq
which means for $f \simeq 0.01$ that we need an extra
7 bits above $\log_2 S$.
The important point to note is the \ind{scaling} of $T$
with $S$ in the two cases (\ref{eq.M2Shash},$\,$\ref{eq.MShash}). If we want
the hash function to be collision-free, then
we must have $T$ greater than $\sim \! S^2$.
If we are happy to have a small frequency of collisions, then
$T$ needs to be of order $S$ only.
% some factor greater than
}
%
%
%
\soln{ex.nines.p}{
The posterior probability ratio for
the two hypotheses, $\H_{+} = $ `calculation correct'
and $\H_{-} = $ `calculation incorrect'
is the product of the prior probability ratio
$P(\H_{+})/P(\H_{-})$ and the likelihood ratio,
$P(\mbox{match} \given \H_{+})/P(\mbox{match} \given \H_{-})$.
This second factor is the answer to the question.
The numerator $P(\mbox{match} \given \H_{+})$ is equal to 1.
The denominator's value depends on our model of errors.
If we know that the human calculator is prone to errors
involving multiplication of the answer by 10, or to transposition
of adjacent digits, neither of which affects the hash value,
then
$P(\mbox{match} \given \H_{-})$ could be equal to 1 also,
so that the correct match gives no evidence
in favour of $\H_{+}$. But if we assume that errors are
`random from the point of view of the hash function' then
the probability of a false positive is
$P(\mbox{match} \given \H_{-}) = 1/9$, and the
correct match gives evidence 9:1 in favour
of $\H_{+}$.
}
%
%
%
\soln{ex.whyonlyCRC}{
If you add a tiny $M=32$ extra bits of hash to a huge $N$-bit
file you get pretty good \ind{error detection}\index{error-correcting code} --
% $1-2^{-M}$
the probability that an
% of detecting an error, less than a one-in-a-billion chance that the
error is undetected is $2^{-M}$,
less than one in a billion. To do error {\em correction\/}
requires far more check bits, the number depending on the expected types of
corruption, and on the file size.
For example, if just eight random bits in a megabyte file
are corrupted, it would take
% $\log_2 {{ 8\times 10^{6}} \choose {8} } \simeq 180$
about $\log_2 {{ 2^{23} }\choose{8} } \simeq 23 \times 8 \simeq 180$
bits
to specify which are the corrupted bits, and the number of \ind{parity-check bits} used by a successful error-correcting code would have to
be at least this number, by the counting argument of \exerciseonlyref{ex.makecode2error}
(solution, \pref{ex.makecode2error.sol}).
% Shannon's \ind{noisy-channel coding theorem}.
}
% see also _se8.tex
\fakesection{se8}
%\begincuttable% NO, I LIKE IT
\soln{ex.uniquestring}{
We want to know the length $L$ of a string
such that it is very improbable that that
string matches any part of the entire writings
of humanity.
Let's estimate that these writings total
about one book for each person living, and that each book
contains two million characters (200 pages with $10\,000$ characters
per page) -- that's
% $5\times 10^9 \times 2 \times 10^6 =
$10^{16}$ characters, drawn from
an alphabet of, say, 37 characters.
The probability that a randomly chosen string of length $L$ matches
at one point in the collected works of humanity is $1/37^{L}$.
So the expected number of matches is
$10^{16} /37^{L}$, which is vanishingly small if
$L \geq 16/\log_{10} 37 \simeq 10$.
% 10.2
Because of the redundancy and repetition of humanity's writings,
it is possible that $L \simeq 10$ is an overestimate.
So, if you want to write something unique, sit down and compose
a string of ten characters. But don't write {\tt{gidnebinzz}}, because
I already thought of that string.
As for a new \ind{melody},\index{music} if we focus on the sequence of notes,
ignoring duration and stress, and
allow leaps of up to an octave at each note,
then the number of choices per note is 23.
The pitch of the first note is arbitrary.
The number of melodies of length $r$ notes in this rather
ugly ensemble of \ind{Sch\"onberg}ian tunes is $23^{r-1}$;
for example, there are $250\,000$ of length $r=5$.
Restricting the permitted intervals will reduce this figure;
including duration and stress will increase it again.
[If we restrict the permitted intervals to
repetitions and tones or semitones,
the reduction is particularly severe; is this why
the melody of
`\ind{Ode to Joy}' sounds so boring?]
The number of recorded compositions is probably less than a
million.
% top of the pops for 50 * 50 weeks with 100 new songs per week
If you learn 100 new melodies per week for every week of your
life then you will have learned $250\,000$ melodies at age 50.
Based on empirical experience of playing the game\index{game!guess that tune}
`{\tt{guess that tune}}',\marginpar{\small\raggedright{In {\tt{guess that tune}},
one player chooses a melody, and sings a gradually-increasing number
of its notes, while the other
participants try to guess the whole melody.\medskip
% aka http://www.melodyhound.com/
The {\dem\ind{Parsons code}\/} is a related hash function for
melodies:
% . To make the Parsons code of a melody,
each pair of consecutive notes is coded as {\tt{U}} (`up')
if the second note is higher than the first, {\tt{R}} (`repeat')
if the pitches are equal, and {\tt{D}} (`down') otherwise.
You can find out how well this hash function
works at {\tt{www.{\breakhere}name-{\breakhere}this-{\breakhere}tune.{\breakhere}com}}.
}}
it seems to me that
whereas many four-note sequences are shared in common
between melodies, the number of collisions between
five-note sequences is rather smaller -- most famous five-note
sequences are unique.
}
%\ENDcuttable
\soln{ex.proteinmatch}{
%\ben
%\item
(a) Let the DNA-binding \ind{protein} recognize a sequence of length $L$ nucleotides.
That is, it binds preferentially to that \ind{DNA} sequence, and not to
any other pieces of DNA in the whole genome. (In reality, the
recognized sequence may contain some wildcard characters, \eg,
the {\tt{*}} in {\tt{TATAA*A}}, which denotes `any of {\tt{A}}, {\tt{C}},
{\tt{G}} and {\tt{T}}';
so, to be precise, we are assuming that the recognized sequence
contains $L$ non-wildcard
characters.)
% in a sequence whose length can be greater than $L$.)
Assuming the rest of the genome is `random', \ie, that the sequence
consists of random nucleotides {\tt{A}}, {\tt{C}},
{\tt{G}} and {\tt{T}} with equal probability -- which is obviously
untrue, but it shouldn't make too much difference to our calculation --
the chance of there being no other occurrence of the target sequence
in the whole genome, of length $N$ nucleotides, is roughly
\beq
(1 - (1/4)^L )^N \simeq \exp ( - N (1/4)^L ) ,
\eeq
which is close to one only if
\beq
N 4^{-L} \ll 1 ,
\eeq
that is,
\beq
L > \log N / \log 4 .
\eeq
Using $N= 3 \times 10^9$,
% from cell p.386
we require the recognized sequence to be longer than $L_{\min} = 16$
nucleotides.
What size of \ind{protein} does this imply?
%\ben
\bit
\item
%
A weak lower bound can be obtained by assuming that the information
content of the protein sequence itself is greater than
the information content of the \ind{nucleotide}
sequence the protein prefers to bind to (which we have argued above
must be at least 32 bits).
This gives a minimum protein length of $32 / \log_2(20) \simeq 7$
\ind{amino acid}s.
\item
Thinking realistically, the \ind{recognition} of the DNA sequence
by the protein presumably involves the protein coming into contact
with all sixteen nucleotides in the target sequence.
If the protein is a monomer, it must be big enough that it can
simultaneously make contact with sixteen nucleotides of DNA.
One helical turn of DNA containing ten nucleotides has a length of
3.4$\,$nm, so a contiguous sequence of sixteen nucleotides has a length
of 5.4$\,$nm. The diameter of the protein must therefore be about 5.4$\,$nm
or greater. Egg-white lysozyme is a small globular protein with
a length of 129 amino acids
% cell p.90
and a diameter of about 4$\,$nm.
% cell p.130.
Assuming that volume is proportional to sequence length
and that volume scales as the cube of the diameter, a protein of
diameter 5.4$\,$nm must have a sequence of length $2.5 \times 129
\simeq 324$ amino acids.
%\een
\eit
% \item
%
(b)
If, however, a target sequence consists of a twice-repeated sub-sequence,
we can get by with a much smaller protein that recognizes
only the sub-sequence, and that binds to the \ind{DNA} strongly only if
it can form a {\em\ind{dimer}},
both halves of which are bound to the recognized sequence.
% , which must appear twice in succession in the DNA.
% with a neighbour.
Halving the diameter of the protein, we now only need a protein whose length
is greater than 324/8 = 40 amino acids.
A protein of length smaller than this cannot by itself serve as a
regulatory protein\index{protein!regulatory} specific to one gene,
because it's simply too small to be able to make a sufficiently
specific match -- its available surface does not have enough
information content.
% \een
}
%
\dvips
%
% ch 8 LINEAR
%\chapter{Linear Error correcting codes and perfect codes}
%\chapter{Linear Error Correcting Codes and Perfect Codes \nonexaminable}
\prechapter{About Chapter}
% prechapter for linear codes / binary codes
In Chapters \ref{ch.prefive}--\ref{ch.ecc},
we established Shannon's noisy-channel coding theorem
for a general channel with any
input and output alphabets.
A great deal of attention in coding theory focuses on the special
case of channels with binary inputs.
Constraining ourselves to these channels simplifies
matters, and leads us into an exceptionally rich world,
which we will only taste in this book.
One of the aims of this chapter is to point out a
contrast between Shannon's aim of achieving reliable communication
over a noisy channel and the apparent aim of many in the
% this wonderful
world of \ind{coding theory}.\index{sphere packing}
Many coding theorists take as their fundamental problem
the task of packing as many spheres as possible, with radius
as large as possible, into an $N$-dimensional space, {\em with
no spheres overlapping}.
Prizes are awarded to people
who find packings that squeeze in an extra few spheres.
% of a given radius.
While this is a fascinating mathematical topic,
we shall see that the aim of maximizing
the \ind{distance} between codewords in a code has only a tenuous
relationship to Shannon's aim of reliable \ind{communication}.
\ENDprechapter
\chapter{Binary Codes \nonexaminable}
\label{ch.linearecc}
\label{ch.linear}
% see also linearblock.tex
%
% chapter 8: linear error correcting codes
%
% distance
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% see also NOTES.tex
We've established Shannon's noisy-channel coding theorem
for a general channel with any
input and output alphabets.
A great deal of attention in coding theory focuses on the special
case of
channels with binary inputs, the first implicit choice being the
binary symmetric channel.\index{channel!binary symmetric}
The optimal decoder for a code, given a binary symmetric channel,
finds the codeword that is closest to the received vector, closest\marginpar[b]{\small{{\sf Example:}\\[0.0012in]
%\begin{center}
\begin{tabular}{rl}
\multicolumn{2}{c}{
The Hamming distance
}\\
{between}& {\tt{00001111}}\\ and & {\tt{11001101}}\\
\multicolumn{2}{c}{
is 3.
}\\
\end{tabular}
%\end{center}
}}
in {\dem\ind{Hamming distance}}.\index{distance!Hamming}
The Hamming distance between two binary vectors is the number
of coordinates in which the two vectors
differ.
Decoding errors will occur
if the noise takes us from the transmitted codeword $\bt$ to a
received vector $\br$ that is closer to some other codeword.
The {\dem{distances\/}} between codewords are thus relevant to the
probability of a decoding error.\index{distance!of code}
\section{Distance properties of a code}
%\begin{description}
%\item[The {\dem{distance}\/} of a\index{distance!of code} code]
The {\dem{distance}\/} of a\index{distance!of code}
% \index{error-correcting code!distance}
code is the smallest separation between two of its
codewords.
% \end{description}
% \begin{ indented
\exampl{ex.hamm74dist}{
%\noindent {\sf Example:}
The $(7,4)$ Hamming code (\pref{sec.ham74})
has distance $d= 3$. All pairs of its
codewords differ in at least 3 bits.
The maximum number of errors it can correct is $t=1$;
in general
a code with distance $d$ is
$\lfloor (d\!-\!1)/2 \rfloor$-error-correcting.
}
% , and
% the distance is related to this quantity by
% $d=2t+1$.
% \end{indented
A more precise term for distance is
the {\dem\ind{minimum distance}\/} of the code.
The distance of a code is often denoted by $d$ or $d_{\min}$.
%
% \section{Weight enumerator function}
% see code/bucky/README
\index{error-correcting code!weight enumerator}%
%\index{error-correcting code!distance distribution}%
We'll now constrain our attention to linear codes.
In a linear code, all codewords have identical
distance properties, so we can summarize
% the dis.
% are equivalent,
% from the point of view of the spectrum of
% distances to other codewords.
% summarizes
all the distances between the code's codewords
by counting the distances from the all-zero codeword.
%\begin{description}
%\item[The {\dem\ind{weight enumerator} function} of a code,] $A(w)$,
The {\dem\ind{weight enumerator} function} of a code, $A(w)$,
% $A(w)$
is defined to be the number of codewords in the code that
have weight $w$.
\amarginfig{b}{%
\footnotesize
\begin{tabular}{c}
\raisebox{0.2in}{\buckypsfig{H74.eps}}
\\
%# weight enumerator of $(7,4)$ code
%# w A(w) C Random Random N-choose-w
\begin{tabular}[b]{rr}
\toprule
$w$ & $A(w)$ \\ \midrule
%%%%%%%%%%%%%%%%%%%%%%%%%%%%
0 & 1 \\
3 & 7 \\
4 & 7 \\
7 & 1 \\ \midrule
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%55
Total & 16\\ \bottomrule
\end{tabular}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\\
\buckypsgraphb{H74.Aw.ps}
\end{tabular}
% see /home/mackay/code/bucky/H74.gnu
\caption[a]{The graph of the $(7,4)$ Hamming code, and its weight enumerator
function.}
\label{fig.wef.h74}
}
%
The weight enumerator function is also
known as the {\dem{{distance distribution}\index{distance!distance distribution}}\/} of the code.
%\end{description}
% original is in graveyard.tex
\begin{figure}
\figuremargin{%
\footnotesize
\begin{tabular}{ccc}
\buckypsfig{dodec.eps}
&
%# weight enumerator of (30,11) code dodec2.G
%# w A(w) C Random Random N-choose-w
\begin{tabular}{rr}
\toprule
$w$ & $A(w)$ \\ \midrule
%%%%%%%%%%%%%%%%%%%%%%%%%%%%
0 & 1 \\
5 & 12 \\
8 & 30 \\
9 & 20 \\
10 & 72 \\
11 & 120 \\
12 & 100 \\
13 & 180 \\
14 & 240 \\
15 & 272 \\
16 & 345 \\
17 & 300 \\
18 & 200 \\
19 & 120 \\
20 & 36 \\ \midrule
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%55
Total & 2048\\ \bottomrule
\end{tabular}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
&
\begin{tabular}{@{}c@{}}
\buckypsgraphB{dodec2.Aw.ps}
\\
\buckypsgraphB{dodec2.Aw.l.ps}
\end{tabular}
\\% see /home/mackay/code/bucky
\end{tabular}
}{
\caption[a]{
The graph defining the $(30,11)$
\ind{dodecahedron code}\index{error-correcting code!dodecahedron}
% first introduced in secref{sec.dodecahedron}
(the circles are the 30 transmitted bits and the triangles are the 20 parity checks,
one of which is redundant) and the
% (b-ii) The
weight enumerator function (solid lines). The
dotted lines show the
average weight enumerator function of all random linear codes
with the
same size of generator matrix,
% (dotted lines),
which will be computed shortly.
The lower
figure shows the same functions on a log scale.
%%%%%%%%%%%%%%% CHECK %%%%%%%%%%%%%%%
% {\em (Check for cross-reference to earlier occurrence?)}
}
\label{fig.Aw}
}
\end{figure}
% \begin{ indented ?
\exampl{ex.hamm74Aw}{
% \noindent {\sf Example:}
The weight enumerator functions
of the $(7,4)$ Hamming code and the \ind{dodecahedron code}\index{error-correcting code!dodecahedron}
are shown in figures \ref{fig.wef.h74} and \ref{fig.Aw}.
% \end{indented
}
\section{Obsession with distance}
Since the maximum number of errors that a code can {\em guarantee\/} to correct,
$t$, is related to its distance $d$ by
$t= \lfloor (d\!-\!1)/2 \rfloor$,\marginpar{\small{%
$d=2t+1$ if $d$ is odd, and\\
$d=2t+2$
if $d$ is even.}}
many coding theorists focus on the\index{distance!of code} distance of a code, searching for
codes of a given size that have the biggest possible distance.
Much of practical coding theory has focused on
decoders that give the optimal decoding for all error patterns
of weight up to the half-distance $t$ of their codes.
\begin{description}
\item[A \ind{bounded-distance decoder}]\index{decoder!bounded-distance}
is a decoder that returns the closest codeword to a received\label{sec.bdd}
binary vector $\br$ if the distance from $\br$ to that codeword
is less than or equal to $t$; otherwise it returns a failure
message.
\end{description}
The rationale for not trying to decode when more than $t$
errors have occurred might be `we can't {\em guarantee\/}
that we can correct more than $t$ errors, so we
won't bother trying -- who would
be interested in a decoder that corrects some\index{sermon!worst-case-ism}
error patterns of weight greater than $t$, but not others?'
This defeatist attitude is an example of {\dem\ind{worst-case-ism}},
a widespread mental ailment
% yes, spell checked
which this book is intended to cure.
The fact is that bounded-distance decoders cannot reach the\wow\
Shannon limit of the binary symmetric channel; only a decoder
that often corrects more than $t$ errors can do this.
The state of the art in error-correcting codes
have decoders that work way beyond the minimum distance
of the code.
\subsection{Definitions of good and bad distance properties}
\index{distance!of code!good/bad}Given
a family of codes of increasing blocklength $N$, and with rates
approaching a limit $R>0$,
we may be able to put that family in one of the following categories,
which have some similarities to the categories of `good' and `bad' codes
defined earlier (\pref{sec.bad.code.def}):\index{error-correcting code!good}\index{error-correcting code!bad}\index{error-correcting code!very bad}\index{distance!good}\index{distance!bad}\index{distance!very bad}
\label{sec.bad.dist.def}
\begin{description}
\item[A sequence of codes has `good' distance]
if $d/N$ tends to a constant greater than zero.
\item[A sequence of codes has `bad' distance]
if $d/N$ tends to zero.
\item[A sequence of codes has `very bad' distance]
if $d$ tends to a constant.
\end{description}
% THIS really belongs over the page
\amarginfig{b}{
\begin{center}
\mbox{
\psfig{figure=/home/mackay/itp/figs/gallager/16.12.G.ps,width=2in,angle=-90}
}\end{center}
\caption[a]{The graph of a rate-\dfrac{1}{2}
low-density generator-matrix code. The rightmost $M$
of the transmitted bits are each connected to a single distinct parity
constraint.
}
\label{fig.ldgmc}
}
\exampl{example.badcode}{
A {\dem\ind{low-density generator-matrix code}\/} is a
linear code whose $K \times N$ generator matrix
$\bG$ has a small number $d_0$ of {\tt{1}}s per row,
regardless of how big $N$ is.
The minimum distance of such a code is at most $d_0$,
so {low-density generator-matrix code}s have `very bad' distance.
}
While having large distance is no bad thing, we'll see, later on, why
an emphasis on distance can be unhealthy.
\begin{figure}[htbp]
\figuremargin{
\mbox{\psfig{figure=figs/caveperfect.ps,angle=-90,width=3in}}
}{
\caption[a]{Schematic picture of part of
Hamming space perfectly filled
by $t$-spheres centred on the codewords of a perfect code.}
\label{fig.caveperfect}
}
\end{figure}
\section{Perfect codes}
A $t$-sphere (or a sphere of radius $t$)
in Hamming space, centred on a point $\bx$,
is the set of points whose Hamming distance from $\bx$
is less than or equal to $t$.
The $(7,4)$ \ind{Hamming code}\index{perfect code}\index{error-correcting code!perfect} has the beautiful property that
if we place 1-spheres
% of radius 1
about each of its 16 codewords,
those spheres perfectly fill Hamming space without overlapping.
As we saw in \chref{ch1},
every binary vector of length 7 is within a distance of $t=1$ of
exactly one codeword of the Hamming code.
\begin{description}
\item[A code is a perfect $t$-error-correcting code]
if the set of $t$-spheres centred on the codewords of the code fill the
Hamming space without overlapping. (See \figref{fig.caveperfect}.)
\end{description}
Let's recap our cast of characters.
The number of codewords is $S=2^K$. The number of points in the
entire Hamming space is $2^N$. The number of points in a
Hamming sphere of radius $t$ is
\beq
\sum_{w=0}^{t} {{N}\choose{w}} .
\eeq
For a code to be perfect with these parameters, we
require $S$ times the number of points in the $t$-sphere to equal $2^N$:
\beqan
\mbox{for a perfect code, } \:\:
2^K \sum_{w=0}^{t} {{N}\choose{w}} & =& 2^N
\\
\mbox{or, equivalently, }\:\:
\sum_{w=0}^{t} {{N}\choose{w}} & =& 2^{N-K} .
\eeqan
For a perfect code, the number of noise vectors
in one sphere must equal the number of possible syndromes.
The $(7,4)$ Hamming code satisfies this numerological condition\index{numerology}
because
\beq
1 + {{7}\choose{1}} = 2^3 .
\label{eq.coincidence}
\eeq
% Interestingly, the first appearance of the ternary Golay code predated
%Golay's publication by a good year. A Finnish devotee of football pools thought it up in list form (!) and published it
% in 1947.
% Covering codes.
%G.Cohen, I.Honkala. S.Litsyn, and A.Lobstein
%North-Holland Publishing Co., Amsterdam, 1997. xxii+542 pp. ISBN 0-444-82511-8
% It is this "ternary" Golay code which was first discovered by a Finn who was
% determining good strategies for betting on blocks of 11 soccer games. Here,
% one places a bet by predicting a Win, Lose, or Tie for all 11 games, and as
% long as you do not miss more than two of them, you get a payoff. If a group
% gets together in a "pool" and makes multiple bets to "cover all the options"
% (so that no matter what the outcome, somebody's bet comes within 2 of the
% actual outcome), then the codewords of a 2-error-correcting perfect code
% provide a very nice option; the balls around its codewords fill all of the
% space, with none left over.
%
% It was in this vein that the ternary Golay code was first constructed; its
% discover, Juhani Virtakallio, exhibited it merely as a good betting system
% for football-pools, and its 729 codewords appeared in the football-pool
% magazine Veikkaaja. For more on this, see Barg's article [1].
%
% [1] Barg, Alexander. "At the Dawn of the Theory of Codes," The Mathematical
% Intelligencer, Vol. 15 (1993), No. 1, pp. 20--26.
\subsection{How happy we would be to use perfect codes}
If there were large numbers of perfect codes to choose from,
with a wide range of blocklengths and rates,
then these would be the perfect solution to Shannon's problem.
We could communicate over a binary symmetric channel with noise
level $f$, for example, by picking a perfect $t$-error-correcting code
with blocklength $N$ and $t=f^* N$, where $f^* = f + \delta$
and $N$ and $\delta$ are chosen such that the probability that
the noise flips more than $t$ bits is satisfactorily small.
However, {\em there are almost no perfect codes}.\wow\
The only nontrivial
perfect binary
codes are
\ben
\item
the Hamming codes, which are perfect codes with $t=1$
% -error-correcting with
and blocklength $N=2^M-1$,
defined below; the rate of a \ind{Hamming code} approaches 1
as its blocklength $N$ increases;
\item
the repetition codes of odd blocklength $N$, which are perfect codes
with $t=(N-1)/2$; the rate of repetition codes goes to zero as $1/N$; and
\item
one remarkable $3$-error-correcting code with $2^{12}$
codewords of
blocklength $N=23$ known
as the binary \ind{Golay code}\index{error-correcting code!Golay}.
[A second 2-error-correcting Golay code of
length $N=11$ over a
ternary alphabet was
% 729 cw's in football-pool magazine Veikkaaja.
discovered by a Finnish football-pool
enthusiast\index{football pools}\index{bet}\index{design theory}
% \index{Finland}
called Juhani Virtakallio\index{Virtakallio, Juhani} in 1947.]
% 1+23+23*11 + 23*11*7 = 2048
% If we allow more symbols in our alphabet than just 0 and 1, then we get analogues of the
% Hamming codes, and another Golay code of length 11, this time on three letters (say 0, +, and -) and with parameters (11,
% 3^6, 5). This completes the list of all linear perfect codes. parameters (11,3^6, 5).
% http://lev.yudalevich.tripod.com/ECC/betting.html
%
% [1] Barg, Alexander. "At the Dawn of the Theory of Codes," The Mathematical Intelligencer, Vol. 15 (1993), No. 1, pp.
% 20--26.
\een
There are no other binary perfect codes.
Why this shortage of perfect codes?
Is it because precise numerological coincidences like those satisfied by the parameters
of the Hamming code (\ref{eq.coincidence}) and the Golay
code,
\beq
1 + {{23}\choose{1}} + {{23}\choose{2}} + {{23}\choose{3}} = 2^{11},
\eeq
are rare? Are there plenty of `almost-perfect' codes for which
the $t$-spheres fill {\em almost\/} the whole space?
No. In fact, the picture
of Hamming spheres centred on the
codewords {\em{almost}\/} filling Hamming space (\figref{fig.cavenotquite})
is a misleading one: for most codes, whether
they are good codes or bad codes,\index{sermon!sphere-packing}
%
almost all the Hamming space is taken up by the space {\em{between}\/}
$t$-spheres
% \wow\
(which is shown in grey in \figref{fig.cavenotquite}).
\begin{figure}
\figuremargin{
\mbox{\psfig{figure=figs/cavenotquite.ps,angle=-90,width=3in}}
}{
\caption[a]{Schematic picture of Hamming space not perfectly filled
by $t$-spheres centred on the codewords of a code.
The grey regions show points that are at a Hamming distance
of more than $t$ from any codeword. This is a misleading picture,
as, for any code with large $t$
in high dimensions, the grey space between the
spheres takes up almost all of Hamming space.
}
\label{fig.cavenotquite}
}
\end{figure}
Having established this gloomy picture, we spend a moment
filling in the properties of the perfect codes mentioned
above.
\subsection{The Hamming codes}
The $(7,4)$ Hamming code can be defined as the linear code
whose $3\times 7$ parity-check matrix contains, as its columns,
all the 7 ($=2^3-1$) non-zero vectors of length 3.
Since these 7 vectors are all different, any single bit-flip
produces a distinct syndrome, so all single-bit errors
can be detected and corrected.
% from \input{tex/_concat2.tex}
We can generalize this code, with $M=3$ parity constraints,
as follows.
The Hamming codes are single-error-correcting codes
defined by picking a number of parity-check constraints, $M$;
the blocklength $N$ is $N = 2^M-1$; the parity-check matrix
contains, as its columns, all the $N$ non-zero vectors
of length $M$ bits.
The first few Hamming codes have the following rates:
\medskip% added because of my change to the center environment
\begin{center}
\begin{tabular}{cr@{,$\,$}llp{1.4in}} \toprule
% checks &
%% (block length, source bits)
% & rate & \\
% $M$ & ($N = 2^M-1$ , $K = N - M$) & $R=K/N$ & \\ \midrule
\multicolumn{1}{c}{Checks, $M$} & \multicolumn{2}{c}{($N,K$)} & $R=K/N$ & \\ \midrule
2 & (3&1) & 1/3 & repetition code $R_3$ \\
3 & (7&4) & 4/7 & $(7,4)$ Hamming code \\
4 & (15&11) & 11/15 & \\
5 & (31&26) & 26/31 & \\
6 & (63&57) & 57/63 & \\ \bottomrule
\end{tabular}
\end{center}
\exercissxA{2}{ex.HammingP}{
What is the probability of block error of the $(N,K)$ Hamming
code to leading order, when the code
is used for a binary symmetric channel with noise density $f$?
}
\section{Perfectness is unattainable -- first proof \nonexaminable}
We will show in several ways
that useful \ind{perfect code}s do not exist (here,
`useful' means `having large blocklength $N$, and rate
close neither to 0 nor 1').
% First, let's study a pithy, no-nonsense example.
Shannon proved that, given a binary symmetric channel
with any noise level $f$, there exist codes with large blocklength $N$
and rate as close as you like to $C(f) = 1 - H_2(f)$
that enable \ind{communication} with
arbitrarily small error probability.
For large $N$, the number of errors per block will typically
be about $\fN$, so these codes of Shannon are
`almost-certainly-$\fN$-error-correcting'
codes.
Let's pick the special case of a noisy channel with $f \in ( 1/3, 1/2)$.
Can we find a large
{\em perfect\/} code that is $\fN$-error-correcting?
% with large blocklength for this channel?
Well, let's suppose that such a code has been found, and examine
just three of its codewords. (Remember that the code
ought to have rate $R \simeq 1-H_2(f)$, so it should have
an enormous number ($2^{NR}$) of codewords.)
\begin{figure}
\figuremargin{
\mbox{\psfig{figure=figs/noperfect3.ps,%
width=64mm,angle=-90}}
}{%
\caption[a]{Three
codewords.
}
\label{fig.noperfect}
}
% load 'gnuR'
\end{figure}
Without loss of generality, we choose one of the codewords
to be the all-zero codeword and define the other two to have
overlaps with it as shown in \figref{fig.noperfect}.
The second codeword differs from the first in a fraction $u+v$
of its coordinates.
The third codeword differs from the first in a fraction $v+w$,
and from the second in a fraction $u+w$. A fraction $x$
of the coordinates have value zero in all three codewords.
Now, if the code is $\fN$-error-correcting, its minimum distance
must be greater than $2\fN$, so
\beq
u+v > 2f, \:\:\: v+w > 2f, \:\:\: \mbox{and} \:\:\: u+w > 2f .
\eeq
Summing these three inequalities and dividing by two, we have
\beq
u +v+w > 3f .
\eeq
So if $f>1/3$, we can deduce $u+v+w > 1$, so that $x<0$,
which is impossible. Such a code cannot exist.
So the code cannot have {\em three\/} codewords, let alone
$2^{NR}$.
We conclude that, whereas Shannon proved there
are plenty of codes for communicating over
a \ind{binary symmetric channel}\index{channel!binary symmetric}\index{perfect code}
with $f>1/3$, {\em there are no perfect codes\index{error-correcting code!perfect}
that can do this.}
We now study a more general argument that indicates
that there are no large perfect linear codes for general rates (other than 0 and 1).
We do this by finding the typical distance of a random linear code.
%\mynewpage
\section{Weight enumerator function of random linear codes \nonexaminable}
\label{sec.wef.random}
Imagine
% H=rand(12,24)>0.5
% octave
\marginfig{\tiny{
\[%\mbox{\footnotesize{$\bH=$}}
\hspace{-2mm}\begin{array}{c}
{N}\\
\overbrace{\left.\hspace{-2mm}\left[\begin{array}{@{}*{24}{c@{\hspace{0.45mm}}}}
1&0&1&0&1&0&1&0&0&1&0&0&1&1&0&1&0&0&0&1&0&1&1&0\\
0&0&1&1&1&0&1&1&1&1&0&0&0&1&1&0&0&1&1&0&1&0&0&0\\
1&0&1&1&1&0&1&1&1&0&0&1&0&1&1&0&0&0&1&1&0&1&0&0\\
0&0&0&0&1&0&1&1&1&1&0&0&1&0&1&1&0&1&0&0&1&0&0&0\\
0&0&0&0&0&0&1&1&0&0&1&1&1&1&0&1&0&0&0&0&0&1&0&0\\
1&1&0&0&1&0&0&0&1&1&1&1&1&0&0&0&0&0&1&0&1&1&1&0\\
1&0&1&1&1&1&1&0&0&0&1&0&1&0&0&0&0&1&0&0&1&1&1&0\\
1&1&0&0&1&0&1&1&0&0&0&1&1&0&1&0&1&1&1&0&1&0&1&0\\
1&0&0&0&1&1&1&0&0&1&0&1&0&0&0&0&1&0&1&1&1&1&0&1\\
0&1&0&0&0&1&0&0&0&0&1&0&1&0&1&0&0&1&1&0&1&0&1&0\\
0&1&0&1&1&1&1&1&0&1&1&1&1&1&1&1&1&0&1&1&1&0&1&0\\
1&0&1&1&1&0&1&0&1&0&0&1&0&0&1&1&0&1&0&0&0&0&1&1
\end{array}\right]\right\} M \hspace{-2mm}\hspace{-0.25in} }
\end{array} \]
}
\caption[a]{A random binary parity-check matrix.}
\label{fig.randommatrix}
}%
making a code by picking the binary entries
in the $M \times N$ parity-check matrix $\bH$ at random.\index{error-correcting code!random linear}
What weight enumerator function should we expect?
The \ind{weight enumerator} of one particular code with
parity-check matrix $\bH$, $A(w)_{\bH}$, is
the number of codewords of weight $w$, which
can be written
\beq
A(w)_{\bH} = \sum_{\bx: |\bx| = w} \truth\! \left[ \bH \bx = 0 \right] ,
\eeq
where the
sum is over all vectors $\bx$ whose weight is $w$ and
the \ind{truth function} $\truth\! \left[ \bH \bx = 0 \right]$
equals one if
% it is true that
$\bH \bx = 0$
and zero otherwise.
We can find the expected value of $A(w)$,
\beqan
\langle A(w) \rangle &=& \sum_{\bH} P(\bH) A(w)_{\bH}
\\
&=& \sum_{\bx: |\bx| = w} \sum_{\bH} P(\bH) \,
\truth\! \left[ \bH \bx \eq 0 \right]
,
\label{eq.expAw}
\eeqan
by evaluating the probability
that a particular word of weight $w>0$ is a codeword of the code (averaging
over all binary linear codes in our ensemble).
By symmetry, this probability depends only on the weight $w$ of the word,
not on the details of the word.
The probability that the entire syndrome
$\bH \bx$ is zero can be found by multiplying together
the probabilities that each of the $M$ bits in the syndrome
is zero. Each bit $z_m$ of the syndrome is a sum (mod 2)
of $w$ random bits, so the probability that $z_m \eq 0$ is $\dhalf$.
The probability that $\bH \bx \eq 0$ is thus
\beq
\sum_{\bH} P(\bH) \, \truth\! \left[ \bH \bx \eq 0 \right]
= (\dhalf)^M = 2^{-M},
\eeq
independent of
$w$.
The expected number of words of weight $w$ (\ref{eq.expAw})
is given by summing, over all words of weight $w$, the probability
that each word is a codeword.
The number of words of weight $w$ is ${{N}\choose{w}}$,
so
\beq
\langle A(w) \rangle = {{N}\choose{w}} 2^{-M} \:\:\mbox{for any $w>0$}.
\eeq
For large $N$, we can use $\log {{N}\choose{w}}
\simeq N H_2(w/N)$ and $R\simeq 1-M/N$ to write
\beqan
\log_2 \langle A(w) \rangle &\simeq& N H_2(w/N) -M
\\
&\simeq& N [ H_2(w/N) - (1-R) ] \:\:\mbox{for any $w>0$}.
\label{eq.wef.random}
\eeqan
As a concrete example, \figref{fig.Aw.540} shows the
expected weight enumerator function of a rate-$1/3$
random linear code\index{error-correcting code!random linear} with $N=540$ and $M=360$.
\marginfig{
\begin{center}
\mbox{%
\small
\hspace{-0.01in}%
\begin{tabular}{c}
\hspace{-0.15in}\mbox{\psfig{figure=/home/mackay/_doc/code/gallager/Am540R.ps,%
width=41.5mm,angle=-90}}\\[-0.01in]
\hspace{0.1in}\mbox{\hspace*{-0.35in}\psfig{figure=/home/mackay/_doc/code/gallager/Am540Rl.ps,%
width=41.5mm,angle=-90}}\\[-0.1in]
\end{tabular}
}
\end{center}
%}{%
\caption[a]{The
expected weight enumerator function
$\langle A(w) \rangle$ of a
\index{error-correcting code!random linear}random linear code with $N=540$ and $M=360$. Lower figure shows
$\langle A(w) \rangle$ on a logarithmic scale.
}
\label{fig.Aw.540}
% load 'gnuR'
}
\subsection{Gilbert--Varshamov distance}
For weights $w$ such that $H_2(w/N) < (1-R)$, the expectation
of $A(w)$ is smaller than 1; for weights such that $H_2(w/N) > (1-R)$,
the expectation is greater than 1. We thus expect, for large $N$,
that the minimum distance
of a random linear code will be close to the distance $d_{\rm GV}$
defined by
\beq
H_2(d_{\rm GV}/N) = (1-R) .
\label{eq.GV.def}
\eeq
% INDENT ME?
\noindent
{\sf Definition.}
This distance, $d_{\rm GV} \equiv N H_2^{-1}(1-R)$,
is
% known as
the
{\dem{Gilbert--Varshamov\index{distance!Gilbert--Varshamov}\index{Gilbert--Varshamov distance}
distance}\/}
for rate $R$ and blocklength $N$.
The {\dem{Gilbert--Varshamov conjecture}},
widely believed, asserts that (for large $N$) it is not possible to\index{Gilbert--Varshamov conjecture}
create binary codes with minimum distance significantly greater than $d_{\rm GV}$.
\medskip
% INDENT ME?
\noindent
{\sf Definition.}
The {\dem{\index{Gilbert--Varshamov rate}Gilbert--Varshamov rate}\/} $R_{\rm GV}$
is the maximum rate at which you can reliably
communicate with a \ind{bounded-distance decoder} (as defined
on \pref{sec.bdd}),
assuming that the
Gilbert--Varshamov conjecture\index{Gilbert--Varshamov conjecture}
is true.
% \section{Perfect codes} A \index{error-correcting code!perfect}\see{perfect code}{code}
\subsection{Why sphere-packing is a bad perspective, and an obsession with
distance is inappropriate}
If one uses a \ind{bounded-distance decoder},\index{sermon!sphere-packing}
the maximum tolerable noise level will flip a fraction $f_{\rm bd} = \half d_{\min}/N$
of the bits. So, assuming $d_{\min}$ is equal to the \index{Gilbert--Varshamov distance}Gilbert distance
$d_{\rm GV}$ (\ref{eq.GV.def}), we have:%
\amarginfig{b}{
\begin{center}
\mbox{\psfig{figure=figs/RGV.ps,angle=-90,width=1.7in}}\\[-0.1in]
$f$
\end{center}
\caption[a]{Contrast between Shannon's channel capacity $C$
and the Gilbert rate $R_{\rm GV}$ --
the maximum communication rate
achievable using a \ind{bounded-distance decoder}, as a function
of noise level $f$.
For any given rate, $R$, the maximum tolerable
noise level for Shannon is twice as big as the
maximum tolerable noise level for a `worst-case-ist'
who uses a bounded-distance decoder.
}
}
\beq
H_2(2 f_{\rm bd}) = (1-R_{\rm GV}) .
\label{eq.idiotf}
\eeq
\beq
R_{\rm GV} = 1 - H_2(2 f_{\rm bd}).
\eeq
Now, here's the crunch: what did Shannon say is achievable?\index{Shannon, Claude}
He said the maximum possible rate of communication is the capacity,
\beq
C = 1 - H_2(f) .
\eeq
So for a given rate $R$,
the maximum tolerable noise level, according to Shannon,
is given by
\beq
H_2(f) = (1-R) .
\label{eq.shannonf}
\eeq
Our conclusion: imagine a good code of rate $R$ has been chosen;
equations (\ref{eq.idiotf}) and (\ref{eq.shannonf})
respectively define
the maximum noise levels tolerable
by a bounded-distance
decoder, $f_{\rm bd}$, and by Shannon's decoder, $f$.
\beq
f_{\rm bd} = f/2 .
\eeq
Bounded-distance decoders can only ever cope with
{\em half\/} the noise-level that Shannon proved is tolerable!
% Need to show implication for perfect codes at the same time.
How does this relate to perfect\index{error-correcting code!perfect}
codes? A code is perfect
if there are $t$-spheres around its codewords that
fill Hamming space without overlapping.
But when a typical random linear code is used to
communicate over a binary symmetric channel near to the
Shannon limit, the typical number of bits flipped
is $\fN$, and the minimum distance between codewords is
also $\fN$, or a little bigger, if we are
a little below the Shannon limit.
So the $\fN$-spheres around the codewords overlap
with each other sufficiently that each sphere almost contains
the centre of its nearest neighbour!
\marginfig{\begin{center}
\mbox{\psfig{figure=figs/overlap.eps,width=1.7in}}\\[-0.02in]
\end{center}
\caption[a]{Two overlapping spheres whose radius
is almost as big as the distance between their centres.
}
\label{fig.overlap}
}
The reason why this overlap is not disastrous is because,
in high dimensions, the volume associated with the overlap,
shown shaded in \figref{fig.overlap}, is a tiny fraction of
either sphere, so the probability of landing in it is
extremely small.
The moral of the story is that \ind{worst-case-ism} can be bad for you,
halving your ability to tolerate noise.
You have to be able to decode {\em way\/} beyond the minimum distance of a code
to get to the Shannon limit!
Nevertheless, the minimum
distance of a code is of interest in practice, because, under some
conditions, the minimum distance dominates the errors made by
a code.
% On to the bat cave. (Could also dissect the random code
% in more detail.)
\section{Berlekamp's bats}
\label{sec.bats}
A blind \ind{bat}\index{Berlekamp, Elwyn} lives in a cave.
It flies about the centre of the cave, which corresponds to
one codeword,
with its typical distance from the centre controlled by
a friskiness parameter $f$. (The displacement of the
bat from the centre corresponds to the noise vector.)
The boundaries of the cave are made up of stalactites that
point in towards the centre of the cave (\figref{fig.cavereal}). Each stalactite
is analogous to the boundary between the home codeword
and another codeword. The stalactite is
like the shaded region in \figref{fig.overlap},
but reshaped to convey the idea that it is a region of very small volume.
Decoding errors correspond to the bat's intended trajectory passing
inside a stalactite. Collisions with stalactites at various distances
from the centre are possible.
If the friskiness
% (noise level)
is very small, the bat is usually very close to the
centre of the cave;
collisions will be rare,
and when they do occur, they will usually involve the
stalactites whose tips are closest to the centre point. Similarly,
under low-noise conditions, decoding errors will be rare,
and they will typically involve low-weight codewords. Under low-noise
conditions, the minimum distance of a code is relevant to
the (very small) probability of error.
\begin{figure}[hbtp]
\figuremargin{
\mbox{\psfig{figure=figs/cavereal.ps,angle=-90,width=3in}}
}{
\caption[a]{Berlekamp's schematic picture of Hamming space in
the vicinity of a codeword. The jagged solid line encloses all points to which
this codeword is the closest.
The $t$-sphere around the
codeword takes up a small fraction of this space.
}
\label{fig.cavereal}
}
\end{figure}
If the friskiness is higher, the bat may often make excursions
beyond the safe distance $t$ where the longest stalactites start,
but
% it is quite possible that
it will collide most frequently
with more distant stalactites, owing to their greater number.
There's only a tiny number of \ind{stalactite}s at the minimum
distance, so they are relatively unlikely to cause the errors.
Similarly, errors in a real
error-correcting code
depend on the properties of the \ind{weight enumerator} function.
At very high friskiness, the \ind{bat} is always a long way from the centre of
the \ind{cave}, and almost all its collisions involve contact with distant stalactites.
% bat in a cave.
Under these conditions,
the bat's collision frequency has nothing to do with
the distance from the centre to the closest stalactite.
%\section{Concatenation}
% see also _concat.tex
% this is the bit where we do the ``hamming are good'' story
\section{Concatenation of Hamming codes\nonexaminable}
\label{sec.concatenation}
It is instructive to play some more with the \ind{concatenation} of
\ind{Hamming code}s,\index{error-correcting code!Hamming}
a concept we first visited in \figref{fig.concath1},
because we will get insights into the notion of good codes
and the relevance or otherwise of the \ind{minimum distance} of a code.\index{distance!of code}
We can create a concatenated code
for a binary symmetric channel with noise density $f$
by encoding with
several Hamming codes in succession.
% /home/mackay/bin/concath.p~
% /home/mackay/_courses/itprnn/hamming/concath
% /home/mackay/_courses/itprnn/hamming/concath.gnu
The table recaps the key properties of
the Hamming codes, indexed by number of constraints, $M$.
All the Hamming codes have minimum distance $d=3$
and can correct one error in $N$.
\medskip% because of modified center
\begin{center}
\begin{tabular}{ll}\toprule
$N = 2^M-1$
& blocklength
\\
% $K$ &
$K = N - M$ & number of source bits \\
$p_{\rm B} = \smallfrac{3}{N} {{N} \choose {2}} f^2$
& probability of block error to leading order \\ \bottomrule
% $R$ & $K/N$ \\
\end{tabular}
\medskip
\end{center}
\marginfig{
\begin{center}
%\mbox{%
\footnotesize
\raisebox{0.3591in}{$R$}%
\hspace{0.2in}%
\begin{tabular}{c}
\mbox{\psfig{figure=hamming/concath.rate.ps,%
width=40.5mm,angle=-90}}\\[0.1in]
\hspace{0.3in}$C$
\end{tabular}
%}
\end{center}
%}{%
\caption[a]{The rate $R$ of the concatenated Hamming code
as a function of the number of concatenations, $C$.
}
\label{fig.concath.rate}
}
%
% \subsection{Proving that good codes can be made by concatenation}
If we make a \ind{product code} by\index{error-correcting code!good}\index{error-correcting code!product code}
concatenating a sequence of $C$ Hamming codes with increasing $M$,
we can choose those parameters $\{ M_c \}_{c=1}^{C}$
in such a way that the rate of the product
code
% $R_C$
\beq
R_C = \prod_{c=1}^C \frac{N_c - M_c}{N_c}
\eeq
tends to a non-zero limit as $C$ increases.
For example, if we set $M_1 =2$, $M_2=3$, $M_3=4$, etc.,
then the asymptotic rate is 0.093 (\figref{fig.concath.rate}).
The blocklength $N$ is a rapidly-growing function of $C$, so these codes
are somewhat impractical.
A further weakness of these codes is\index{distance!of concatenated code}
that\index{error-correcting code!concatenated}
their\index{error-correcting code!product code}
minimum distance is not very good (\figref{fig.concath.n.d}).%
\amarginfig{b}{
\begin{center}
\small\footnotesize
%
%\hspace{0.042in}%
%\begin{tabular}{c}
%\mbox{\psfig{figure=hamming/concath.n.k.l.ps,%
%width=40.5mm,angle=-90}}
%% \\[-0.1in] $C$
%\end{tabular}\\[0.13in]
\hspace*{0.2042in}%
\begin{tabular}{c}
\mbox{\psfig{figure=hamming/concath.n.d.ps,%
width=40.5mm,angle=-90}}\\[0.1in]
\hspace{0.3in}$C$\\[-0.05in]
\end{tabular}
\end{center}
%}{%
\caption[a]{The blocklength $N_C$ (upper curve)
and
% $(N,K)$ (upper figure) and
minimum distance $d_C$ (lower curve)
% (lower figure)
of the concatenated Hamming code
as a function of the number of concatenations $C$.
}
\label{fig.concath.n.k.l}
\label{fig.concath.n.d}
}
%
% why is this fig not taking up its correct space?
%
% The blocklength $N$ is a rapidly growing function of $C$, so these codes
% are mainly of theoretical interest.
%
Every one of the constituent
Hamming codes has
\ind{minimum distance}\index{distance!of code} 3, so the minimum
distance of the $C$th product is $3^C$. The blocklength $N$ grows faster
% with $C$
than $3^C$, so the ratio $d/N$ tends to zero as $C$ increases. In contrast,
for typical random codes, the ratio $d/N$ tends to a constant\index{random code}
% distance tends to a fraction of $N$,
such that $H_2(d/N) = 1-R$.\index{Hamming code}
Concatenated Hamming codes\index{distance!bad}\index{distance!of product code}
thus have `bad' distance.% \pref{distance.defs}
Nevertheless, it turns out that this simple sequence of codes
yields good codes\index{error-correcting code!good} for some channels -- but
not very good codes
(see \sectionref{sec.good.codes} to recall the definitions of the terms
`good' and `very good').
Rather than prove this result, we will simply explore it numerically.
\Figref{fig.concath.rateeb} shows the bit error probability $p_{\rm b}$
of the concatenated
codes assuming that the constituent codes are decoded in sequence,
as described in section \ref{sec.concatdecode}. [This one-code-at-a-time
decoding is suboptimal, as we saw there.]
% refers to {tex/_concat.tex}% contains simple example
%
% concath.p
The horizontal axis shows the rates of the codes.
As the number of concatenations increases, the rate drops
to 0.093 and the error probability drops towards zero.
The channel assumed in the figure is the binary
symmetric channel with $\q=0.0588$. This is the highest noise level that
can be tolerated using this concatenated code.
\amarginfig{c}{
\begin{center}
\footnotesize
\mbox{%
\raisebox{0.591in}{$p_{\rm b}$}%
\hspace{0.2042in}%
\begin{tabular}{c}
\mbox{\psfig{figure=hamming/concath.rate.058.ps,%
width=40mm,angle=-90}}\\[0.1in]
\hspace{0.54in}$R$\\[-0.03in]
\end{tabular}}
\end{center}
%}{%
\caption[a]{The bit error probabilities versus the rates
$R$
of the concatenated Hamming codes, for the binary
symmetric channel with $\q=0.0588$. Labels alongside the points show the
blocklengths, $N$. The solid line shows the Shannon
limit for this channel.
The bit error probability drops to zero while the rate tends to
0.093, so the concatenated Hamming codes are a `good' code family.
}
\label{fig.concath.rateeb}
}
%%%%%%%%%%%%%%%%%%%% there is a major margin object problem here,
% don't understand it!
The take-home message from this story is
{\em{distance isn't everything}}.\index{distance!isn't everything}
% Indeed, t
The minimum distance of a code, although widely worshipped by coding
theorists, is not of fundamental importance\index{coding theory}
to Shannon's
mission of achieving reliable \ind{communication} over noisy channels.\index{Shannon, Claude}\index{coding theory}
\exercisxB{3}{ex.distancenotE}{
Prove that there exist families of codes with `bad' distance
that are `very good' codes.
}
% soln in _linear.tex
\section{Distance isn't everything}
Let's
% look at this assertion some more in order to
get a
quantitative feeling for the effect of the minimum distance
of a code, for the special case of a \ind{binary symmetric channel}.\index{channel!binary symmetric}
%\exampl{ex.bhat}{
\subsection{The error probability associated with one low-weight codeword}
\label{sec.err.prob.one}
% begin INTRO
Let a binary code have blocklength $N$ and
just two codewords, which differ in $d$ places. For simplicity, let's
assume $d$ is even.
What is the error probability if this code is used on a binary
symmetric channel with noise level $f$?
Bit flips matter only in places where the two codewords differ.
% Only flips of bits in the places that differ matter.
The error probability is dominated by the probability that $d/2$
of these bits are flipped.
What happens
to the other bits is irrelevant, since the optimal decoder ignores them.
\beqan
P(\mbox{block error}) & \simeq & {{d}\choose{d/2}} f^{d/2} (1-f)^{d/2} .
% \geq here if you want
\eeqan
This error probability associated with a single codeword of weight $d$
is plotted in \figref{fig.dist}.%
\amarginfig{c}{%
\footnotesize
\begin{tabular}{c}
\hspace*{0.2in}\psfig{figure=gnu/errorVdist.ps,width=1.8in,angle=-90}\\[0.1in]
\end{tabular}
% see /home/mackay/itp/gnu/dist.gnu
\caption[a]{ The error probability associated with a single codeword of weight $d$,
${{d}\choose{d/2}} f^{d/2} (1-f)^{d/2}$, as
a function of $f$.}
\label{fig.dist}
}
Using the approximation for the binomial coefficient (\ref{eq.stirling.choose}), we
can further approximate
\beqan
P(\mbox{block error})
% \leq here if you want
& \simeq & \left[ 2 f^{1/2} (1-f )^{1/2} \right]^{d} \\
& \equiv & [\beta(f)]^{d} ,
\label{eq.bhatta}
\eeqan
where $\beta(f) = 2 f^{1/2} (1-f )^{1/2}$
is called the \ind{Bhattacharyya parameter} of the channel.\nocite{Bhattacharyya}
%\marginpar{\footnotesize{You don't need
% to memorize this name; indeed, I need to check this is the correct name, as it is not in the
%index of any coding theory books on my shelf! Must check in McEliece.}}
%
% Bhattacharyya, A.On a measure of divergence between two statistical
% populations defined by their probability distributions. Bull.
% Calcutta Math. Soc. 35 (1943), pp. 99-110.
%
% A recent book that calls your $\beta$ the Bhattacharyya parameter is
% Johanesson and Zigangirov's book on convolutional codes. I think some
% of Viterbi's books also use the term.
%
% end INTRO
% \subsection{Recap of `very bad' distance}
Now, consider a general linear code with distance $d$.
Its block error probability
must be at least ${{d}\choose{d/2}} f^{d/2} (1-f)^{d/2}$,
independent of the blocklength $N$ of the code.
For this reason, a sequence of codes of increasing blocklength
$N$ and constant distance $d$ (\ie, `very bad' distance)\label{sec.verybadisbad}
cannot have a block error probability
that tends to zero, on any binary symmetric channel.
If we are interested in making superb error-correcting
codes with tiny, tiny error probability,
we might therefore shun codes with bad distance.
However, being pragmatic, we should look more carefully
at \figref{fig.dist}.
In \chref{ch1} we argued that codes for disk drives
need an error probability smaller than about $10^{-18}$.
If the raw error probability in the \ind{disk drive} is
about $0.001$, the error probability associated
with one codeword at distance $d=20$ is smaller than
$10^{-24}$.
If the raw error probability in the disk drive is
about $0.01$, the error probability associated
with one codeword at distance $d=30$ is smaller than
$10^{-20}$.
For practical purposes, therefore, it is not essential for
a code to have good distance. For example,
codes of blocklength $10\,000$, known to
have many codewords of weight 32, can nevertheless
correct errors of weight 320 with tiny error probability.
I wouldn't want you to think I am {\em recommending\/}
the use of codes with bad distance; in \chref{ch.ldpcc}
we will discuss low-density parity-check codes,
my favourite codes, which have both excellent performance
and {\em good\/} distance.
% These are my favourite codes.
% It's as a matter of honesty that I am pointing out
% that having good distance scarcely matters.
% So regardless of the blocklength used,
\section{The union bound}
The error probability of a code on the binary symmetric
channel can be bounded in terms
of its \ind{weight enumerator} function by adding up
appropriate multiples of
the error probability associated with a single codeword (\ref{eq.bhatta}):
\beq
P(\mbox{block error}) \leq \sum_{w>0} A(w) [\beta(f)]^w .
\label{eq.unionB}
\eeq
% could include Bob's poor man's coding theorem here.
This inequality, which is an example of a {\dem\ind{union bound}},
is accurate for low noise levels $f$,
but inaccurate for high noise levels, because it overcounts
the contribution of errors that cause confusion with more than
one codeword at a time.
%MNBV\newpage
\exercisxB{3}{ex.poormancoding}{
{\sf Poor man's noisy-channel coding theorem}.\index{noisy-channel coding theorem!poor
man's version}\index{poor man's coding theorem}
Pretending
that the union bound
(\ref{eq.unionB}) {\em is\/}
accurate, and using the
average {\ind{weight enumerator} function of a random linear code} (\ref{eq.wef.random}) (\secref{sec.wef.random})
as $A(w)$, estimate the maximum rate $R_{\rm UB}(f)$ at which
one can communicate over a binary symmetric channel.
Or, to look at it more positively, using the union bound
(\ref{eq.unionB}) as an inequality, show that communication
at rates up to $R_{\rm UB}(f)$ is possible over the binary symmetric channel.
% In proving this result, you are proving a `poor man's version' of
% {Shannon}'s noisy-channel coding theorem.
}
In the following chapter, by analysing the probability of error
of {\em \ind{syndrome decoding}\/} for a binary linear code,
and using a union bound, we will prove
Shannon's noisy-channel coding theorem (for
symmetric binary channels), and thus show that {\em very good linear codes exist}.
% possible point for exercise from exact.tex to be included.
\section{Dual codes\nonexaminable}
A concept that has some importance in coding theory,\index{error-correcting code!dual}
though we will have no immediate use for it in this book,
is the idea of the {\dem\ind{dual}} of a linear error-correcting code.
An $(N,K)$
linear error-correcting code can be thought of as a set of $2^{K}$
codewords
generated by adding together all combinations of $K$ independent basis
codewords. The generator matrix of the code consists of
those $K$ basis codewords, conventionally written as row vectors.
For example, the $(7,4)$ Hamming code's generator matrix (from \pref{eq.Generator})
% \eqref{eq.Generator},
is
\beq
\bG = \left[ \begin{array}{ccccccc}
\tt 1& \tt 0& \tt 0& \tt 0& \tt 1& \tt 0& \tt 1 \\
\tt 0& \tt 1& \tt 0& \tt 0& \tt 1& \tt 1& \tt 0 \\
\tt 0& \tt 0& \tt 1& \tt 0& \tt 1& \tt 1& \tt 1 \\
\tt 0& \tt 0& \tt 0& \tt 1& \tt 0& \tt 1& \tt 1 \\
\end{array} \right]
\label{eq.Generator2}
\eeq
and its sixteen codewords were displayed in
\tabref{tab.74h} (\pref{tab.74h}).
The codewords of this code are linear combinations of
the four vectors $\left[
\tt 1 \: \tt 0 \: \tt 0 \: \tt 0 \: \tt 1 \: \tt 0 \: \tt 1 \right]$,
$\left[
\tt 0 \: \tt 1 \: \tt 0 \: \tt 0 \: \tt 1 \: \tt 1 \: \tt 0 \right]$,
$\left[
\tt 0 \: \tt 0 \: \tt 1 \: \tt 0 \: \tt 1 \: \tt 1 \: \tt 1 \right]$,
and
$\left[
\tt 0 \: \tt 0 \: \tt 0 \: \tt 1 \: \tt 0 \: \tt 1 \: \tt 1 \right]$.
An $(N,K)$ code may also be described in terms
of an $M \times N$ parity-check matrix (where $M=N-K$)
as the set of vectors $\{ \bt \}$ that satisfy
\beq
\bH \bt = {\bf 0} .
\eeq
One way of thinking of this equation is that each row
of $\bH$ specifies a vector to which $\bt$ must be orthogonal
if it is a codeword.
\medskip
\noindent
\begin{conclusionbox}
The generator matrix specifies $K$ vectors {\em from
which\/} all codewords can be built, and
the parity-check matrix specifies a set of $M$ vectors
{\em to which\/}
all codewords are orthogonal. \smallskip
The dual of a code is obtained by exchanging the generator
matrix and the parity-check matrix.
\end{conclusionbox}
\medskip
\noindent
{\sf Definition.}
The set of {\em all\/} vectors of length $N$ that are orthogonal to all
codewords in a code, $\C$, is called the dual of the code, $\C^{\perp}$.
\medskip
If $\bt$ is orthogonal to $\bh_1$
and $\bh_2$, then it is also orthogonal to $\bh_3 \equiv \bh_1 + \bh_2$;
so all codewords are orthogonal to
any linear combination of the $M$ rows of $\bH$.
So
the set of all linear combinations of the rows of the parity-check matrix
is the dual code.
% called the dual of the code.
% The dual is itself a linear
% error-correcting code, whose generator matrix is $\bH$.
%% And similarly, t
% The parity-check matrix of the dual is $\bG$,
% the generator matrix of the first code.
For our Hamming $(7,4)$ code, the parity-check matrix is
(from \pref{eq.pcmatrix}):
\beq
\bH = \left[ \begin{array}{cc} \bP & \bI_3 \end{array}
\right] = \left[
\begin{array}{ccccccc}
\tt 1&\tt 1&\tt 1&\tt 0&\tt 1&\tt 0&\tt 0 \\
\tt 0&\tt 1&\tt 1&\tt 1&\tt 0&\tt 1&\tt 0 \\
\tt 1&\tt 0&\tt 1&\tt 1&\tt 0&\tt 0&\tt 1
\end{array} \right] .
\label{eq.pcmatrix2}
\eeq
% and the three vectors to which the codewords are
% orthogonal are
%$\left[
%\tt 1\: \tt 1\: \tt 1\: \tt 0\: \tt 1\: \tt 0\: \tt 0
% \right]$,
%$\left[
%\tt 0\: \tt 1\: \tt 1\: \tt 1\: \tt 0\: \tt 1\: \tt 0
% \right]$,
% and
%$\left[
%\tt 1\: \tt 0\: \tt 1\: \tt 1\: \tt 0\: \tt 0\: \tt 1
% \right]$.
% The codewords are not orthogonal to these $M$
% vectors only, however. I
The dual of the $(7,4)$ Hamming code $\H_{(7,4)}$
is the code shown in
\tabref{tab.74h.dual}.
\begin{table}[htbp]
\figuremargin{%
\begin{center}
\mbox{\small
\begin{tabular}{c} \toprule
% Transmitted sequence
% $\bt$ \\ \midrule
\tt 0000000 \\% yes
\tt 0010111 \\% yes
\bottomrule
\end{tabular} \hspace{0.02in}
\begin{tabular}{c} \toprule
% $\bt$ \\ \midrule
\tt 0101101 \\% yes
\tt 0111010 \\ \bottomrule % yes
\end{tabular} \hspace{0.02in}
\begin{tabular}{c} \toprule
% $\bt$ \\ \midrule
\tt 1001110 \\% yes
\tt 1011001 \\ \bottomrule % yes
\end{tabular} \hspace{0.02in}
\begin{tabular}{c} \toprule
% $\bt$ \\ \midrule
\tt 1100011 \\% yes
\tt 1110100 \\ % yes
\bottomrule
\end{tabular}
}%%%%%%%%% end of row of four tables
\end{center}
}{%
\caption[a]{The eight codewords
% $\{ \bt \}$
of the dual of the $(7,4)$ Hamming code.
[Compare with \protect\tabref{tab.74h},
\protect\pref{tab.74h}.]
}
\label{tab.74h.dual}
}
\end{table}
% STRANGE MISREF????????? CHECK
A possibly unexpected property
of this pair of codes is that the dual, $\H_{(7,4)}^{\perp}$,
is contained within the code $\H_{(7,4)}$ itself:
every word in the dual code is a codeword of the
original $(7,4)$ Hamming code.
This relationship can be written using set notation:
\beq
\H_{(7,4)}^{\perp} \subset \H_{(7,4)}
.
\eeq
The possibility that the set of dual vectors
can overlap the set of codeword vectors is counterintuitive
if we think of the vectors as real vectors -- how
can a vector be orthogonal to itself?
But when we work in modulo-two arithmetic, many non-zero vectors
are indeed orthogonal
% perpendicular
to themselves!
\exercissxB{1}{ex.perp}{
Give a simple rule that distinguishes
whether a binary vector is orthogonal to itself, as is each of the
three vectors
$\left[
\tt 1\: \tt 1\: \tt 1\: \tt 0\: \tt 1\: \tt 0\: \tt 0
\right]$,
$\left[
\tt 0\: \tt 1\: \tt 1\: \tt 1\: \tt 0\: \tt 1\: \tt 0
\right]$,
and
$\left[
\tt 1\: \tt 0\: \tt 1\: \tt 1\: \tt 0\: \tt 0\: \tt 1
\right]$.
}
\subsection{Some more duals}
In general, if a code has a systematic generator matrix,
\beq
\bG = \left[ \bI_K | \bP^{\T} \right] ,
\eeq
where $\bP$ is a $K \times M$ matrix,
then its parity-check matrix is
\beq
\bH = \left[ \bP | \bI_M \right] .
\eeq
\exampl{example.rthreedual}{
The repetition code $\Rthree$ has generator matrix
\beq
\bG =\left[
\begin{array}{ccc}
\tt 1 &\tt 1 &\tt 1
\end{array}
\right];
% [{\tt 1\:1\:1} ] ;
\eeq
its parity-check matrix is
\beq
\bH = \left[
\begin{array}{ccc}
\tt 1 &\tt 1 &\tt 0 \\
\tt 1 &\tt 0 &\tt 1
\end{array}
\right] .
\eeq
The two codewords are [{\tt 1 1 1}] and [{\tt 0 0 0}].
The dual code has generator matrix
\beq
\bG^{\perp} = \bH = \left[
\begin{array}{ccc}
\tt 1 &\tt 1 &\tt 0 \\
\tt 1 &\tt 0 &\tt 1
\end{array}
\right]
\eeq
or equivalently, modifying $\bG^{\perp}$ into systematic form
by row additions,
% manipulations,
\beq
\bG^{\perp} = \left[
\begin{array}{ccc}
\tt 1 &\tt 0 &\tt 1 \\
\tt 0 &\tt 1 &\tt 1
\end{array}
\right] .
\eeq
We call this dual code the {\dem{simple parity code}} P$_3$;\index{error-correcting code!P$_3$}\index{error-correcting code!simple parity}\index{error-correcting code!dual}
it is the code with one parity-check bit, which is equal to
the sum of the two source bits.
The dual code's four codewords are
$ \left[
\tt 1 \: \tt 1 \: \tt 0
\right]
$,
$ \left[
\tt 1 \: \tt 0 \: \tt 1
\right]
$,
$ \left[
\tt 0 \: \tt 0 \: \tt 0
\right]
$,
and
$ \left[
\tt 0 \: \tt 1 \: \tt 1
\right]
$.
In this case, the only vector common to the code and the dual is
the all-zero codeword.
}
\subsection{Goodness of duals}
If a sequence of codes is `good', are their \index{error-correcting code!dual}duals
{good} too?\index{error-correcting code!good}
Examples can be constructed of all cases:
good codes with good duals (random linear codes);
bad codes with bad duals; and good codes with bad duals.
The last category is especially important:
many state-of-the-art codes have the property that
their duals are bad.
The classic example is the low-density parity-check code,
whose dual is a low-density generator-matrix code.\index{error-correcting code!low-density generator-matrix}
\exercisxB{3}{ex.ldgmbad}{
Show that low-density generator-matrix codes
are bad.
A family of low-density generator-matrix codes
is defined by two parameters $j,k$, which are the column
weight and row weight of all rows and columns respectively
of $\bG$. These weights are fixed, independent of $N$;
for example, $(j,k)=(3,6)$.
[Hint: show that the code has low-weight codewords, then
use the argument from \pref{sec.verybadisbad}.]
}
\exercisxD{5}{ex.ldpcgood}{
Show that low-density parity-check codes
are good, and have good distance.\index{error-correcting code!low-density parity-check}
(For solutions, see \citeasnoun{Gallager63} and
\citeasnoun{mncN}.)
}
\subsection{Self-dual codes}
The $(7,4)$ Hamming code had the property that the dual
was contained in the code itself.
% used to say -
% A code is {\dem{\ind{self-orthogonal}}} if it contains its dual.
A code is {\dem{\ind{self-orthogonal}}\/} if it is contained in its dual.
For example,
the dual of the $(7,4)$ Hamming code is a self-orthogonal code.
One way of seeing this is that the overlap between any pair
of rows of $\bH$ is even.
%\marginpar{Is
% it an accepted abuse of terminology to also say
% a code is self-orthogonal if it contains its dual?}
Codes that contain their duals are important in quantum error-correction
\cite{ShorCSS}.
It is intriguing, though not necessarily useful, to
look at codes that are {\dem\ind{self-dual}}.
A code $\C$ is self-dual if
the dual
of the code is identical to the code.
% Here, we are looking for codes that satisfy
\beq
\C^{\perp} = \C .
\eeq
Some properties of self-dual codes can be deduced:
%
\ben
\item
If a code is self-dual, then its generator matrix is also a parity-check
matrix for the code.
\item
Self-dual codes have rate $1/2$, \ie, $M=K=N/2$.
\item
All codewords have even weight.
\een
\exercissxB{2}{ex.selfdual}{
What property must the matrix $\bP$ satisfy, if the code
with generator matrix
$\bG = \left[ \bI_K | \bP^{\T} \right]$
is self-dual?
}
\subsubsection{Examples of self-dual codes}
\ben
\item
The repetition code R$_2$ is a simple example of
a self-dual code.
\beq
\bG = \bH = \left[
\begin{array}{cc}
\tt 1 &\tt 1
\end{array}
\right] .
% [{\tt 1 \: 1 } ]
\eeq
\item
The smallest non-trivial self-dual code is the following
$(8,4)$ code.
\beq
\bG = \left[ \begin{array}{c|c} \bI_4 & \bP^{\T} \end{array}
\right] = \left[
\begin{array}{cccc|cccc}
\tt 1&\tt 0&\tt 0 &\tt 0 &\tt 0&\tt 1&\tt 1&\tt 1\\
\tt 0&\tt 1&\tt 0 &\tt 0 &\tt 1&\tt 0&\tt 1&\tt 1\\
\tt 0&\tt 0&\tt 1 &\tt 0 &\tt 1&\tt 1&\tt 0&\tt 1\\
\tt 0&\tt 0&\tt 0 &\tt 1 &\tt 1&\tt 1&\tt 1&\tt 0
\end{array} \right] .
\label{eq.selfdual84G}
\eeq
\een
\exercissxB{2}{ex.dual84.74}{
Find the relationship of the above $(8,4)$ code to the $(7,4)$ Hamming code.
}
\subsection{Duals and graphs}
Let a code be represented by a graph in which there are
nodes of two types, parity-check constraints and equality
constraints, joined by edges which represent the bits
of the code (not all of which need be transmitted).
The dual code's graph is obtained by replacing all
\ind{parity-check nodes} by equality nodes and {\em vice versa}.
This type of graph is called a \ind{normal graph} by
\citeasnoun{Forney2001}.
% Forney
% added Thu 16/1/03
\subsection*{Further reading}
Duals are important in coding theory because functions
involving a code (such as the posterior distribution over
codewords) can be transformed by a \ind{Fourier transform}
into functions over the dual code.
For an accessible introduction to Fourier analysis on
finite groups, see \citeasnoun{Terras99}.
See also \citeasnoun{macwilliams&sloane}.
\section{Generalizing perfectness to other channels}
Having given up on the search for \ind{perfect code}s
for the binary symmetric channel, we could console
ourselves by changing channel.
We could call a code
`a perfect $u$-error-correcting code for the binary \ind{erasure channel}'\index{channel!erasure}
if it can restore any $u$ erased bits, and never more than $u$.%
\marginpar{\small\raggedright{In a perfect $u$-error-correcting code for the
binary {erasure channel}, the number of redundant bits must be $N-K=u$.
}}
Rather than using the word perfect, however,
the conventional term for such a code is a `\ind{maximum distance separable} code', or MDS code.
\label{sec.RAIDII}
% Examples:
As we already noted in \exerciseref{ex.raid3},
the $(7,4)$ \ind{Hamming code} is {\em not\/}
an MDS
% maximum distance separable
code.
It can recover {\em some\/} sets of 3 erased bits,
but not all. If any 3 bits corresponding to a codeword of weight 3
are erased, then one bit of information is unrecoverable.
This is why the $(7,4)$ code is a poor choice for a \ind{RAID} system.
%A maximum distance separable (MDS) block code is a linear code whose distance is maximal among all linear
% block codes of rate k/n. It is well known that MDS block codes do exist if the field size is more than n.
A tiny example of a
maximum distance separable code\index{erasure-correction}\index{error-correcting code!maximum distance separable}\index{error-correcting code!parity-check code}\index{MDS}
is the simple parity-check code $P_{3}$
whose parity-check matrix is
$\bH = [{\tt 1\, 1\, 1}]$.
This code has 4 codewords, all of which have even parity. All codewords
are separated by a distance of 2. Any single erased bit can be restored
by setting it to the parity of the other two bits.
The repetition codes are also maximum distance separable codes.
\exercissxB{5}{ex.qeccodeperfect}{
Can you make an $(N,K)$ code, with $M=N-K$ parity symbols,
for a $q$-ary erasure channel, such that the decoder can recover
the codeword when {\em{any}\/} $M$ symbols
are erased in a block
of $N$?
[Example: for
% There do exist some such codes: for example, for
the channel with
$q=4$ symbols there is
an $(N,K) = (5,2)$ code which can correct any $M=3$ erasures.]
% ; and for $q=8$ there is a $(9,2)$ code.]
}
For the $q$-ary erasure channel with $q>2$, there are large numbers
of MDS codes, of which the Reed--Solomon codes are the most
famous and most widely used.
As long as the field size $q$ is bigger than the blocklength $N$,
MDS block codes of any rate can be found. (For further reading, see \citeasnoun{lincostello83}.)
% according to my notes.
% 4-ary erasure channel.
% Include tournament example. GF4, 16 individuals. can tolerate 3 erasures.
% Reed--Solomon codes.
\section{Summary}
Shannon's codes for the binary symmetric channel
can almost always correct $\fN$ errors, but they
are not $\fN$-error-correcting codes.
%\noindent
\subsection*{Reasons why the distance of a code has little relevance}
\ben
\item
The Shannon limit shows that the best codes must be able to
cope with a noise level twice as big as the maximum
noise level for a bounded-distance decoder.
\item
When the binary symmetric channel has
$f>1/4$, no code with a bounded-distance decoder
can communicate at all; but Shannon says good codes exist
for such channels.
\item
Concatenation shows that we can get good performance even if
the distance is bad.\index{concatenation}\index{distance!of code}
\een
%
% Furthermore, `distance isn't everything' -- you can actually
% get to the Shannon limit with a code whose distance is `bad'.
%
% Exercise - prove that if a sequence of codes is very bad then it can't
% have arbitrarily small error probability.
The whole weight enumerator function is relevant to the question
of whether a code is a good code.
The relationship between good codes and
distance properties is discussed further in \exerciseref{ex.prob.error.match}.
% ex.equal.threshold}.
%\section*{Further reading}
% For a paper with codes having the property
% distance, but for practical purposes a code with blocklength $N=10\,000$
% can have codewords of weight $d=32$ and the error probability
% can remain negligibly small even when the channel
% is creating errors of weight 320.
% {mackaymitchisonmcfadden2003}
\section{Further exercises}
% also known as {ex.equal.threshold}
\exercissxC{3}{ex.prob.error.match}{
A codeword $\bt$ is selected from a linear $(N,K)$
code $\C$, and it is transmitted
over a noisy channel; the received signal is
$\by$.
We assume that the channel is a memoryless
channel such as a Gaussian channel.
Given an assumed channel model $P(\by \given \bt)$, there are
two decoding problems.
\begin{description}
\item[The codeword decoding problem] is the task of\index{decoder!codeword}
inferring which codeword $\bt$ was transmitted given the
received signal.
\item[The bitwise decoding problem] is the task of inferring\index{decoder!bitwise}
for each transmitted bit $t_n$ how likely it is that that
bit was a one rather than a zero.
\end{description}
Consider optimal decoders for these two decoding problems.
%
% these will be presented again in
% section \ref{sec.decoding.problems}
% exact.tex
%
Prove that the probability of error of the optimal
bitwise-decoder is closely related to the probability of error of
the optimal codeword-decoder, by proving the following
theorem.\index{decoder!probability of error}
\begin{ctheorem}
If a binary linear code\index{distance!of code, and error probability}
has minimum distance $d_{\min}$,
then,
for any given channel, the codeword bit error probability of the optimal
bitwise decoder, $p_{\rm b}$,
and the block error probability of the maximum likelihood decoder, $p_{\rm B}$,
are related by:
\beq
p_{\rm B} \geq p_{\rm b} \geq \frac{1}{2} \frac{d_{\min}}{N} p_{\rm B} .
\label{eq.thmpBpb}
\eeq
% [I am sure this theorem is well-known; I am not claiming it is original.]
\end{ctheorem}
}
\exercisaxA{1}{ex.HammingD}{
What are the minimum distances of the $(15,11)$ Hamming
code and the $(31,26)$ Hamming
code?
}
\exercisaxB{2}{ex.estimate.wef}{
Let $A(w)$ be the
average weight enumerator function of a rate-$1/3$
random linear code with $N=540$ and $M=360$.
Estimate, from first principles, the value of $A(w)$ at $w=1$.
}
\exercisaxC{3C}{ex.handshakecode}{
{\sf A code with minimum distance\index{Gilbert--Varshamov distance}\index{distance!Gilbert--Varshamov}
greater than $d_{\rm GV}$.}
% Another way to make a code is to define a generator matrix
% or parity-check matrix.
A rather nice $(15,5)$ code
is generated by this generator matrix, which is based on measuring the parities
of all the ${{5}\choose{3}} = 10$ triplets of source bits:
\beq
\bG = \left[
\begin{array}{*{15}{c}}
1&\tinyo&\tinyo&\tinyo&\tinyo&\tinyo&1&1&1&\tinyo&\tinyo&1&1&\tinyo&1 \\
\tinyo&1&\tinyo&\tinyo&\tinyo&\tinyo&\tinyo&1&1&1&1&\tinyo&1&1&\tinyo \\
\tinyo&\tinyo&1&\tinyo&\tinyo&1&\tinyo&\tinyo&1&1&\tinyo&1&\tinyo&1&1\\
\tinyo&\tinyo&\tinyo&1&\tinyo&1&1&\tinyo&\tinyo&1&1&\tinyo&1&\tinyo&1\\
\tinyo&\tinyo&\tinyo&\tinyo&1&1&1&1&\tinyo&\tinyo&1&1&\tinyo&1&\tinyo
\end{array} \right] .
\eeq
Find the minimum distance and weight enumerator function
of this code.
}
\exercisaxC{3C}{ex.findAwmonodec}{
% {\sf A code with minimum distance\index{Gilbert--Varshamov distance}\index{distance!Gilbert--Varshamov}
% slightly greater than $d_{\rm GV}$.}
Find the minimum distance of the `{pentagonful}\index{pentagonful code}'\index{error-correcting code!pentagonful}%
\amarginfig{t}{
\begin{center}
\buckypsfigw{pentagon.eps}
\end{center}
\caption[a]{The graph of the pentagonful
low-density parity-check code with
15 bit nodes (circles) and 10 parity-check nodes (triangles).
}
}
low-density parity-check code whose
parity-check matrix is
\beq
\bH = \left[ \begin{array}{*{5}{c}|*{5}{c}|*{5}{c}}
1 & \tinyo & \tinyo & \tinyo & 1 & 1 & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo \\
1 & 1 & \tinyo & \tinyo & \tinyo & \tinyo & 1 & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo \\
\tinyo & 1 & 1 & \tinyo & \tinyo & \tinyo & \tinyo & 1 & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo \\
\tinyo & \tinyo & 1 & 1 & \tinyo & \tinyo & \tinyo & \tinyo & 1 & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo \\
\tinyo & \tinyo & \tinyo & 1 & 1 & \tinyo & \tinyo & \tinyo & \tinyo & 1 & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo \\ \hline
\tinyo & \tinyo & \tinyo & \tinyo & \tinyo & 1 & \tinyo & \tinyo & \tinyo & \tinyo & 1 & \tinyo & \tinyo & \tinyo & 1 \\
\tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & 1 & \tinyo & 1 & 1 & \tinyo & \tinyo & \tinyo \\
\tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & 1 & \tinyo & \tinyo & \tinyo & \tinyo & 1 & 1 & \tinyo & \tinyo \\
\tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & 1 & \tinyo & \tinyo & 1 & 1 & \tinyo \\
\tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & 1 & \tinyo & \tinyo & \tinyo & \tinyo & \tinyo & 1 & 1
\end{array} \right] .
\label{eq.monodec}
\eeq
Show that nine of the ten rows are independent, so the
code has parameters $N=15$, $K=6$.
Using a computer, find its weight enumerator function.
% Find its weight enumerator function.
}
\exercisxB{3C}{ex.concateex}{
Replicate the calculations used to produce
\figref{fig.concath.rate}.
Check the assertion that the highest noise level
that's correctable is 0.0588.
Explore alternative concatenated
sequences of codes. Can you find a better sequence of concatenated
codes -- better in the sense that it
has either higher asymptotic rate $R$ or can tolerate
a higher noise level $\q$?
}
\exercissxA{3}{ex.syndromecount}{
Investigate the possibility of achieving the Shannon
limit with linear block codes, using the following \ind{counting argument}.
Assume a linear code of large blocklength $N$ and rate $R=K/N$.
The code's parity-check matrix $\bH$ has $M = N - K$ rows.
Assume that the code's optimal decoder, which solves the
syndrome decoding problem $\bH \bn = \bz$, allows reliable communication
over a binary symmetric channel with flip probability $f$.
How many `typical' noise vectors $\bn$ are there?
Roughly how many distinct syndromes $\bz$ are there?
Since $\bn$ is reliably deduced from $\bz$ by the optimal decoder,
the number of syndromes must be greater than or equal to the number of
typical noise vectors. What does this tell you about the largest
possible value of rate $R$ for a given $f$?
}
\exercisxB{2}{ex.zchanneldeficit}{
Linear binary codes use the input symbols {\tt{0}} and {\tt{1}} with
equal probability, implicitly treating the channel as a symmetric
channel. Investigate how much loss in communication rate is caused by
this assumption, if in fact the channel is a highly asymmetric channel.
Take as an example a Z-channel. How much smaller is the maximum possible rate
of communication using symmetric inputs than the capacity of the channel?
[Answer: about 6\%.]
}
\exercisxC{2}{ex.baddistbad}{
Show that codes with `very bad' distance are `bad' codes, as defined
in \secref{sec.bad.code.def} (\pref{sec.bad.code.def}).
%
% Show that there exist codes with `bad' distance
% that are `very good' codes.
%
% this bit already done in {ex.distancenotE}{
}
\exercisxC{3}{ex.puncture}{
One linear code can be obtained from another
by {\dem{\ind{puncturing}}}. Puncturing
means taking each codeword and deleting a defined set of bits.
Puncturing turns an $(N,K)$ code into
an $(N',K)$ code, where $N'2$, some MDS codes can be found.
As a simple example, here is a $(9,2)$ code for the
$8$-ary erasure channel.
The code is defined in terms of the\index{Galois field}
% \index{finite field}
multiplication and addition rules of $GF(8)$,
which are given in \appendixref{sec.gf8}.
The elements of the input alphabet are $\{0,1,A,B,C,D,E,F\}$
and
the
generator matrix of the code is
\beq
\bG = \left[ \begin{array}{*{9}{c}}
1 &0 &1 &A &B &C &D &E &F \\
0 &1 &1 &1 &1 &1 &1 &1 &1 \\
\end{array} \right] .
\eeq
The resulting 64 codewords are:\smallskip
{\footnotesize\tt
\begin{narrow}{0in}{-\margindistancefudge}%
\begin{realcenter}
\begin{tabular}{*{8}{c}}
000000000 &
011111111 &
0AAAAAAAA &
0BBBBBBBB &
0CCCCCCCC &
0DDDDDDDD &
0EEEEEEEE &
0FFFFFFFF
\\
101ABCDEF &
110BADCFE &
1AB01EFCD &
1BA10FEDC &
1CDEF01AB &
1DCFE10BA &
1EFCDAB01 &
1FEDCBA10
\\
A0ACEB1FD &
A1BDFA0EC &
AA0EC1BDF &
AB1FD0ACE &
ACE0AFDB1 &
ADF1BECA0 &
AECA0DF1B &
AFDB1CE0A
\\
B0BEDFC1A &
B1AFCED0B &
BA1CFDEB0 &
BB0DECFA1 &
BCFA1B0DE &
BDEB0A1CF &
BED0B1AFC &
BFC1A0BED
\\
C0CBFEAD1 &
C1DAEFBC0 &
CAE1DC0FB &
CBF0CD1EA &
CC0FBAE1D &
CD1EABF0C &
CEAD10CBF &
CFBC01DAE
\\
D0D1CAFBE &
D1C0DBEAF &
DAFBE0D1C &
DBEAF1C0D &
DC1D0EBFA &
DD0C1FAEB &
DEBFAC1D0 &
DFAEBD0C1
\\
E0EF1DBAC &
E1FE0CABD &
EACDBF10E &
EBDCAE01F &
ECABD1FE0 &
EDBAC0EF1 &
EE01FBDCA &
EF10EACDB
\\
F0FDA1ECB &
F1ECB0FDA &
FADF0BCE1 &
FBCE1ADF0 &
FCB1EDA0F &
FDA0FCB1E &
FE1BCF0AD &
FF0ADE1BC
\\
\end{tabular}
\end{realcenter}
\end{narrow}
}
}
% from exercise section in _linear.tex
%
% this was in _sexact
%
% ex.prob.error.match
\soln{ex.prob.error.match}{% ex.equal.threshold}{
{\sf Quick, rough proof of the theorem.} Let $\bx$ denote the difference
between the reconstructed codeword and the transmitted codeword.
For any given channel output $\br$, there is a posterior distribution over
$\bx$. This posterior distribution is positive only
on vectors $\bx$ belonging to the code; the sums
that follow are over codewords $\bx$. The block error probability is:
\beq
p_{\rm B} = \sum_{\bx \neq 0} P(\bx \given \br) .
\label{eq.pBdef}
\eeq
The average bit error probability, averaging over all bits in
the codeword, is:
\beq
p_{\rm b} = \sum_{\bx \neq 0} P(\bx \given \br) \frac{w(\bx)}{N} ,
\label{eq.pbdef}
\eeq
where $w(\bx)$ is the weight of codeword $\bx$.
Now the weights of the non-zero codewords satisfy
\beq
1 \geq \frac{w(\bx)}{N} \geq \frac{d_{\min}}{N} .
\label{eq.ineq}
\eeq
Substituting the inequalities (\ref{eq.ineq}) into
the definitions (\ref{eq.pBdef},$\,$\ref{eq.pbdef}),
we obtain:
%
\beq
p_{\rm B} \geq p_{\rm b} \geq
% \frac{1}{2}
\frac{d_{\min}}{N} p_{\rm B} ,
\label{eq.thmpBpbA}
\eeq
which is a factor of two stronger, on the right, than
the stated result (\ref{eq.thmpBpb}).
In making the proof watertight, I have weakened the result a little.\medskip
% So the bit and block {\em thresholds\/} of a code with good distance
% are identical.
%\section
\noindent
{\sf Careful proof.}
The theorem relates the performance of the optimal
block decoding algorithm and the optimal bitwise decoding algorithm.
We introduce another pair of decoding algorithms, called the block-guessing
decoder and the bit-guessing decoder. The idea is that
these two algorithms are similar to
the optimal block decoder and the optimal bitwise decoder,
but lend themselves more easily to
analysis.
We now define these decoders. Let $\bx$ denote
the inferred codeword. For any given code:
\begin{description}
\item[The optimal block decoder] returns the codeword $\bx$ that maximizes
the posterior probability
$P(\bx \given \br)$, which is proportional to the likelihood
$P( \br \given \bx)$.
The probability of error of this decoder is called
$\PB$.
\item[The optimal bit decoder] returns for each of the $N$ bits, $x_n$,
the value of $a$ that maximizes
the posterior probability
$P( x_n \eq a \given \br ) = \sum_{\bx} P(\bx \given \br) \,\truth\! [ x_n\eq a ]$.
The probability of error of this decoder is called
$\Pb$.
\item[The block-guessing decoder] returns a random codeword $\bx$
with probability distribution given by the posterior probability
$P(\bx \given \br)$.
The probability of error of this decoder is called
$\PGB$.
\item[The bit-guessing decoder] returns for each of the $N$ bits, $x_n$,
a random bit from the probability distribution $P( x_n \eq a \given \br )$.
The probability of error of this decoder is called
$\PGb$.
\end{description}
The theorem states that
the optimal bit error probability $\Pb$
is bounded above by
$\PB$ and below by a given multiple of $\PB$ (\ref{eq.thmpBpb}).
%
%\beq
% P_B \geq P_b \geq \frac{1}{2} \frac{d_{\min}}{N} P_B .
%\label{eq.thmpBpb.again}
%\eeq
The left-hand inequality in (\ref{eq.thmpBpb})
is trivially true -- if a block is correct,
all its constituent bits are correct; so if the optimal
block decoder outperformed the optimal bit decoder, we could
make a better bit decoder from the block decoder.
We prove the right-hand inequality by establishing that:
% the following two lemmas:
\ben
\item
the bit-guessing decoder is nearly
as good as the optimal bit decoder:
\beq
\PGb \leq 2 \Pb .
\label{eq.guess}
\eeq
\item
the bit-guessing decoder's error probability
is related to the block-guessing decoder's
by
\beq
\PGb \geq \frac{d_{\min}}{N} \PGB .
\eeq
\een
Then since $\PGB \geq \PB$, we have
\beq
\Pb > \frac{1}{2} \PGb \geq \frac{1}{2} \frac{d_{\min}}{N} \PGB
\geq \frac{1}{2} \frac{d_{\min}}{N} \PB .
\eeq
We now prove the two lemmas.\medskip
\noindent
%\subsection
{\sf Near-optimality of guessing:}
Consider first the case of
a single bit, with posterior probability $\{ p_0, p_1 \}$.
% Without loss of generality, let $p_0 \geq p_1$.
The optimal bit decoder
% picks $\argmax_a p_a$,
% \ie, 0,
% and
has probability of error
\beq
% \Pb
P^{\rm{optimal}} = \min (p_0,p_1).
\eeq
% $p_1$.
The guessing decoder picks from 0 and 1. The truth is also
distributed with the same probability. The probability
that the guesser and the truth match is
$p_0^2 + p_1^2$; the probability that they
mismatch is the guessing error probability,
\beq
% \PGb
P^{\rm guess} = 2 p_0 p_1 \leq 2 \min (p_0,p_1) = 2 P^{\rm{optimal}} .
\eeq
Since $\PGb$ is the average
of many such error probabilities, $P^{\rm guess}$,
and $\Pb$ is the average of the corresponding optimal
error probabilities, $P^{\rm{optimal}}$,
we obtain the desired relationship (\ref{eq.guess})
between $\PGb$ and $\Pb$.\ENDproof
%
\medskip
%\subsection
\noindent
{\sf Relationship between bit error probability
and block error probability:}
The bit-guessing and block-guessing decoders
can be combined in a single system:
% The posterior probability of a bit $x_n$ and a block $\bx$
% is given by
%\beq
% P( x_n = a , \bx \given \br ) =
% P( \bx \given \br ) P( x_n = a \given \bx, \br ) =
%\eeq
% So w
we can draw a sample $x_n$ from the marginal distribution
$P(x_n \given \br)$ by
drawing a sample $( x_n , \bx )$
from the joint distribution $P( x_n , \bx \given \br )$,
then discarding the value of $\bx$.
We can distinguish between two cases: the discarded value of $\bx$
is the correct codeword, or not.
The probability of bit error for the bit-guessing decoder
can then be written as a sum of two terms:
\beqa
\PGb &\eq &
P(\mbox{$\bx$ correct}) P(\mbox{bit error} \given \mbox{$\bx$ correct})
\nonumber
\\
& & + \,
P(\mbox{$\bx$ incorrect}) P(\mbox{bit error} \given \mbox{$\bx$ incorrect})
\\
&=&
% P(\mbox{$\bx$ correct}) \times
0 + \PGB P(\mbox{bit error} \given \mbox{$\bx$ incorrect}) .
\eeqa
% The first of these terms is zero.
Now, whenever the guessed $\bx$ is incorrect, the true
$\bx$ must differ from it in at least $d$ bits, so
the probability of bit error in these cases is at least $d/N$.
So
\[%beq
\PGb \geq \frac{d}{N} \PGB .
% \eepf
\]%eeq
QED.\hfill $\epfsymbol$
}
\soln{ex.syndromecount}{
The number of `typical' noise vectors $\bn$ is
roughly $2^{NH_2(f)}$.
% , where $H=H_2(f)$.
The number of distinct syndromes $\bz$ is $2^M$.
So reliable communication implies
\beq
M \geq NH_2(f) ,
\eeq
or, in terms of the rate $R = 1-M/N$,
\beq
R \leq 1 - H_2(f) ,
\eeq
a bound which agrees precisely with the capacity of the channel.
This argument is turned into a proof in the following chapter.
}
% BORDERLINE
\soln{ex.hat.puzzle}{
% Mathematicians credit the problem to Dr. Todd Ebert, a computer
% science instructor at the University of California at Irvine, who
% introduced it in his Ph.D. thesis at the University of California at
% Santa Barbara in 1998.
In the three-player case,
it is possible for the group to win three-quarters of the time.
Three-quarters of the time, two of the players will have hats of the
same colour and the third player's hat will be the opposite colour. The
group can win every time this happens by using the following strategy.
Each player looks at the other two players'
hats. If the two hats are {\em different\/}
colours, he passes. If they are the
{\em same\/} colour, the player guesses his own hat is the {\em opposite\/}
colour.
This way, every time the hat colours are distributed two and one, one
player will guess correctly and the others will pass, and the group
will win the game. When all the hats are the same colour, however, {\em all
three\/} players will guess incorrectly and the group will lose.
When any particular player guesses a colour, it is true
that there is only a 50:50 chance that their guess is right.
The reason that the group wins 75\% of the time is that their
strategy ensures that when players are guessing wrong, a great many are
guessing wrong.
For larger numbers of players, the aim is
to ensure that most of the time no one
is wrong and occasionally everyone is wrong at once.
In the game with 7 players, there is a strategy for
which the group wins 7 out of every 8 times they play.
In the game with 15 players, the group can win 15 out of 16 times.
If you have not figured out these winning strategies for teams
of 7 and 15, I recommend thinking about the
solution to the three-player game in terms of the locations
of the winning and losing states on the three-dimensional hypercube,
then thinking laterally.
\begincuttable
If the number of players, $N$, is $2^r-1$,
the optimal strategy can be defined using a Hamming code of length $N$,
and the probability of winning the prize is $\linefrac{N}{(N+1)}$.
Each player
is identified with a number $n \in 1\ldots N$.
The two colours
are mapped onto {\tt{0}} and {\tt{1}}. Any state of their hats
can be viewed as a received vector out of a binary channel.
A random binary vector of length $N$
is either a codeword of the Hamming code, with probability
$1/(N+1)$, or it differs
in exactly one bit from a codeword.
% There is a probability
Each player looks at all the other bits and considers whether his bit
can be set to a colour
such that the state is a codeword (which can be deduced
using the decoder
of the Hamming code). If it can, then
the player guesses that his hat is the {\em other\/} colour.
If the state is actually a codeword, all players will guess and
will guess wrong. If the state is a non-codeword, only
one player will guess, and his guess will be correct.
It's quite easy to train seven players to follow the optimal
strategy if the cyclic representation of the $(7,4)$ Hamming code
is used (\pref{sec.h74cyclic}).
% I am not sure of the optimal solution for the `Scottish version'
% of the rules in which the prize is only awarded to the group
% if they {\em all\/} guess correctly.
% As a starting point, if one flips the guesses of the winning strategy
% for the original game, the group
% will win whenever it is in a codeword state, which
% happens with probability $1/(N+1)$. The question is
% what to do with the `passes'.
%% since passing is never in one's interests.
% Can the group do better than replacing passes with random guessing?
}
% \soln{ex.selforthog}{
% removed to cutsolutions.tex
% end from _linear.tex
\dvips
%\section{Solutions to Chapter \protect\ref{ch.linearecc}'s exercises} %
%\section{Solutions to Chapter \protect\ref{ch.linearecc}'s exercises} %
\dvipsb{solutions linear}
\dvips
\prechapter{About Chapter}
In this chapter we will draw together several ideas
that we've encountered so far in one nice short proof.
We will simultaneously prove both
Shannon's noisy-channel coding theorem (for
symmetric binary channels)
and his source coding theorem (for binary sources).
While this proof has connections to many preceding chapters
in the book, it's not essential to have read them all.
On the noisy-channel coding side,
our proof will be more constructive than the
proof given in \chref{ch.six}; there, we proved that
almost any random code is `very good'.
Here we will show that
almost any {\em linear\/} code is very good.
We will make use of the idea of typical sets (Chapters \ref{ch.two} and \ref{ch.six}),
and we'll borrow from the previous chapter's
calculation of the weight enumerator function of random linear codes (\secref{sec.wef.random}).
On the source coding side,
our proof will show that {\em random linear \ind{hash function}s} can be used
for compression of compressible binary sources, thus giving
a link to \chref{ch.hash}.
\ENDprechapter
\chapter{Very Good Linear Codes Exist}
\label{ch.lineartypical}
%
% very good linear codes exist
%
In this chapter we'll use a single calculation
to prove simultaneously
the \ind{source coding theorem} and the\index{noisy-channel coding theorem}
noisy-channel coding theorem for the \ind{binary symmetric channel}.\index{channel!binary symmetric}\index{noisy-channel coding theorem!linear codes}\index{linear block code!noisy-channel coding theorem}\index{error-correcting code!linear!noisy-channel coding theorem}
{Incidentally,
this proof works for much more general channel models,
not only the binary symmetric channel. For example,
the proof can be reworked for channels with
non-binary outputs, for time-varying channels
and for channels with memory, as long as they
have binary inputs satisfying a symmetry property,
\cf\ \secref{sec.Symmetricchannels}.}
%
\label{ch.linear.good}
\section{A simultaneous proof of the source coding and
noisy-channel coding theorems}
We consider a linear error-correcting code with binary \ind{parity-check
matrix} $\bH$. The matrix has $M$ rows and $N$ columns.
Later in the proof we will increase $N$ and $M$, keeping $M \propto N$.
The
rate of the code satisfies
\beq
R \geq 1 - \frac{M}{N}.
\eeq
If all the rows of $\bH$ are independent then this
is an equality, $R = 1 -M/N$. In what follows,\index{error-correcting code!rate}\index{error-correcting code!linear}
we'll assume the equality holds. Eager readers
may work out the expected rank of
a random binary matrix $\bH$ (it's very close to $M$)
and pursue the effect that the difference ($M - \mbox{rank}$) has
% small number of linear dependences have
on the rest of this proof (it's negligible).
A codeword $\bt$ is selected, satisfying
\beq
\bH \bt = {\bf 0} \mod 2 ,
\eeq
and a binary symmetric channel adds noise $\bx$, giving
the received signal\marginpar{\small\raggedright{In this chapter
$\bx$ denotes the noise added by the channel,
not the input to the channel.}}
\beq
\br = \bt + \bx \mod 2.
\eeq
The receiver aims to infer both $\bt$ and $\bx$ from
$\br$ using a \index{syndrome decoding}{syndrome-decoding} approach.
Syndrome decoding was first introduced in
\secref{sec.syndromedecoding} (\pref{sec.syndromedecoding} and \pageref{sec.syndromedecoding2}).
% and \secref{sec.syndromedecoding2}.
The receiver computes the syndrome
\beq
\bz = \bH \br \mod 2 = \bH \bt + \bH \bx \mod 2
= \bH \bx \mod 2 .
\eeq
% Since $\bH \bt = {\bf 0}$, t
The syndrome only depends on the noise $\bx$,
and the decoding problem is to find the most probable $\bx$ that
satisfies
\beq
\bH \bx = \bz \mod 2.
\eeq
This best estimate for the noise vector, $\hat{\bx}$, is then
subtracted from $\br$ to give the best guess for $\bt$.
Our aim is to show that,
as long as $R < 1-H(X) = 1-H_2(f)$,
where $f$ is the flip probability of the binary symmetric channel,
the optimal decoder for this syndrome-decoding
problem has vanishing probability of error, as $N$ increases,
for random $\bH$.
% and averaging over all binary matrices $\bH$.
We prove this result by studying a sub-optimal
strategy for solving the decoding problem. Neither the optimal decoder
nor this {\em \ind{typical-set decoder}\/} would be easy to implement,
but the typical-set decoder is easier to \analyze.
The typical-set decoder examines the typical
set $T$ of noise vectors, the set of
noise vectors $\bx'$ that satisfy $\log \dfrac{1}{P(\bx')} \simeq
NH(X)$,\marginpar{\small\raggedright{We'll leave out the $\epsilon$s and $\beta$s that make
a typical-set definition rigorous. Enthusiasts are encouraged
to revisit \secref{sec.ts} and put these details into this proof.}}
checking to see if any of those typical vectors
$\bx'$ satisfies the observed syndrome,
\beq
\bH \bx' = \bz .
\eeq
If exactly one typical vector $\bx'$ does so, the typical
set decoder reports that vector as the hypothesized
noise vector.
If no typical vector matches the observed syndrome,
or more than one does, then the typical
set decoder reports an error.
The probability of error of the typical-set decoder, for
a given matrix $\bH$, can be written as a sum of two terms,
\beq
P_{{\rm TS}|\bH} = P^{(I)} + P^{(II)}_{{\rm TS}|\bH} ,
\eeq
where $P^{(I)}$ is the probability that the true noise
vector $\bx$ is itself not typical,
and $P^{(II)}_{{\rm TS}|\bH}$ is the probability
that the true $\bx$ is typical and at least one other typical vector
clashes with it.
The first probability vanishes as $N$ increases,
as we proved when we first studied typical sets (\chref{ch.two}).
We concentrate on the second probability.
% , the probability of a type-II error.
To recap, we're imagining a true noise vector, $\bx$;
and if {\em any\/} of the typical noise vectors
$\bx'$, different from $\bx$, satisfies $\bH (\bx' - \bx) = 0$,
then we have an error.
We use the truth function
\beq
\truth \! \left[ \bH (\bx' - \bx) = 0 \right],
\eeq
whose value is one if the statement $\bH (\bx' - \bx) = 0$ is true
and zero otherwise.
We can bound the number of type II errors made when the noise is
$\bx$ thus:
\newcommand{\xprimecondition}{\raisebox{-4pt}{\footnotesize\ensuremath{\bx'}:}
\raisebox{-3pt}[0.025in][0.0in]{% prevent it from hanging down and pushing other stuff down
\makebox[0.2in][l]{\tiny$\!\begin{array}{l} {\tiny\bx' \!\in T}\\
{\tiny\bx' \! \neq \bx} \end{array}$}}}
\beq
\left[\mbox{Number of errors given $\bx$ and $\bH$}\right] \leq \sum_{\xprimecondition}
\truth\! \left[ \bH (\bx' - \bx) = 0 \right] .
\label{eq.lt.union}
\eeq
The number of errors is either zero or one; the sum on the
right-hand side may exceed one,\marginpar{\small\raggedright{\Eqref{eq.lt.union}
is a \ind{union bound}.}}
in cases where several typical noise
vectors have the same syndrome.
We can now write down the probability of a type-II error
by averaging over $\bx$:
\beq
P^{(II)}_{{\rm TS}|\bH} \leq \sum_{\bx \in T} P(\bx)
\sum_{\xprimecondition} \truth\! \left[ \bH (\bx' - \bx) = 0 \right] .
\eeq
Now, we will find the average of this probability of type-II error
over all linear codes by averaging over $\bH$.
By showing that the {\em average\/} probability of type-II error
vanishes, we will thus show that there exist linear
codes with vanishing error probability, indeed, that
almost all linear codes are very good.
We denote averaging over all binary matrices $\bH$ by $\left< \ldots \right>_{\bH}$.
The average probability of type-II error is
\beqan
\bar{P}^{(II)}_{{\rm TS}}
& =&
\sum_{\bH} P(\bH)
P^{(II)}_{{\rm TS}|\bH} \: = \:
\left< P^{(II)}_{{\rm TS}|\bH} \right>_{\bH}
\\
&=&
\left<
\sum_{\bx \in T} P(\bx)
\sum_{\xprimecondition} \truth\! \left[ \bH (\bx' - \bx) = 0 \right]
\right>_{\!\bH}
\\
&=&
\sum_{\bx \in T} P(\bx)
\sum_{\xprimecondition}
\left<
\truth\! \left[ \bH (\bx' - \bx) = 0 \right]
\right>_{\bH}
.
\eeqan
Now, the quantity
$\left<
\truth\! \left[ \bH (\bx' - \bx) = 0 \right]
\right>_{\bH}$ already cropped up
when we
were calculating the
expected weight enumerator function of random linear codes (\secref{sec.wef.random}):
for any non-zero binary vector $\bv$, the probability that $\bH \bv =0$,
averaging over all matrices $\bH$, is $2^{-M}$.
So
\beqan
\bar{P}^{(II)}_{{\rm TS}}
& = &
\left( \sum_{\bx \in T} P(\bx) \right)
\left( |T| - 1 \right)
2^{-M}\\
& \leq &
|T| \: 2^{-M}
,
\eeqan
where $|T|$ denotes the size of the typical set.
As you will recall from \chref{ch.two}, there are roughly
$2^{NH(X)}$ noise vectors in the typical set.
So
\beqan
\bar{P}^{(II)}_{{\rm TS}}
& \leq &
2^{NH(X)} 2^{-M}
.
\eeqan
This bound on the probability of error either vanishes
or grows exponentially as $N$
increases (remembering that
% , as we are fixing the code rate
% $R = 1-M/N$,
we are keeping $M$ proportional to $N$ as $N$ increases).
It vanishes if
\beq
H(X) < M/N .
\eeq
% this clause is cuttable
% CUT ME?
% and grows if
%\beq
% NH(X) > M .
%\eeq
% end CUT ME
Substituting $R=1-M/N$,
we have thus established the
% positive half of Shannon's
noisy-channel coding theorem for the binary symmetric channel:
very good linear codes exist
%as long as
%$H(X) < M/N$, \ie, as long as
for any rate $R$ satisfying
\beq
R < 1-H(X) ,
\eeq
where $H(X)$ is the entropy of the channel
noise, per bit.\ENDproof
\exercisxC{3}{ex.generalchannel}{
Redo the proof for a more general channel.
}
\section{Data compression by linear hash codes}
The decoding game we have just played can also\index{random code!for compression}
be viewed as an {\dem\ind{uncompression}\/} game.\index{hash code}
The world produces a binary noise vector $\bx$
from a source $P(\bx)$. The noise has redundancy (if the flip probability is not 0.5). We
compress it with a linear compressor
that maps the $N$-bit input $\bx$ (the noise) to
the $M$-bit output $\bz$ (the syndrome).\index{hash function!linear}\index{hash code}
Our uncompression task is to recover the
input $\bx$ from the output $\bz$.
The rate of the compressor is
\beq
R_{\rm compressor} \equiv M/N .
\eeq
[We don't care about the possibility of linear redundancies
in our definition of the rate, here.]
The result that we just found, that
the decoding problem can be solved, for
almost any $\bH$, with vanishing error probability,
as long as $H(X) < M/N$, thus instantly
proves a \ind{source coding theorem}:
\begin{quote}
Given a binary source $X$ of entropy $H(X)$, and
a required compressed rate $R > H(X)$, there exists a
linear compressor $\bx \rightarrow \bz = \bH \bx \mod 2$
having rate $M/N$ equal to that required rate $R$,
and an associated uncompressor,
that is virtually lossless.
\end{quote}
% To put it another way, if you have a source of
% entropy $H(X)$ and you encode a string of
% $N$ bits from it using a \ind{hash code} (\chref{ch.hash})
% where the hash $\bz$ is of length $M$ bits,
% where $M > N H(X)$,
% a random linear hash function $\bz = \bH \bx \mod 2$
% is just as good (for collision avoidance) as a
% fully random hash function.
%% there are very unlikely to be any collisions among
%% the hashes
{This theorem is true
not only for a source of independent identically distributed
symbols but also for any source for which a typical set can be defined:
sources with memory, and time-varying sources, for example; all that's
required is that the source be ergodic.
}
\subsection*{Notes}
This method for proving that codes are
good can be applied to
other linear codes,
such as low-density parity-check codes
\cite{mncN,McElieceMacKay00}.
For each code we need an approximation of its expected weight
enumerator function.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%55
%
\dvips
% \chapter{Further exercises on information theory}
\chapter{Further Exercises on Information Theory}
% this was two chapters once
\label{ch_fInfo}
% {noisy channels}
\label{ch_f8}
\fakesection{Further exercises on noisy channels}
% I've been asked to include some exercises {\em without\/} worked
% solutions. Here are a few. Numerical solutions to some of them
% are provided on page \pageref{sec.solf8}.
%
The most exciting exercises, which will introduce you
to further ideas in information theory,
are towards the end of this chapter.
%\section{Exercises}
\subsection*{Refresher exercises on source coding and noisy channels}
\exercisaxB{2}{ex.X100}{
% from Yaser
Let $X$ be an ensemble with $\A_X = \{0,1\}$ and $\P_X = \{ 0.995,
0.005\}$. Consider source coding
using the block coding of $X^{100}$ where every $\bx
\in X^{100}$ containing 3 or fewer 1s is assigned a distinct
codeword, while the other $\bx$s are ignored.
\ben
\item
If the assigned codewords are all of the same length, find the minimum length
required to provide the above set with distinct codewords.
\item
Calculate the probability of getting an $\bx$ that will be ignored.
\een
}
\exercisaxB{2}{ex.0001}{
Let $X$ be an ensemble with $\P_X = \{ 0.1,0.2,0.3,0.4 \}$.
The ensemble is encoded using the symbol
code $\C = \{ 0001 , 001 , 01 , 1 \}$.
Consider the codeword corresponding to $\bx \in X^N$, where
$N$ is large.
\ben
\item
Compute the entropy of the fourth bit of transmission.
\item
Compute the conditional entropy of the fourth bit given
the third bit.
\item
Estimate the entropy of the hundredth bit.
\item
Estimate the conditional entropy of the hundredth bit given the
ninety-ninth bit.
% \item
\een
}
\exercisaxA{2}{ex.dicetree}{
Two fair dice are rolled by Alice and the sum is recorded.
Bob's task is to ask a sequence of questions with yes/no answers to
find out this number.
Devise in detail a strategy that achieves the minimum possible
average number of questions.
}
% added Wed 22/1/03
\exercisxB{2}{ex.fairstraws}{
How can you use a coin to \ind{draw straws} among 3 people?\index{straws, drawing}
}% my solution: arithmetic coding.
% perhaps use this in exam?
% - could also use exact sampling method! (see mcexact.tex)
\exercisxB{2}{ex.magicnumber}{
In a {magic} trick,\index{puzzle!magic trick}
there are three participants: the \ind{magician}, an assistant, and a volunteer.
The assistant, who
claims to have \ind{paranormal}\index{conjuror}\index{puzzle!magic trick}
abilities, is in a soundproof room.
The magician gives the volunteer six blank cards, five white and one blue.
The volunteer writes a different integer from 1 to 100
on each \ind{card}, as the magician is watching.
The volunteer keeps the blue card.
The magician arranges the five white cards in some order and passes them to the assistant.
The assistant then announces the number on the blue card.
How does the trick work?
}
% card trick
\exercisxB{3}{ex.magicnumber2}{
How does {\em this\/} trick work?
\begin{quote}
`Here's an ordinary pack of cards, shuffled into random
order. Please choose five cards from the pack, any that you wish. Don't
let me see their faces. No, don't give them to me: pass them to my
assistant Esmerelda. She can look at them.
`Now, Esmerelda, show me four of the cards. Hmm$\ldots$ nine of spades, six of
clubs, four of hearts, ten of diamonds. The hidden card, then, must be the
queen of spades!'
\end{quote}
The trick can be performed as described above\index{puzzle!magic trick}
for a pack of 52 cards. Use information theory
to give an upper bound
on the number of cards for which the trick can be performed.
% (This exercise is much harder than \exerciseonlyref{ex.magicnumber}.)
% Hint: think of X = the 5 cardds, Y = the seque of 4 cards. how does H(X) compare with H(Y)?
% n choose 5 cf. n....(n-3) -> (n-4)/5! = 1 -> n=124.
}
% see l/iam for soln
\exercisxB{2}{ex.Hinfty}{
Find a probability sequence $\bp = (p_1,p_2, \ldots)$ such that
$H(\bp) = \infty$.
}
\exercisaxB{2}{ex.typical2488}{
Consider a discrete memoryless source with $\A_X = \{a,b,c,d\}$
and $\P_X =$ $\{1/2,1/4,$ $1/8,1/8\}$. There are $4^8 = 65\,536$ eight-letter
words that can be formed from the four letters. Find the total number
of such words that are in the typical set $T_{N\beta}$ (equation \ref{eq.TNb})
where $N=8$ and $\beta = 0.1$.
%The definition of $T_{N\b}$, from
% chapter \chtwo, is:% equation \ref{eq.TNb}
%\beq
% T_{N\b} = \left\{ \bx\in\A_X^N :
% \left| \frac{1}{N} \log_2 \frac{1}{P(\bx)} - H \right| < \b
% \right\} .
%\eeq
}
% source coding and channels...........
\exercisxB{2}{ex.sourcechannel}{
Consider the source
$\A_S = \{ a,b,c,d,e\}$,
$\P_S = \{ \dthird, \dthird, \dfrac{1}{9}, \dfrac{1}{9}, \dfrac{1}{9} \}$ and the
channel whose transition probability matrix is
\beq
Q =
\left[
\begin{array}{cccc}
1 & 0 & 0 & 0 \\
0 & 0 & \dfrac{2}{3} & 0 \\
0 & 1 & 0 & 1 \\
0 & 0 & \dthird & 0 \\
% 1 & 0 & 0 & 0 \\
% 0 & 0 & 1 & 0 \\
% 0 & \dfrac{2}{3} & 0 & \dthird \\
% 0 & 0 & 1 & 0 \\
\end{array}\right] .
\eeq
Note that the source alphabet
% $\A_S = \{a,b,c,d,e\}$
has five symbols, but the channel
alphabet $\A_X = \A_Y = \{0,1,2,3\}$
has only four. Assume that the source produces symbols at
exactly 3/4 the rate that the channel accepts channel symbols. For a
given (tiny) $\epsilon>0$, explain how you would design a system for
communicating the source's output over the channel with an
% overall
average error probability per source symbol
less than $\epsilon$. Be as explicit as possible.
In particular, {\em do not\/} invoke Shannon's noisy-channel coding theorem.
}
% \subsection{Noisy Channels}
\exercisxB{2}{ex.C0000}{Consider a binary symmetric channel and a code
$C = \{ 0000,0011,1100,1111 \}$; assume that the
four codewords are used with probabilities
$\{ 1/2, 1/8,1/8,1/4\}$.
What is the decoding rule that minimizes the probability of
decoding error? [The optimal decoding rule depends on
the noise level $f$ of the binary symmetric channel. Give
the decoding rule for each range of values of $f$, for $f$ between 0 and
$1/2$.]
}
\exercisaxA{2}{ex.C3channel}{
Find the capacity and \optens\
% optimizing input distribution
for the three-input, three-output
channel whose transition probabilities are:
\beq
Q = \left[
\begin{array}{ccc}
1 & 0 & 0 \\
0 & \dfrac{2}{3} & \dthird \\
0 & \dthird & \dfrac{2}{3}
\end{array}\right] .
\eeq
}
%
% I am not sure I like this ex:
%
%\exercis{ex.Herrors}{
% Consider the $(7,4)$ Hamming code.
%\ben\item
% What is the probability of bit error if 3 channel errors occur
% in a single block?
%\item
% What is the probability of bit error if 4 channel errors occur
% in a single block?
%\een
%}
% \end{document}
% see also _e6.tex
%
% extra exercises do-able after chapter 6.
%
\fakesection{e6 exam qs}
\exercissxA{3}{ex.85channel}{
% Describe briefly the encoder for a $(7,4)$ Hamming code.
%
% Assuming that one codeword of this code is sent over a
% binary symmetric channel, define the {\em syndrome\/} $\bf z$
% of the received vector $\bf r$; state how many different possible syndromes
% there are; and state
% the maximum number of channel errors that the optimal decoder
%% code
% can correct.
%
% Define the {\em capacity\/} of a channel with input $x$ and output $y$
% and transition probability matrix $Q(y|x)$.
%
The input to a channel $Q$ is a word of 8 bits. The output is also
a word of 8 bits.
% A message block consisting of 8 bits is transmitted over a channel which
Each time it is used, the channel
flips {\em exactly one\/} of the transmitted bits, but
the receiver does not know which one. The other
seven bits are received without error. All 8 bits are equally likely to
be the one that is flipped. Derive the capacity
of this channel.
% Tough version:
%
% {\bf Either} show, by constructing an explicit encoder and decoder using a
% linear (8,5) code that it
% is possible to reliably communicate 5 bits per cycle
% over this channel, {\bf or} prove that no such linear (8,5) code exists.
%
% Wimps version:
% practical
Show, by describing an {\em explicit\/} encoder
% {\em and\/}
and
decoder that it
is possible {\em reliably\/} (that is, with
{\em zero\/} error probability) to communicate 5 bits per cycle
over this channel.
% Your description should be
% {\em should I give a hint here?}
% [Hint: a solution exists that involves a simple $(8,5)$ code.]
}
\exercisxB{2}{ex.rstu}{
A channel with input $x \in \{ {\tt a},{\tt b},{\tt c} \}$
and output $y \in \{ {\tt r},{\tt s},{\tt t} ,{\tt u} \}$
has conditional probability matrix:
\[
\bQ = \left[
\begin{array}{ccc}
\dhalf & 0 & 0 \\
\dhalf & \dhalf & 0 \\
0 & \dhalf & \dhalf \\
0 & 0 & \dhalf \\
\end{array}
\right] .
\hspace{1in}
\begin{array}{c}
\setlength{\unitlength}{0.13mm}
\begin{picture}(100,140)(0,-20)
\put(18,0){\makebox(0,0)[r]{\tt c}}
\put(18,40){\makebox(0,0)[r]{\tt b}}
\put(18,80){\makebox(0,0)[r]{\tt a}}
%
\multiput(20,0)(0,40){3}{\vector(2,1){36}}
\multiput(20,0)(0,40){3}{\vector(2,-1){36}}
%
\put(62,-20){\makebox(0,0)[l]{\tt u}}
\put(62,20){\makebox(0,0)[l]{\tt t}}
\put(62,60){\makebox(0,0)[l]{\tt s}}
\put(62,100){\makebox(0,0)[l]{\tt r}}
\end{picture}
\end{array}
\]
What is its capacity?
}
\exercisxB{3}{ex.isbn}{
The ten-digit number on the cover of a book known as the\index{book ISBN}
\ind{ISBN}\amargintab{t}{
\begin{center}
\begin{tabular}{l}
0-521-64298-1 \\
1-010-00000-4 \\
\end{tabular}
\end{center}
\caption[a]{Some valid ISBNs.
[The hyphens
are included for legibility.]
}
}
incorporates an error-detecting code.
The number consists of nine source digits $x_1,x_2,\ldots,x_{9}$,
satisfying $x_n \in \{ 0,1,\ldots,9 \}$, and a tenth check
digit whose value is given by
\[
x_{10} = \left( \sum_{n=1}^{9} n x_n \right) \mod 11 .
\]
Here $x_{10} \in \{ 0,1,\ldots,9 , 10 \}.$ If $x_{10} = 10$ then
the tenth digit is shown using the roman numeral X.
% $\tt X$.
% For example, 1-010-00000-4 is a valid ISBN.
% bishop
% 0-19-853864-2
% see lewis:con/isbn.p
Show that a valid ISBN satisfies:
\[
\left( \sum_{n=1}^{10} n x_n \right) \mod 11 = 0 .
\]
Imagine that an ISBN is communicated over an unreliable human
channel which sometimes {\em modifies\/} digits and sometimes
{\em reorders\/} digits.
Show that this code can be used to detect (but not correct)
all errors in which
any one of the ten digits is modified (for example,
1-010-00000-4 $\rightarrow$ 1-010-00080-4).
Show that this code can be used to detect all errors in which
any two adjacent digits are transposed (for example,
1-010-00000-4 $\rightarrow$ 1-100-00000-4).
What other transpositions of pairs of {\em non-adjacent\/}
digits can be detected?
% What types of error can be detected {\em and corrected?}
If the tenth digit were defined
to be
\[
x_{10} = \left( \sum_{n=1}^{9} n x_n \right) \mod 10 ,
\]
why would the code not work so well? (Discuss the detection of
% errors
% involving
both modifications of single digits and transpositions
of digits.)
}
\exercisaxA{3}{ex.two.bsc.choose}{
A\marginpar{\[
\setlength{\unitlength}{0.17mm}
\begin{picture}(100,140)(0,-45)
\put(15,-40){\makebox(0,0)[r]{d}}
\put(15,0){\makebox(0,0)[r]{{c}}}
\put(15,40){\makebox(0,0)[r]{b}}
\put(15,80){\makebox(0,0)[r]{a}}
\put(20,0){\vector(1,0){34}}
\put(20,40){\vector(1,0){34}}
\put(20,-40){\vector(1,0){34}}
\put(20,80){\vector(1,0){34}}
\put(20,40){\vector(1,1){34}}
% \put(20,40){\vector(1,-1){34}}
\put(20,-40){\vector(1,1){34}}
\put(20,0){\vector(1,-1){34}}
% \put(20,0){\vector(1,1){34}}
\put(20,80){\vector(1,-1){34}}
%
\put(65,-40){\makebox(0,0)[l]{d}}
\put(65,0){\makebox(0,0)[l]{c}}
\put(65,40){\makebox(0,0)[l]{b}}
\put(65,80){\makebox(0,0)[l]{a}}
\end{picture}
\]
}
channel with input $x$ and output $y$ has transition probability matrix:
\[
Q = \left[
\begin{array}{cccc}
1-f & f & 0 & 0 \\
f & 1-f & 0 & 0 \\
0 & 0 & 1-g & g \\
0 & 0 & g & 1-g
\end{array}
\right] .
\]
Assuming an input distribution of the form
\[
{\cal P}_X
= \left\{ \frac{p}{2}, \frac{p}{2} , \frac{1-p}{2} , \frac{1-p}{2} \right\},
\]
write down the entropy of the output, $H(Y)$, and the
conditional entropy of the output given the input, $H(Y|X)$.
Show that the optimal input distribution
is given by
\[
% corrected!
p = \frac{1}{1 + 2^{-H_2(g) + H_2(f) }} ,
\]
where $H_2(f) = f \log_2 \frac{1}{f} +
(1-f) \log_2 \frac{1}{(1-f)}$.
% CUTTABLE
% [You may find the identity
% $\frac{\d}{\d p} H_2(p) = \log_2 \frac{1-p}{p}$ helpful.]
\marginpar{\small\raggedright{Remember
$\frac{\d}{\d p} H_2(p) = \log_2 \frac{1-p}{p}$.}}
Write down the optimal input distribution and
the capacity of the channel in the case $f=1/2$, $g=0$,
and comment on your answer.
}
\exercisxB{2}{ex.detect.vs.correct}{
What are the differences in the redundancies needed
in an error-detecting code (which can reliably
detect that a block of data has been corrupted)
and an error-correcting code (which can detect and
correct errors)?
}
% difficult exercises see _e7
% \input{tex/_fInfo.tex}
% included directly by thebook.tex after _f8.tex
\subsection{Further tales from information theory}
The following exercises give you the chance to
discover for yourself the answers to some more surprising
results of information theory.
% \subsection{Further tales from information theory}
% \input{tex/_e7.tex}
% \noindent
\ExercisxC{3}{ex.corrinfo}{
% \item[Communication of correlated information.]
{\sf Communication of information from correlated
% dependent <--- would be better, but I want to keep same name for exercise as in first edn.
sources.}\index{channel!with dependent sources}
Imagine that we want to communicate data from
two data sources $X^{(A)}$ and $X^{(B)}$ to a central
location C via noise-free one-way \index{communication!of dependent information}{communication} channels (\figref{fig.achievableXY}a).
The signals $x^{(A)}$ and $x^{(B)}$ are strongly
dependent, so their joint information
content is only a little greater than the marginal information
content of either of them.
For example,
C is a \ind{weather collator} who wishes to receive a string of
reports saying
whether it is raining in Allerton ($x^{(A)}$)
and whether it is raining in Bognor ($x^{(B)}$).
The joint probability of $x^{(A)}$ and $x^{(B)}$ might be
\beq
\fourfourtabler{{$P(x^{(A)},x^{(B)})$}}{$x^{(A)}$}{{\mathsstrut}0}{{\mathsstrut}1}{{\mathsstrut}$x^{(B)}$}{0.49}{0.01}{0.01}{0.49}
%\fourfourtable{\makebox[0.2in][r]{$P(x^{(A)},x^{(B)})$}}{$x^{(A)}$}{{\mathsstrut}0}{{\mathsstrut}1}{{\mathsstrut}$x^{(B)}$}{0.49}{0.01}{0.01}{0.49}
%\:\:
%\begin{array}{c|cc}
%x^{(A)} :x^{(B)} & 0 & 1 \\ \hline
%0 & 0.49 & 0.01 \\
%1 & 0.01 & 0.49 \\
%\end{array}
\eeq
The weather collator would like to know $N$ successive
values of $x^{(A)}$ and $x^{(B)}$
exactly, but, since he has to pay for every bit
of information he receives,
he is interested in the possibility of avoiding buying
$N$ bits from source $A$
{\em and\/} $N$ bits from source $B$.
Assuming that variables $x^{(A)}$ and $x^{(B)}$ are generated
repeatedly from this distribution, can they be encoded at rates $R_A$
and $R_B$
in such a way that C can reconstruct all the variables, with the
sum of information transmission rates on the two lines being less than two
bits per cycle?
% For simplicity, assume that the
% one-way communication channels are noise-free binary channels.
% Encoding of correlated sources. Slepian Wolf (409)
\begin{figure}
\figuremargin{%
\begin{center}\small
\begin{tabular}{cc}
\raisebox{0.71in}{(a)\hspace{0.2in}{\input{tex/corrinfo.tex}}} &
\mbox{(b)\footnotesize
\setlength{\unitlength}{0.075in}
\begin{picture}(28,21)(-7.5,-1)
\put(0.3,0){\makebox(0,0)[bl]{\psfig{figure=figs/achievableXY.eps,width=1.5in}}}
\put(0,6.5){\makebox(0,0)[r]{\footnotesize$H(X^{(B)} \given X^{(A)})$}}
\put(0,14){\makebox(0,0)[r]{\footnotesize$H(X^{(B)})$}}
\put(0,17.5){\makebox(0,0)[r]{\footnotesize$H(X^{(A)},X^{(B)})$}}
\put(0,20){\makebox(0,0)[r]{\footnotesize$R_B$}}
%
\put(20,-0.27){\makebox(0,0)[t]{\footnotesize$R_A$}}
\put(2.5,-0.5){\makebox(0,0)[t]{\footnotesize$H(X^{(A)} \given X^{(B)})$}}
\put(12,-0.5){\makebox(0,0)[t]{\footnotesize$H(X^{(A)})$}}
%\put(15,-0.5){\makebox(0,0)[t]{\footnotesize$H(X^{(A)},X^{(B)})$}}
\end{picture}
}\\
\end{tabular}
\end{center}
}{%
\caption[a]{Communication of
% correlated
information from dependent sources.
(a)
% The communication situation:
$x^{(A)}$ and $x^{(B)}$ are dependent
sources (the dependence is represented by the dotted arrow).
Strings of values of each variable are encoded using
codes of rate $R_A$ and $R_B$ into transmissions
$\bt^{(A)}$ and $\bt^{(B)}$, which are communicated
over noise-free channels to a receiver $C$.
(b) The achievable rate region.
Both strings can be conveyed
without error even though $R_A < H(X^{(A)})$ and
$R_B < H(X^{(B)})$.
}
%
% this copy is all ready to work on......
%
% cp achievableXY.fig achievableXYAB.fig
\label{fig.achievableXY}
}%
\end{figure}
The answer, which you should demonstrate,\index{dependent sources}\index{correlated sources}
%\index{Slepian--Wolf|see{dependent sources}}
is indicated in \figref{fig.achievableXY}.
In the general
case of two dependent sources $X^{(A)}$ and $X^{(B)}$, there exist codes for
the two transmitters that can achieve reliable communication
of both $X^{(A)}$ and $X^{(B)}$ to C, as long as: the information rate from
$X^{(A)}$, $R_A$, exceeds $H(X^{(A)} \given X^{(B)})$; the information rate from
$X^{(B)}$, $R_B$, exceeds $H(X^{(B)} \given X^{(A)})$; and the total information rate
$R_A+R_B$ exceeds the joint entropy $H(X^{(A)},X^{(B)})$ \cite{SlepianWolf}.
% In the general
% case of two correlated sources $X$ and $Y$, there exist codes for
% the two transmitters that can achieve reliable communication
% of both $X$ and $Y$ to C, as long as: the information rate from
% $X$, $R(X)$, exceeds $H(X \given Y)$; the information rate from
% $Y$, $R(Y)$, exceeds $H(Y \given X)$; and the total information rate
% $R(X)+R(Y)$ exceeds the joint information $H(X,Y)$.
So in the case of $x^{(A)}$ and $x^{(B)}$ above, each transmitter must transmit
at a rate greater than $H_2(0.02) = 0.14$ bits, and the total
rate $R_A+R_B$ must be greater than 1.14 bits, for example $R_A=0.6$, $R_B=0.6$.
There exist codes that can achieve these rates. Your task is to
figure out why this is so.
Try to find an explicit solution in which one of the sources
is sent as plain text, $\bt^{(B)} = \bx^{(B)}$, and the other is
encoded.
}
% \end{description}
%\noindent
\ExercisxC{3}{ex.multaccess}{
{\sf \index{multiple access channel}Multiple
access channels}.\index{channel!multiple access}
Consider a channel with two sets of
inputs and one output --
for example, a shared telephone line (\figref{fig.achievableAB}a).
A simple model system has two binary inputs $x^{(A)}$ and $x^{(B)}$ and a ternary output $y$
equal to the arithmetic sum of the two inputs, that's 0, 1 or 2.
There is no noise. Users $A$ and $B$ cannot communicate with each other, and they
cannot hear the output of the channel.
If the output is a 0, the receiver can be certain that both inputs
were set to 0;
and if the output is a 2, the receiver can be certain that both inputs
were set to 1. But if the output is 1, then it could be that the input
state was $(0,1)$ or $(1,0)$.
How should users $A$ and $B$ use this channel so that their messages
can be deduced from the received signals? How fast can $A$
and $B$ communicate?
Clearly the total information rate from $A$ and $B$
to the receiver cannot be two bits. On the other hand it is easy to achieve
a total information rate $R_A + R_B$ of one bit. Can reliable communication
be achieved at rates $(R_A,R_B)$ such that $R_A + R_B> 1$?
\begin{figure}
\figuremargin{%
\begin{center}
\begin{tabular}{l}
(a) \hspace{0.1in}{\input{tex/multacc.tex}} \\[0.1in]
(b)\hspace{0.2in}\fourfourtabler{$y$}{$x^{(A)}$}{{\mathsstrut}$\:0\:$}{{\mathsstrut}$\:1\:$}{{\mathsstrut}$x^{(B)}$}{0}{1}{1}{2}\hspace{0.5492in}
%(c)\raisebox{-0.425in}{\psfig{figure=figs/achievableAB.eps,angle=-90,width=2in}}
(c)\raisebox{-0.25in}{\mbox{\epsfbox{metapost/channels.1}}}
\end{tabular}
\end{center}
}{%
\caption[a]{Multiple access channels.
(a) A general multiple access channel with two transmitters and one receiver.
(b) A binary multiple access channel with output
% given by adding the
equal to the sum of
two inputs.
(c) The achievable region. }
\label{fig.achievableAB}
}%
\end{figure}
The answer is indicated in \figref{fig.achievableAB}.
% There exist codes for
% the two transmitters such that the rates $(R(A),R(B))$ can be
% any point in the convex hull of
% $\{(1,0),$ $(1,.5),$ $(.5,1),$ (0,1), $(0,0)\}$.
Some practical codes for multi-user channels are presented in \citeasnoun{RatzerMacKay2003}.
}
%
% answer anything in the convex hull of 1,0, 1,.5 .5,1 0,1, 0,0
%
\ExercisxC{3}{ex.broadcast}{
{\sf \index{broadcast channel}Broadcast channels}\index{channel!broadcast}.
A broadcast channel consists of a single transmitter and
two or more receivers.