%Proof Complexity Scribe Notes template \ifx\CompleteCourse\relax \ClassScribeSetupA \else \documentclass[11pt]{article} \newtheorem{theorem}{Theorem} \newtheorem{lemma}[theorem]{Lemma} \newtheorem{corollary}[theorem]{Corollary} \newtheorem{proposition}[theorem]{Proposition} \newtheorem{homework}{Homework} \newenvironment{definition}{\begin{trivlist}\item[]{\bf Definition}\ }% {\end{trivlist}} \newenvironment{fact}{\begin{trivlist}\item[]{\bf Fact}\ }% {\end{trivlist}} \newenvironment{example}{\begin{trivlist}\item[]{\bf Example}\ }% {\end{trivlist}} \newenvironment{proof}{\begin{trivlist}\item[]{\bf Proof}\ }% {\end{trivlist}} % Make the page large \addtolength{\textwidth}{1.50in} \addtolength{\textheight}{1.00in} \addtolength{\evensidemargin}{-0.75in} \addtolength{\oddsidemargin}{-0.75in} \addtolength{\topmargin}{-.50in} % \vdashsub{X} makes a turnstyle with subscript "X" % \vdashsup{X} makes a turnstyle with superscript "X" \newdimen\srbdimenA \newcommand{\vdashsupsub}[2]{ \mathop{ \setbox251 = \hbox{$\scriptstyle #1$} \setbox252 = \hbox{$\scriptstyle #2$} \ifdim \wd251<\wd252 \srbdimenA = \wd252 \else \srbdimenA = \wd251 \fi \setbox255 = \hbox {${\srbAvdash \vphantom( \kern -\srbdimenA \kern +.05em} ^{\hbox to\srbdimenA{\hfill \box251\hfill}} _{\hbox to\srbdimenA{\hfill \box252\hfill}}$} \box255 \kern .05em}} \newcommand{\srbAvdash}{\hbox{ \vrule height1.4ex width0.02em \dimen255 = \srbdimenA \advance\dimen255 by 0.1em \vbox{\hrule width\srbdimenA height0.02em \kern .65ex }}} \newcommand{\vdashsup}[1]{\vdashsupsub{{#1}}{\mbox{~}}} \newcommand{\vdashsub}[1]{\vdashsupsub{\mbox{~}}{#1}} \fi % FOR THE SCRIBE: CUSTOMIZE THE ENTRIES BELOW: % Fill in the following information particular to these scribe notes: \def\scribeone{Danny Heap} % Who is the scribe? \def\classdate{15 September 2005} % Date of the class \def\classnumber{1} % Is this the first, second, ...? % Here are some commands that stay the same for the whole class. \def\classinstructor{Toniann Pitassi} \def\classtitle{Machine Learning Theory} \def\doctitle{\textup{CS 2416 - Machine Learning Theory}} \def\classid{\textup{Lecture \#\classnumber: \classdate}} % Put your macros for these scribe notes HERE % It is best to use as few as possible. % environments for "theorem", "corollary", "lemma", "fact" "definition" % "homework", "proof", "example", "proposition" % are already defined above. % Start the document \ifx\CompleteCourse\relax \ClassScribeSetupB \else \def\makeatletter{\catcode`\@=11\relax} \def\makeatother{\catcode`\@=12\relax} \makeatletter \def\ps@scribeheadings{\let\@mkboth\@gobbletwo \def\@oddhead{\sl\doctitle \hfill \classid }\def\@oddfoot{\hfil \rm \thepage \hfil}\def\@evenhead{\@oddhead}% \def\@evenfoot{\@oddfoot}\def\sectionmark##1{}\def\subsectionmark##1{}} \makeatother \pagestyle{scribeheadings} \begin{document} \bibliographystyle{siam} \fi \begin{center} \Large\bf\doctitle\\[1em] \Large\bf\classid\\[1em] {\large\bf Lecturer: \classinstructor}\\[.5em] {\large\bf Scribe Notes by: \scribeone} \end{center} \vspace*{.4in} % HERE IS WHERE YOUR SCRIBE NOTES SHOULD START % DELETE ALL OF ROB'S TEXT AND ENTER YOUR OWN. Machine Learning tackles many real-world problems and tries to develop algorithms where (we suspect) an explicit algorithm doesn't exist. For example: distinguish chairs from tables. In Machine Learning Theory (MLT) we develop models for machine learning and evaluate ML algorithms with respect to those models. Benefits of MLT include identifying intractable problems (which need to be re-specified to have any hope of being tractable), and suggest approaches to developing new ML algorithms. Here are some ML models we'll consider: \begin{itemize} \item Consistency model \item On-line model \item Occam \item PAC \end{itemize} Some concepts we'll look at: \begin{itemize} \item VC-dimension (a measure over vectors that gives an idea of the necessary sample size for ML). \item boosting (use an existing ML algorithm with error $\epsilon$ to build a better ML algorithm with error whose reciprocal is exponential in $1/\epsilon$. \item hard-core sets and distributions, from complexity theory. \end{itemize} We're heading towards the PAC model. But first, we'll look at some other models in (chronological?) order. Assume that we're restricted to searching for boolean-valued functions on $\{0,1\}^n$. We're given access to instance examples that may include a label from $\{-,+\}$ indicating whether they are negative or positive instances. Our instances may be provided in a batch, after which we're expected to make predictions about the labelling of subsequent instances (or output a hypothesis function that produces such a label), or our predictions may be required after each instance is supplied, in a sequence. \section*{Consistency model} \begin{definition} Concept $c$ is a boolean function over $\{0,1\}^n$. The domain variables are denoted by $x_1,\ldots, x_n$, A {\it labelled example} is a pair $\langle \alpha, \beta\rangle$ where $\alpha\in \{0,1\}^n$ and $\beta \in \{-,+\}$ indicates whether $\alpha$ is a positive or negative example. A {\it concept class} $\cal{C}$ is a set of concepts over $x_1, \ldots, x_n$ \textbf{usually} with an associated representation. For example, all CNF formulas over $x_1, \ldots, x_n$. \end{definition} \begin{definition} Algorithm $A$ learns $\cal{C}$ in the Consistency Model if given any set $S$ of labelled examples, algorithm $A$ produces concept $c\in \cal{C}$ consistent with $S$ if such a concept exists, and outputs "there is no consistent concept" otherwise. Furthermore, the runtime of $A$ should be polynomial in the size of $S$. \end{definition} Note that in the above definition, we require that $c\in \cal{C}$, otherwise simply taking the disjunction of all the positive examples in $S$ will suffice (and would be ugly, slow, and of no predictive value). If we don't care about run-time, devising $A$ is simple: just try concepts from $\cal{C}$. For example, if $\cal{C}$ is the set of all monotone conjunctions, then $A$ would simply try all $2^n$ possible conjunctions of variables in $\{x_1, \ldots, x_n\}$. Thus we also require that $A$ should run in time polynomial in the size of $S$. (each example in $S$ requires $n+1$ bits). % We % might further learning polynomially-many samples in polynomial % time.\marginpar{what does this last complexity bound mean?} \begin{itemize} \item[1.] Example 1: monotone conjunctions over $x_1, \ldots, x_n$. Suppose the target concept (which distinguishes the positive and negative labels) is $x_3 x_4x_7$, so $|\cal{C}|$ $=$ $2^n$, and you're given labelled pairs: \begin{eqnarray*} && \langle 10100110, -\rangle \\ && \langle 11111010, +\rangle \\ && \langle 00110010, +\rangle \\ && \langle 11011111, -\rangle \\ && \langle 11001100, -\rangle \end{eqnarray*} Algorithm $A$ simply takes the conjunction of all the $x_i$ where all positive examples have a 1 in position $i$, and verifies that this conjunction is falsified by all the negative examples (otherwise output ``no consistent concept''). Note that any variable that does not appear in the conjunction had to be eliminated because it was set to zero by some positive example. Thus if our conjunction is not consistent with the negative examples, then no conjunction will be. \item[2.] Example 2: monotone disjunctions over $x_1, \ldots, x_n$. Suppose the target concept $c= x_3\vee x_4\vee x_7$, and you're given labelled pairs: \begin{eqnarray*} && \langle 01000100, -\rangle \\ && \langle 11001100, -\rangle \\ && \langle 00001101, -\rangle \\ && \langle 10101001, +\rangle \\ && \langle 01011100, +\rangle \end{eqnarray*} Algorithm $A$ simply takes the disjunction of all the $x_i$ where all negative examples have a zero in position $i$, and verifies that this disjunction satisfies all the positive examples (otherwise output ``no consistent concept''). \item[3.] Example 3: non-monotone conjunctions/disjunctions. These are taken over $\{x_1, \overline{x_1}, \ldots, x_n, \overline{x_n}\}$. We can perform a reduction to the monotone case. Define $y_i = \overline{x_i}$. Under this transformation, the target function is a monotone conjunction over this space of $2n$ variables, $x_1, \ldots, x_n$ and $y_1, \ldots, y_n$. \item[4.] Example 4: $k$-DNF formula. These are disjunctions of terms, each term being the conjunction of at most $k$ literals. Again we can reduce this to the case of monotone disjunctions as follows. For each possible term of size at most $k$, introduce a new variable representing this term, and use Example~2. \item[5.] Example 5: 2-term DNF. For example, $c= x_1\overline{x_3}x_5 \vee x_1\overline{x_4}x_5\overline{x_3}$. This \textbf{cannot} be learned in polytime unless $P= NP$. This (effective) intractability is due to the fact that $A$ must find a hypothesis that is a 2-term DNF. See the book for a proof of intractability. \item[6.] Example 6: DNF formula. Find a consistent DNF concept, with no restriction on the number of terms. This is trivially learnable in the consistency model, since we can take the disjunction of terms, one term for each positive example. This example raises a red flag, since we are simply memorizing the data. We don't expect that the hypothesis returned will be very similar to the true DNF that the examples came from. We will see later that for true learning (being able to predict well on future examples), there needs to be compression. That is, the hypothesis returned should be of size that is sublinear in the size of the samples, $S$. \end{itemize} \bigskip \noindent{\bf Some problems with the Consistency Model:} \begin{itemize} \item DNF learner is just memorizing the input (we'd like the output concept to be substantially smaller than the input). \item Insisting on the algorithm found by $A$ being in $\cal{C}$ seems a bit extreme. \item If the examples aren't completely consistent with any $c\in \cal{C}$ there's no way to output something ``close.'' \end{itemize} \section*{Online/Mistake Bound (MB) model} \label{mistakeBound} As we saw above, the consistency model has the problem that a positive result doesn't always imply finding a rule with good predictive power. The Mistake Bound model addresses this problem by expliitly modelling learning as an on-line process. Predictions are interleaved with examples (as opposed to learning from a batch of examples before making a prediction). Learning takes place in stages. Each stage consists of: \begin{enumerate} \item $A$ (the learner) is provided with an unlabelled example. \item $A$ makes a prediction about the label (equivalently produces a hypothesis for $h\in \cal{C}$ that would produce the right label). \item $A$ is told the correct label (the one produced by the target concept $c\in \cal{C}$. \end{enumerate} \begin{definition} Let $\cal C$ be a concept class over $x_1, \ldots x_n$ together with an associated representation. $A$ learns $\cal{C}$ over $x_1, \ldots, x_n$ in MB model: if for any sequence of examples consistent with some $c\in \cal{C}$ the total number of mistakes made by $A$, when run on this sequence of examples, is polynomial in $n$ and $s$, where $s$ is the description size of $c$. (The extra parameter $s$ is necessary since a concept $c$ may be exponentially long in $n$.) $A$ learns $\cal{C}$ in polytime if it completes each stage/step in time polynomial in $n$ and $s$. \end{definition} \bigskip \noindent {\bf Example 1}: non-monotone conjunctions: The learner starts with $h= x_1 \overline{x_1}x_2\overline{x_2} \cdots x_n\overline{x_n}$. For each mistake on a positive example, remove all unsatisfied literals from $h$ (there'll be $n$ of them removed on the first mistake, and at least 1 removed on each subsequent mistake). If $h$ makes a mistake on a negative example, output ``no consistent concept.'' The mistakes on the positive examples are bounded by $n+1$, since $n$ literals are removed after the first example, and at least one literal is removed per subsequent mistake, and there are at most $2n$ literals altogether. Further, there will be no mistakes on the negative examples as long as the examples are consistent with some conjunction. This follows because we only threw out literals when we absolutely had to (to be consistent with the positive examples), so if *any* conjunction is consistent with the negative examples as well, then our current conjunction will be. \end{document}