
\documentclass[12pt]{article}

\fontfamily{lmss}
\usepackage{fullpage}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{url}
\usepackage{multicol}
\usepackage{enumerate}
\usepackage{graphicx}
\usepackage{color}
\usepackage{hyperref}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,      
    urlcolor=blue,
}

\setlength{\parskip}{\baselineskip}%
\setlength{\parindent}{0pt}%

\newenvironment{sol}[1]{
\color{red}
	{\bf Solution:}
}{
}

\newenvironment{markscheme}[1]{
\color{blue}
	{\bf Marking Scheme:}
}{
}

\title{CS 486/686 Assignment 2 (126 marks in total)}
\author{Instructor: Alice Gao}
\date{Due Date: 6:00 pm on Tuesday, June 25 2019}

\begin{document}

\maketitle

\section*{Instructions}

\begin{itemize}
\item
Submit the assignment in the Dropbox labeled Assignment 2 Submissions in the Assignment 2 folder on LEARN.  
No late assignment will be accepted.  This assignment is to be done individually.
\item
For any programming question, you may use the language of your choice.  We highly recommend using Python.  If you don't know Python, it may be worthwhile to learn it for this course.
\item
Lead TAs: 
\begin{itemize}
\item
Wei Sun (w55sun@uwaterloo.ca)
\item
Ji Xin (j9xin@uwaterloo.ca)
\item
Ronghao (Steve) Yang (r39yang@uwaterloo.ca)
\end{itemize}
The TAs' office hours will be posted on the course website.
\item
Submit two files with the following naming conventions.  {\bf We will deduct 10 marks if you do not follow these conventions.}

\begin{itemize}
\item
{\bf writeup.pdf}

\begin{itemize}
\item
Include your name, email address and student ID in the writeup file.
\item
If you hand-write your solutions, make sure your handwriting is legible and take good quality pictures. You may get a mark of 0 if we cannot read your handwriting.

\end{itemize}
\item
{\bf code.zip}

\begin{itemize}
\item
Include your program, a script to run your program, and a README.txt file with instructions to run the script. You may get a mark of 0 if we cannot run your program.

\end{itemize}

\end{itemize}
\end{itemize}



\newpage
\section*{Learning goals}

{\bf Decision Trees}
\begin{itemize}
\item
Determine the prediction accuracy of a decision tree on a data set.
\item
Trace the execution of and implement the ID3 algorithm, which constructs a decision tree by selecting a feature for each node using the expected information gain metric.
\item
Extend the IDS algorithm to handle real-valued features with binary tests at each node.
\item
Implement cross-validation to prevent over-fitting.
\end{itemize}

{\bf Artificial Neural Networks}
\begin{itemize}
\item
Trace the execution of the back-propagation algorithm for learning a simple function.
\end{itemize}




\newpage
\section{Decision Trees (110 marks)}

{\bf 
Your TA Ji Xin has implemented the decision tree.  He used python and spent roughly 4.5 hours including generating the tree and plotting the tree.  Take 4.5 and multiply it by 2 or 3, that would be roughly the number of hours you will spend working on this problem.  Please start working on this problem as early as possible.  Feel free to reach out to Ji if you have implementation specific questions.  He will also have office hours on Saturday afternoons.
}

You will implement a decision tree learning algorithm to detect whether a room is occupied based on the temperature, humidity, light and carbon dioxide in the room.   The data set is adapted from the \href{https://archive.ics.uci.edu/ml/datasets/Occupancy+Detection+}{Occupancy Detection Data Set} in the UCI Machine Learning Repository.

You are provided with two data sets {\bf occupancy\_A} and {\bf occupancy\_B} on the course website. Data set A has five features: Temperature, Relative Humidity, Light, CO2, and Humidity Ratio.  Data set B is identical to data set A except that the feature Light is removed.

%Both data sets have real-valued features.  When generating a decision tree, use only binary tests at each node.  That is, you should choose a threshold value for each feature and each node in the decision tree should test whether a feature has a value greater than a threshold value or not.  At each node, decide the feature to test and the threshold value using the information gain metric.  Along a path from the root node to a leaf node, a real-valued feature may be tested multiple times with different threshold values.

%{\bf If your code is incomplete or not working well, use the following command to answer the questions to earn up to 48 marks.  See the detailed mark breakdowns in each section.}
%\begin{verbatim}
%sklearn.tree.DecisionTreeClassifier(criterion=’entropy’) 
%\end{verbatim}

{\bf Please complete the following tasks.}
\begin{enumerate}[A.]
\item
\label{part_1}

Write a program to learn a decision tree (denoted by T1) using the ID3 algorithm on the data set $A$.  For each node in the tree, consider every feature and consider all possible split points for each feature using the rules described in class.    Choose a feature and a split point for the feature using the expected information gain metric.

The rules for choosing split points will be described in class on Monday, June 10. If you want to get started on this during this weekend, you can implement the following to start with. 
\begin{itemize}
\item
Consider a real-valued feature.  Sort the data set based on the values of the feature.
\item
All the possible split points are the values that are midway between two consecutive but different values for the feature.
\end{itemize}

We have broken down this problem into several parts for you. This is to help you build
your program incrementally and debug the individual helpers functions before you test
the entire program. You do not have to follow the steps below exactly. {\bf However, if
a step requires you to implement a function, make sure your function uses
the provided name.} If your program does not run or does not produce the correct
solution, you may receive part marks for implementing the individual helper functions
correctly.

If you like, follow the recommended steps below to implement your program.

\begin{enumerate}[(a)]
\item
Implement a function {\bf read\_data(id)} which reads in the data set.   The id parameter can be used to specify which dataset the function is trying to read in.
\item
Implement a function {\bf entropy(dist)} which returns the entropy of the given probability distribution.  If you are using Python, you may want to use the Counter object to represent the distribution.
\item
Implement a function {\bf choose\_feature\_split(data)} which iterates through all the features and all possible split points of each feature and returns one feature and one split point for the feature that maximizes the expected information gain.
\item
Implement a few functions to operate on a node in the decision tree.  In our implementation, we wrote a node class and implemented the following functions inside the class.
\begin{itemize}
\item
{\bf init\_node(node)} stores useful initial information in the node.
\item
{\bf split\_node(node, data, max\_depth)} produces the node based on the data points left and recursively produces the children of the current node if necessary.
\item
{\bf get\_decision(node, one\_data\_point)} returns the decision/label given a data point.  If the given node is a leaf node, this function returns the decision in the leaf node.  Otherwise, this function recursively calls itself on the child nodes until it reaches a decision.
\end{itemize}
\item
Implement a few functions to operate on a decision tree.  In our implementation, we wrote a tree class and implemented the following functions inside the class.  The tree class is essentially a wrapper of the node class with additional information stored in it.
\begin{itemize}
\item
{\bf init\_tree(tree)} initializes the decision tree by creating the root node.
\item
{\bf train\_tree(tree, data, max\_depth)} trains the decision tree using the provided data and up to the maximum depth if specified.   In our implementation, this function calls the {\bf split\_node} function to produce the root node of the decision tree.
\item
{\bf get\_prediction\_accuracy(tree, data)} returns the prediction accuracy of the decision tree on the provided data.  In our implementation, this function calls the {\bf get\_decision} function on the root node of the tree.
\item
{\bf plot\_tree(tree, file\_path)} produces a plot of the tree and saves the plot in the file at the provided path.  If you are using Python, we recommend using the \href{https://anytree.readthedocs.io/en/latest/}{anytree} python package to plot the tree.
\end{itemize}
\end{enumerate}

{\bf What to submit:}

In your writeup.pdf, answer the following questions.
\begin{itemize}
\item
What is the maximum depth of the decision tree generated?  The depth of the root node is 1.
\item
What is the prediction accuracy of the decision tree on data set $A$?
\item
Based on the tree you generated, what is the most informative feature in this data set?  Why?  Discuss your thoughts in no more than 1 paragraph.
\end{itemize}

In your code.zip, save the decision tree T1 in a file named {\bf t1.pdf}.

In your code.zip, submit your program for generating the decision tree T1.  If it is complicated to run your program, be sure to write a script to run your program and to generate the decision tree.  In your README file, provide clear instructions for running your program.   If you want the TA to run your program in a particular environment and use a particular version of a programming language, be sure to provide enough information on these.  The TA will not modify your source code to run your program.

\begin{markscheme}

(30 marks)

\begin{itemize}
\item
(2 marks) Maximum depth of the tree
\item
(2 marks) Prediction accuracy of the tree on data set A
\item
(6 marks) Discussion of the most informative feature
\item
(10 marks) The TA can run your program to produce the decision tree T1 in under 5 minutes and the produced tree is the same as the tree in {\bf t1.pdf} you provided.
\item
(10 marks) Your decision tree T1 is similar to our decision tree T1 except for any differences caused by noise in the data.
\end{itemize}

\end{markscheme}



\item
\label{part_2}
The decision tree T1 should have achieved pretty good accuracy on data set A.  However, this does not mean that it will generalize well to unseen data.  In this part, you will experiment with generating a smaller decision tree using cross-validation.

Implement a modified version of the ID3 algorithm which takes as input a maximum depth.  You will use this modified ID3 algorithm and five-fold cross-validation to determine the best maximum depth of the decision tree.

In five-fold cross-validation, each data point serves double duty --- as training data and validation data.  First, randomly shuffle all the data points.  Next, split the data into five equal subsets.  For each maximum depth, perform five rounds of learning.  In each round, $\frac{1}{5}$ of the data is held out as a validation set and the remaining data points are used as training set.  Generate a decision tree with the maximum depth using only the training data.  Then, calculate the prediction accuracy of the decision tree on the training data and on the validation data.  (Note that, in each round, you will likely generate a different decision tree.  However, we do not care about all the different decision trees generated.  We only care about the average prediction accuracy of the trees.) After the five rounds of learning, calculate the average prediction accuracy of the decision tree on the training data and on the validation data, where the average is taken over the five rounds of learning.  That is, for each maximum depth, you will produce two numbers, the average prediction accuracy on the training set and the average prediction accuracy on the validation set.  Finally, choose the maximum depth with the highest average prediction accuracy on the validation set. 

Note that I did not specify the range of values for the maximum depth.  Experiment with different ranges and come up with one that is reasonable.  You will want the range to be large enough so that you can find the maximum depth that maximizes the prediction accuracy on the validation set.

Having determined the best maximum depth for our decision tree, you will generate a decision tree (denoted by T2) with the best maximum depth using the entire data set A.  

For the implementation, in addition to the functions specified in part~\ref{part_1}, we also recommend that you implement a function {\bf cross\_validation(data, file\_path)} to perform five-fold cross-validation on the provided data set, saves a plot of the training accuracy and validation accuracy in the file at the provided path, and returns the best maximum depth of the decision tree.

{\bf What to submit:}

In your writeup.pdf, include the following items.
\begin{itemize}
\item
Plot the results of the five-fold cross-validation.  That is, plot the average prediction accuracy on the training set (on the y-axis) and on the validation set (on the y-axis) with respect to the maximum depth of the decision tree (on the x-axis).  Please put both plots on the same graph.
\item
What is the maximum depth of the decision tree that maximizes the average prediction accuracy on the validation set?

\item
What is the prediction accuracy of the decision tree {\bf T2} on data set A?
\item
Compare the decision trees T1 (from part~\ref{part_1}) and T2 (from this part).  What are their similarities and differences? Which decision tree would you prefer and why? Discuss your observations in no more than 2 paragraphs.
\end{itemize}

In your code.zip, save the decision tree T2 in a file named {\bf t2.pdf}.

In your code.zip, include the program you used to determine the best maximum depth of the decision tree using five-fold cross validation and to generate the decision tree with the best maximum depth.  Provide clear instructions for running your program.  See more details in the last paragraph of part~\ref{part_1}.  In particular, provide clear instructions for the TA to run your program and to produce
\begin{itemize}
\item
the plot of the results of five-fold cross-validation and
\item 
the decision tree T2.
\end{itemize}

\begin{markscheme}

(50 marks)

\begin{itemize}
\item
(10 marks) The TA can run your program to reproduce the plot of the results of the five-fold cross-validation in under 5 minutes and the produced plot is the same as the one provided in your writeup.pdf.
\item
(10 marks) Plot of the results of the five-fold cross-validation.

5 marks for the plot of the average prediction accuracy on the training set with respect to the maximum depth.

5 marks for the plot of the average prediction accuracy on the validation set with respect to the maximum depth. 

\item
(2 marks) Maximum depth of the decision tree
\item
(2 marks) Prediction accuracy of the decision tree T2 on data set A
\item
(6 marks) Discussion of a comparison between T1 and T2.
\item
(10 marks) The TA can run your program to reproduce the decision tree T2 in under 5 minutes and the produced tree is the same as the tree in {\bf t2.pdf} you provided.
\item  
(10 marks) Your decision tree T2 is similar to our decision tree T2.  These 10 marks will be awarded based on the TA's discretion.
\end{itemize}

\end{markscheme}

\item
\label{part_3}
We've constructed data set B, which is identical to data set A except that the feature Light is removed.  Repeat part~\ref{part_2} for data set B.  First, figure out the best maximum depth of the tree using five-fold cross-validation.  Next, train a decision tree (denoted T3) with the best maximum depth on the entire data set B.

{\bf What to submit:}

In your writeup.pdf, include the following items.
\begin{itemize}
\item
Plot the results of the five-fold cross-validation.  That is, plot the average prediction accuracy on the training set (on the y-axis) and on the validation set (on the y-axis) with respect to the maximum depth of the decision tree (on the x-axis).  Please put both plots on the same graph.
\item
What is the maximum depth of the decision tree that maximizes the average prediction accuracy on the validation set?

\item
What is the prediction accuracy of the decision tree with the best maximum depth on data set B?
\item
What are the similarities and differences between the decision trees T2 (from part~\ref{part_2}) and T3 (from this part)?  How did removing the feature Light impact the shape and the size of the decision tree?  Discuss your observations in no more than 2 paragraphs.
\end{itemize}

In your code.zip, save the decision tree T3 in a file named {\bf t3.pdf}.

The TA will not be running your program for this part.  If your program for part~\ref{part_2} runs, it likely runs for this part as well. =)


\begin{markscheme}

(30 marks)

\begin{itemize}
\item
(10 marks) Plot of the results of the five-fold cross-validation.

5 marks for the plot of the average prediction accuracy on the training set with respect to the maximum depth.

5 marks for the plot of the average prediction accuracy on the validation set with respect to the maximum depth. 

\item
(2 marks) Maximum depth of the decision tree
\item
(2 marks) Prediction accuracy of the decision tree T3 on data set B
\item
(6 marks) Discussion of a comparison between T2 and T3.
\item  
(10 marks) Your decision tree T3 is similar to our decision tree T3.  These 10 marks will be awarded based on the TA's discretion.
\end{itemize}

\end{markscheme}

\end{enumerate}

\newpage
\section{Neural Networks (16 marks)}

Consider the function $f$ defined by the following table.

\begin{table}[ht]
\label{dataset_1}
\centering
\begin{tabular}{cc|c}
$x_1$ & $x_2$ & $y$ \\ \hline
1 & 1 & 1 \\ \hline
2 & 2 & 0 \\ \hline
3 & 2 & 1 \\ \hline
2 & 3 & 1 \\ 
\end{tabular}
\caption{A simple function defined by four points}
\end{table}

Answer the following questions.
\begin{enumerate}
\item
Consider a perceptron with two inputs $x_1$ and $x_2$, a bias input $x_0 = 1$, three weights $w_0$, $w_1$ and $w_2$, and an output $o$.  Let $o = 1$ when $\displaystyle \sum_{i = 0}^2 (w_i x_i) \ge 0$ and $o = 0$ when $\displaystyle \sum_{i = 0}^2 (w_i x_i) < 0$.  Can the function $f$ be represented by this perceptron?  Why or why not?  Justify your answer in no more than 1 paragraph.

\begin{markscheme}

(6 marks)

\begin{itemize}
\item
(2 marks) Correct answer.
\item
(4 marks) Correct justification.
\end{itemize}

\end{markscheme}

\item
We can learn this function $f$ in Table~\ref{dataset_1} using a multi-layer feed-forward neural network shown in Figure~\ref{nn_1} below.  The activation function is the sigmoid function 
$\displaystyle y = \frac{1}{1 + e^{-x}}$. To determine the weights, we will use the back-propagation algorithm discussed in class.  

\begin{figure}[ht]
\label{nn_1}
\centering
\includegraphics[width=0.5\textwidth]{images/neural_network.png}
\caption{A neural network with one hidden layer}
\end{figure}

For this part, you will {\bf execute one iteration of the back-propagation algorithm}.  One iteration consists of performing the forward propagation and updating all the weights by summing the gradients for all four points in the data set.  

Use the learning rate $\alpha = 1$ and start with the following initial weights.
\begin{itemize}
\item
$w1_{01} = 0.9$, $w1_{02} = 0.4$, $w1_{11} = 0.7$, $w1_{12} = 0.3$, $w1_{21} = 0.6$, and $w1_{22} = 0.8$
\item
$w2_{01} = 0.8$, $w2_{11} = 0.7$, $w2_{21} = 0.7$.
\end{itemize}

Be sure to include the following values in your answer.
\begin{itemize}
\item
The values of the hidden units $h_1$ and $h_2$
\item
The values of the output unit $o_1$
\item
The updated values of $w1$ and $w2$
\end{itemize}

\begin{markscheme}

(10 marks)

\begin{itemize}
\item
(4 marks) values of $h_1$ and $h_2$
\item
(2 marks) values of $o_1$
\item
(4 marks) updated values of $w_1$ and $w_2$
\end{itemize}

\end{markscheme}

\end{enumerate}

%\newpage
%\section{Peer Grading A1}

\end{document}





























