\documentclass[12pt]{article}

\fontfamily{lmss}
\usepackage{fullpage}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{url}
\usepackage{multicol}
\usepackage{enumerate}
\usepackage{graphicx}
\usepackage{color}
\usepackage{hyperref}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,      
    urlcolor=blue,
}

\setlength{\parskip}{\baselineskip}%
\setlength{\parindent}{0pt}%

\newenvironment{sol}[1]{
\color{red}
	{\bf Solution:}
}{
}

\newenvironment{markscheme}[1]{
\color{blue}
	{\bf Marking Scheme:}
}{
}

\usepackage{tikz}
\usetikzlibrary{arrows.meta}
\usetikzlibrary{arrows}
\usetikzlibrary{shapes}

\usepackage{array}
\newcolumntype{L}[1]{>{\raggedright\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{R}[1]{>{\raggedleft\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}



\title{CS 486/686 Assignment 4 (84 marks in total)}
\author{Instructor: Alice Gao}
\date{Due Date: 6:00 pm on Tuesday, July 30, 2019}

\begin{document}

\maketitle

\section*{Instructions}

\begin{itemize}
\item
Submit the assignment in the Dropbox labeled Assignment 4 Submissions in the Assignment 4 folder on LEARN.  
No late assignment will be accepted.  This assignment is to be done individually.
\item
For any programming question, you may use the language of your choice.  We highly recommend using Python.  If you don't know Python, it may be worthwhile to learn it for this course.
\item
Lead TAs: 
\begin{itemize}
\item
Atrisha Sarkar (a9sarkar@uwaterloo.ca)
\item
Alexandre Parmentier (aparment@uwaterloo.ca)
\end{itemize}
The TAs' office hours will be posted on the course website.
\item
Submit two files with the following naming conventions.  {\bf We will deduct 10 marks if you do not follow these conventions.}

\begin{itemize}
\item
{\bf writeup.pdf}

\begin{itemize}
\item
Include your name, email address and student ID in the writeup file.
\item
If you hand-write your solutions, make sure your handwriting is legible and take good quality pictures. You may get a mark of 0 if we cannot read your handwriting.

\end{itemize}
\item
{\bf code.zip}

\begin{itemize}
\item
Include your program, a script to run your program, and a README.txt file with instructions to run the script. You may get a mark of 0 if we cannot run your program.

\end{itemize}

\end{itemize}
\end{itemize}



\newpage
\section*{Learning goals}

{\bf Markov Decision Process}
\begin{itemize}
\item
Trace the execution of and implement the value iteration algorithm to solve a Markov decision process.
\end{itemize}


\section{Markov Decision Processes (84 marks)}

In this question, we will continue to study the grid world discussed in class.  We will vary the reward function $R(s)$ for any state $s \ne s_{24}$ and $s \ne s_{34}$ and investigate how the optimal policy changes as $R(s)$ changes.

Assume that $R(s_{24}) = -1$, $R(s_{34}) = +1$, and the discount factor $\gamma = 1$. 
The transition model stays the same: the agent moves in the intended direction with probability $0.8$, moves to the left of the intended direction with probability $0.1$, and moves to the right of the intended direction with probability $0.1$.

\begin{center}
\begin{tabular}{ | C{2cm} | C{2cm} | C{2cm} | C{2cm} | C{2cm} |} \hline
& 1 & 2 & 3 & 4  \\ \hline
1& &     &  &  \\ \hline
2& & X &  & -1  \\ \hline
3& &     &  & +1 \\ \hline
\end{tabular}
\end{center}

\begin{enumerate}
\item
Consider $R(s) = -1.0, \forall s \ne s_{24} \land s \ne s_{34}$.  In iteration $0$, let the true utility of every state to be $0$.  Implement the value iteration algorithm.  Execute the value iteration algorithm until the true utilities of all the states have converged.  Each true utility has converged when it stops changing up to three decimal places. 

Please show the optimal policy in a table format.  See the table below for an example.  $s_{23}$ shows a case when there is a tie between multiple actions for the optimal policy.

\begin{center}
\begin{tabular}{ | C{2cm} | C{2cm} | C{2cm} | C{2cm} | C{2cm} |} \hline
& 1 & 2 & 3 & 4  \\ \hline
1& $\uparrow$ & $\downarrow$ & $\leftarrow$ & $\rightarrow$  \\ \hline
2& $\uparrow$ & X & $\downarrow$  $\uparrow$ & -1  \\ \hline
3& $\leftarrow$ & $\rightarrow$ & $\uparrow$ & +1 \\ \hline
\end{tabular}
\end{center}


{\bf What to submit:}
\begin{itemize}
\item
Show the optimal policy of the MDP.  If there is a tie between multiple actions, show all of them as the optimal policy for a state.
\item
Show the true utilities of all the states until the iteration when all the true utilities have stopped changing up to three decimal places.  That is, execute the algorithm until the earlier iteration $K$ such that the true utilities for all the states are the same in iteration $K-1$ and iteration $K$.  
\item 
A program you used to execute the value iteration algorithm and produce the output.
\end{itemize}

\begin{markscheme}

(36 marks)

\begin{itemize}
\item
(6 marks) The TA can run your program to produce the specified output.
\item
(24 marks) The output shows the correct true utilities of all the states.
\item 
(6 marks) The optimal policy is correct.
\end{itemize}

\end{markscheme}

\item
Determine how the optimal policy changes when $R(s)$ changes from $-1.6$ to $-0.5$.  

You only need to consider the values of $R(s)$ up to one decimal place.  That is, you only need to consider $R(s) = -1.6, -1.5, -1.4, -1.3, -1.2, -1.1, -1.0, -0.9, -0.8, -0.7, -0.6, -0.5$.

{\bf What to submit:}
\begin{itemize}
\item
Divide up the values of $R(s)$ into ranges such that the optimal policy for each range of values remains the same.
\item
State each range of values and state the optimal policy.  If there is a tie between multiple actions, show all of them as the optimal policy for a state.
\item
Indicate how the optimal policy changes from the last value in one range to the first value in the next range.
\end{itemize}

\begin{markscheme}

(24 marks)

\begin{itemize}
\item
(6 marks) Correct ranges of values
\item
(12 marks) Correct optimal policies
\item
(6 marks) Correct descriptions of how the optimal policy changes from one range to the next
\end{itemize}

\end{markscheme}

\item
Determine how the optimal policy changes when $R(s)$ changes from $-0.08$ to $-0.03$. 

You only need to consider the values of $R(s)$ up to two decimal places.  That is, you only need to consider $R(s) = -0.08, -0.07, -0.06, -0.05, -0.04, -0.03$.

{\bf What to submit:}
\begin{itemize}
\item
Divide up the values of $R(s)$ into ranges such that the optimal policy for each range of values remains the same.
\item
State each range of values and state the optimal policy.  If there is a tie between multiple actions, show all of them as the optimal policy for a state.
\item
Indicate how the optimal policy changes from the last value in one range to the first value in the next range.
\end{itemize}

\begin{markscheme}

(24 marks)

\begin{itemize}
\item
(6 marks) Correct ranges of values
\item
(12 marks) Correct optimal policies
\item
(6 marks) Correct descriptions of how the optimal policy changes from one range to the next
\end{itemize}

\end{markscheme}

\end{enumerate}

\end{document}





























