CSC350F Numerical Algebra and Optimisation -- Undergraduate Computer Science Course

CSC350F Numerical Algebra and Optimisation

Fall 2013

Course information for current students:

14-12-2013: Marks for the final exam and unofficial course marks are at the cdf website. Have a successful remaining exam term, and a relaxing holiday season!
5-12-2013: Please remember to submit the course evaluations. You must have received a message by the automated system. Deadline is Sunday. So far there are only 4 responses, and they are too few to form any statistical results.
6-12-2013: Assignment 3 is marked and available for pickup. You can pick it up even earlier than 4PM, as long as I am in the office. Marks are available through the cdf website.
3-12-2013: Office hours Wednesday 4-12-2013, 1:30 - 2:30 PM, Thursday 5-12-2013, 4:30 - 5:30 PM, Friday 6-12-2013, 4:00 - 5:00 PM (note the change), Monday 9-12-2013 12:30 - 1:30 PM.
28-11-2013: Regarding the marks posted on cdf: There have been some issues reported about students not finding their term marks on the cdf system. I don't know exactly how this system works, but, from what I understand, you go to https://www.cdf.toronto.edu/students/ and type your login and password. (I am not sure if you also have to type the student id). Then you pick the course you are registered in and click "view courses grades". The system should normally find your marks using the internally stored student id.
If at some point you are asked to type in your student id, there are two ways of typing it in: (a) if your id has 9 digits (usually starts with a 9), then either give it as is, or give it with a preceding 0 (to make it 10-digit); (b) if, on the other hand, your id has 10 digits (usually starts with a 1), then either give it as is, or give it without the leading 1 (to make it 9-digit) . I hope one of the methods will work. If not, e-mail me.
6-11-2013: Assignment 3 is out. Due date Tue 3-12-2013. Please note that no late assignments accepted, as the final exam is early, and we may need to discuss solutions.
You may find the following code useful for question 1:
```
eA = [-4.57e-01; -6.39e-02; 1.32e-01; 3.42e-02; -1.49e-02; 9.69e-03; -2.58e-03; 3.56e-03; 4.87e-04];
eB = [ 3.29e-01; -2.34e-02; 3.33e-05; 4.72e-11];
eC = [-4.57e-01; -1.52e-02; 1.73e-03;-1.25e-06; -9.10e-11];
```
As we mentioned in class, the absolute value of these errors should be taken.
5-11-2013: Extra office hours, Wed 6-11-2013, 2:15-3:15 PM.
30-10-2013: Extra office hours, Friday 1-11-2013, 2:15-3:15 PM.
Assignment 2 correction: Bot mid node equation should be x_5 - x_6 + x_9 = 0
21-10-2013: Extra office hours, Wed 23 Oct 2013, 2:15 - 3:15 PM.
Material covered for the midterm: Computer arithmetic; Direct methods for solving square linear systems, including norms and condition numbers, but not including eigenvalues/vectors.
Midterm: Please come on time and sit sparsely, with two empty seats, between any two of you.
21-10-2013: Today, my office hours will shift 15' earlier, i.e. 3:15-4:15 PM.
11-10-2013: Assignment 2 is out. Due date Thu 7-11-2013.
22-9-2013: Assignment 1 is out. Due date Tue 8-10-2013.
You will need the following ``exact'' values for the expression in q3, starting from x = 1e-5 up to x = 1e-10:
```
s = [0.316227766016838e-7;
     0.10000000000002e-8;
     0.31622776600e-10;
     0.999999999e-12;
     0.3162277e-13;
     0.99999e-15];
```
17-9-2013: I have posted a sheet of paper with Taylor's theorem in various forms, as I have seen that some people in the class don't recall these things. You may want to take a look -- see towards the bottom of the page.
See also the log of matlab commands demonstrated in the tutorial.
11-9-2013: Some re-arrangement of lecture/tutorial times: The Thursday 12-9-2013, 1-2 PM slot will be used for a lecture, and the tutorial is postponed for Tuesday 17-9-2013.
10-9-2013: Wallet found in BA024 given to CS undergraduate office.
2-9-2013: General note on assignments: All assignments should be done individually by each student. No collaboration is allowed.
2-9-2013: We will hold a tutorial starting the first week, Thursday 12 September 2013, 1-2 p.m. BA B024.
The tutorial will briefly introduce MATLAB. Students who are quite familiar with MATLAB don't need to attend, but other students may find this tutorial useful.

Bulletin board for CSC350 Fall 2013

Important note on the use of bulletin boards: No parts of or whole answers to the assignment problems should be posted to the boards. Any violation of this rule will bring trouble to the poster.
Please use judgement before posting.

Textbook web page (see educational modules)

Material already covered in the course (with textbook sections in parentheses)

10-9-2013 (2- hrs)
0.   What is Scientific Computing? [1.1, 1.2.1]
1.   Computer arithmetic; data and computational errors
1.1  (Human) Representation of nonnegative integers
   - Algorithm for converting base b integers to decimal
   - Algorithm for converting decimal integers to base b
1.2  (Human) Representation of reals
   - Algorithm for converting base b fractions to decimal
   - Algorithm for converting decimal fractions to base b
1.3  Computer representation of numbers [1.3.1-7]
   - floating-point numbers, mantissa, exponent, normalized mantissa,
     significant digits, overflow, underflow, range of representable numbers,
     representable numbers, chopping, rounding
   - The IEEE Standard
1.4  Round-off error [1.3.5]
1.5  Absolute and relative errors [1.2.2]
1.6  Computer arithmetic [1.3.8]
1.7  Machine epsilon [1.3.5]
1.8  Error propagation in simple arithmetic calculations [1.3.8-9]
   - Multiplication, division, addition/subtraction
   - catastrophic cancellation
12-9-2013 (1 hr)
1.9  Error propagation in computation: conditioning of problems [1.2.6]
   - condition number of function
1.10 Error propagation in computation: stability of algorithms [1.2.7]
1.11 Forward and backward errors [1.2.3-5]
     Propagated data error
     Truncation (discretization) and rounding errors, computational error
     Total error

2.   Direct methods for solving square linear systems
2.1  Vectors and matrices -- review of terminology (start)

17-9-2013 (1 hr)
2.1  Vectors and matrices -- review of terminology
2.2  Solving lower triangular linear systems [2.4.2]
     Forward substitution (f/s)
2.3  Solving upper triangular linear systems [2.4.2]
     Back substitution (b/s)
24-9-2013 (2 hrs)
2.4  Equivalent linear systems - row operations [2.4.1]
2.5  An example of solving a linear system by GE and b/s
2.6  Gaussian elimination (GE) [2.4.3, 2.4.4, 2.4.6-7]
2.7  LU factorization [2.4.3, 2.4.4, 2.4.7]
     elementary Gauss (elimination) transformation matrices
2.8  Symmetric and symmetric positive definite matrices [2.5.1, 2.5.2]
     LDL^T and Choleski factorization
2.9  Banded matrices [2.5.3]
     Banded LU/GE and b/f/s
1-10-2013 (2 hrs)
2.10 Computing the inverse of a matrix [2.4.7]
2.11 GE with partial pivoting [2.4.5-6] (first four slides from notes)
     breakdown or instability of GE, interchanges of rows, columns
     types of pivoting
     elementary Gauss transformation matrices; elementary permutation matrices
8-10-2013 (2 hrs)
2.12 Scaled partial pivoting [2.4.10]
2.13 Complete pivoting
2.14 Iterative refinement [2.4.10]
2.15 Mathematical software [2.7]

2.16 Inner products
2.17 Norms [2.3]
2.18 Vector norms [2.3.1]
2.19 Matrix norms [2.3.2]
2.20 Condition number of a matrix [2.3.3-5]
15-10-2013 (2 hrs)
2.20 Condition number of a matrix [2.3.3-5] -- end
2.21 Function norms
2.22 Computing norms

3.1  Eigenvalues and eigenvectors [4.1-2, 4.4]
     Definition, characteristic equation/polynomial of a matrix,
     multiplicity of eigenvalue, matrix polynomial,
     similarity transformation, diagonalization of matrix,
     Jordan decomposition, Schur decomposition, singular value decomposition,
     symmetric, orthogonal, triangular matrices,
22-10-2013 (2 hrs)
     Gerschgorin theorem, Cayley-Hamilton theorem
3.2  Least squares approximation [3.1]
3.3  Overdetermined linear systems [3.1-2, 3.4.1, 3.7]
     The normal equations for overdetermined systems
3.4  Underdetermined linear systems [3.5.4]
     The normal equations for underdetermined systems
24-10-2013 (1 hr)
midterm test
29-10-2013 (2 hrs)
3.5  Orthogonal transformations [3.4.3]
3.6  Householder reflections (orthogonal transformations) [3.5]
3.7  QR factorization [3.4.3, 3.4.5, 3.5.1, 3.5.4]
     Solving linear systems with the QR factorization
     square, overdetermined, underdetermined
3.8  Gram-Schmidt orthogonalization [3.5.3, 3.5.4]
     Solving linear systems with Gram-Schmidt
     square and overdetermined, underdetermined
3.9  MATLAB and over- or under-determined linear systems [3.8]
5-11-2013 (2 hrs)
3.10 Data fitting, curve fitting [3.1]
3.11 MATLAB, data fitting and least squares problems
3.12 Review of decompositions

3.13 Computing eigenvalues and eigenvectors [4.5.1, 4.5.6]
     The power method
     The QR iteration

4.   Nonlinear equations and systems [5.1, 5.2]
     roots, multiplicity
19-11-2013 (2 hrs)
4.1  Fixed points and roots of functions [5.2]
     systolic or contractive functions
4.2  Existence and uniqueness of root and fixed point [5.2]
4.3  Numerical methods for solving nonlinear equations [5.3, 5.4, 5.5]
     Nonlinear solvers, residual, stopping criteria
4.4  Convergence rate of iterative methods [5.4]
4.5  The bisection method [5.5]
4.6  Fixed-point (functional) iteration methods [5.5]
     Convergence of fixed-point iteration
26-11-2013 (2 hrs)
     Convergence rate of fixed-point iteration
4.7  Newton's method (Newton-Raphson method) [5.5]
4.8  The secant method [5.5]
4.9  Numerical methods for solving nonlinear systems of equations [5.6]
     Newton's method

Material to be covered in the course (with textbook sections in parentheses)

     Broyden's method

5.   Optimization problems [6.1, 6.2.2]
     minimization, maximization,
     constrained, unconstrained, duality, feasible points,
     continuous, discrete, linear programming, nonlinear programming,
     global min, local min, gradient, Hessian, directional derivative,
     descent direction, direction of negative gradient
5.1  1D opt: golden section search [6.4, 6.4.1]
5.2  1D opt: Newton's method [6.4.3]
5.3  multi-D opt: steepest descent [6.5, 6.5.2]
5.4  multi-D opt: Newton's method [6.5.3]
5.5  multi-D opt: BFGS     method [6.5.5]
5.6  multi-D opt: conjugate gradient method [6.5.6]
5.7  General comments

To access these notes use your cdf username, and the last 5 digits of your student id number as password.