=========================================================================== CSC 373H / L0101 Lecture Summary for Week 3 Winter 2005 =========================================================================== Remarks about greedy algorithms. - General form of problem: . Input: set of "candidates" C_1, C_2, ..., C_n, each one with weight (or cost) w(C_i). . Output: subset of candidates S subset of {C_1, C_2, ..., C_n} that satisfies certain constraints and such that total weight of S is maximal (or minimal). - General form of algorithm: sort candidates by weight S := {} for i := 1 to n: if S U {C_i} satisfies constraints: S := S U {C_i} return S - General form of correctness proof: . Partial solution S_i is "promising" if it can be extended to an optimal solution, i.e., if there exists S^opt_i such that S_i subset of S^opt_i and S^opt_i subset of S_i U {C_{i+1}, ..., C_n}. . Prove by induction that S_i is promising for 0 <= i <= n. . In proof, use "exchange lemma": if S_i is promising (with optimal solution S^opt_i) and S_{i+1} = S_i U {C_{i+1}}, then there exists S^opt_{i+1} that extends S_{i+1}. - Not all greedy algorithms fit this pattern exactly, but most are close. Knapsack problems. - General problem: given a set of items, each with a weight w_i and value v_i, together with a fixed maximum capacity C (all numbers are positive integers), find a subset of items of maximal value whose total weight does not exceed the capacity. - Fractional knapsack problem: . Output: Fractions a_1, a_2, ..., a_n in [0,1] (amount of each item to take) such that SUM a_i w_i <= C and SUM a_i v_i is maximal. Greedy algorithms: . Largest weight first doesn't work. Counter-example: C = 100, items = (100,100), (50,10), (50,10), ..., (50,10). . Largest value/weight first works: pick as much of each item as possible to fill up full capacity. Proof: exercise (think of algorithm as choosing fraction of each item in turn, in order of value/weight). - 0-1 knapsack problem: . Cannot break up items, so output becomes subset S of {1,2,...,n} s.t. SUM_{i in S} w_i <= C and SUM_{i in S} v_i is maximal. . If value/weight ratio is constant, then greedy by value not guaranteed to produce optimal answer but gives approx. ratio 2 (knapsack guaranteed to be at least 1/2 full). . If value/weight ratio is not constant, no greedy strategy works. ------------------- Dynamic Programming [Chapter 5] ------------------- Matrix Chain Multiplication. - Reminder: matrix multiplication, associativity, complexity. - Given matrix chain product: A_0 A_1 ... A_{n-1}, many ways to parenthesize (e.g., A(BC) or (AB)C). All will yield same answer but not same running time. Example: A 1x10 B 10x10 C 10x100 (AB)C = 1*10*10 + 1*10*100 = 100 + 1000 = 1100 ops A(BC) = 10*10*100 + 1*10*100 = 10000 + 1000 = 11000 ops - Matrix Chain Multiplication problem: Input: A_0, A_1, ..., A_{n-1} with dimensions [d_0 x d_1], [d_1 x d_2], ..., [d_{n-1} x d_n] Output: Fully parenthesized product with smallest total cost. - Brute force algorithm: How many possible ways to put in parentheses? Answer is called "Catalan number" and is Omega(4^n). - Greedy algorithm: . Product with smallest cost first. Counter-example: 10 1 10 100 greedy: 10 1 10 + 10 10 100 = 10100 other: 1 10 100 + 10 1 100 = 2000 . Product with smallest dimension last, or with largest dimension eliminated first. Counter-example: 1 10 100 1000 greedy: 10 100 1000 + 1 10 1000 = 1,010,000 other: 1 10 100 + 1 100 1000 = 101,000 . Nothing works! - Structure of optimal subproblems: . Idea: instead of trying to find where to put first product, try to find where to put last product. A_0 (A_1 ... A_{n-1}) -- last product costs d_0 d_1 d_n (A_1 A_2) (A_2 ... A_{n-1}) -- last product costs d_0 d_2 d_n ... ... (A_0 ... A_{n-2}) A_{n-1} -- last product costs d_0 d_{n-1} d_n . Greedy: take smallest last product? Counter-example: 1 10 100 1000 . Only n-1 possibilities. What information would help us find best answer? Knowing best cost of doing each subproduct. . Note that best overall product must include optimal subproducts. - Definition of array of subproblem values: . N[i,j] = smallest cost of multiplying A_i ... A_j . From structure of optimal solution, best way of doing A_i ... A_j (including all parentheses) must have the form (A_i ... A_{k-1}) (A_k ... A_j) for some i < k <= j, where each subproduct A_i ... A_{k-1} and A_k ... A_j is done in the best way possible (otherwise wouldn't be best overall). - Array recurrence: From reasoning above, N[i,i] = 0 and for i < j, N[i,j] = min{ d_i d_k d_{j+1} + N[i,k-1] + N[k,j] : i < k <= j } - Basic recursive solution: BestCost(d, i, j): if i = j: return 0 else: // i < j best := oo for k := i+1 to j: try := d[i]*d[k]*d[j+1] + BestCost(d, i, k-1) + BestCost(d, k, j) if try < best: best := try return best Running time? Combinatorial explosion! Just like simple Fibonacci, many subproblems are recomputed over and over giving exponential running time. (For example, trace calls for BestCost(d, 0, 4).) - Bottom-up algorithm: . Instead of recomputing values many times, compute smaller values first and store them in an array to be looked up (so we never need to make recursive calls). In class, I suggested to compute values one diagonal at a time, but other ways possible as long as all entries N[i,k-1] and N[k,j] are present by the time N[i,j] is computed. For example, MatrixChain(d, n): for i := n-1 downto 0: N[i,i] := 0 for j := i+1 to n-1: N[i,j] := oo for k := i+1 to j: try := d[i]*d[k]*d[j+1] + N[i,k-1] + N[k,j] if try < N[i,j]: N[i,j] := try return N[0,n-1] . Trace on example input: 2, 3, 5, 1, 8 snapshots for each value of i (each row filled left-to-right): i = 3: i = 2: 0 1 2 3 0 1 2 3 0 0 1 1 2 2 0 40 3 0 3 0 i = 1: i = 0: 0 1 2 3 0 1 2 3 0 0 0 30 21 37 1 0 15 39 1 0 15 39 2 0 40 2 0 40 3 0 3 0 - Running time: O(n^3) (nested loops iterating O(n) times). =========================================================================== Note: The following material was not covered in lecture (I didn't have time to get to it), but you may find it useful to help you complete Assignment 1 so I include it below. Think of it as a "sneak preview" of what I'll cover during next week's lectures. - Algorithm computes minimum cost but does not give order of multiplications to achieve this cost. - Trick: use a second array B[i,j] to store best value of k used to compute N[i,j]. At the end, B[0,n-1] = index of last multiplication to perform, and we can recursively print each subproduct. MatrixChain(d, n): for i := n-1 downto 0: N[i,i] := 0; B[i,i] := i for j := i+1 to n-1: N[i,j] := oo; B[i,j] := -1 for k := i+1 to j: try := d[i]*d[k]*d[j+1] + N[i,k-1] + N[k,j] if try < N[i,j]: N[i,j] := try; B[i,j] := k parenthesize(B, 0, n-1) This yields the following values in array B for example above: i = 3: i = 2: 0 1 2 3 0 1 2 3 0 0 1 1 2 2 2 3 3 3 3 3 i = 1: i = 0: 0 1 2 3 0 1 2 3 0 0 0 1 1 3 1 1 2 3 1 1 2 3 2 2 3 2 2 3 3 3 3 3 The following subroutine prints the final answer recursively. // print best way to compute A_i...A_j parenthesize(B, i, j): if i = j: print "A_i" else: print "(" parenthesize(B, i, B[i,j]-1) print ") (" parenthesize(B, B[i,j], j) print ")" For the example above, the result would be: ((A_0) ((A_1) (A_2))) (A_3)