CS4413 Numeric Algorithms (These materials are used in the classroom only)

1. CS4413 Numeric Algorithms (These materials are used in the classroom only)

2. Goals • Evaluate polynomials. • More efficient matrix multiplication. • Solve linear equations. • Describe growth rates and order.

3. Why numeric algorithms? • Mathematical calculation forms the basis for a wide range of programs. • Computer Graphics and Computer Vision both require a large number of calculations involving polynomials and matrices. • Because these are typically done for each location in an image, small improvements can have a great impact.

4. Why numeric algorithms? • For example: a typical image can be created with 1024 pixels per row and 1024 pixels per column. • Improving the calculation for each of these locations by even one multiplication would reduce the creation of this image by 1,048,576 multiplications overall. • Some software will repeatedly evaluate complex polynomial equations. • Another application is trigonometric functions such as sine and cosine having power series expansions that take the form of polynomial equations.

5. Calculate polynomials • The standard evaluation algorithm is very straightforward. Evaluate (x) x the value to use for evaluation of the polynomial result = a[0] + a[1] * x xPower = x for i = 2 to n do xPower = xPower * x result = result + a[i] * xPower end for return result

6. Horner’s method • Horner’s method gives a better way to do evaluation without making the process very complex. p(x) = ({…[(anx + an-1) * x + an-2] * x + … + a2} * x + a1) * x + a0 HornersMethod(x) x the value to use for evaluation of the polynomial result = a[n] for i = n – 1 down to 0 do result = result * x result = result + a[i] end for return result

7. Complexity • Standard: 2N – 1 Multiplications and N Additions. • Horner’s:N Multiplications and N Additions.

8. Preprocessed Coefficients • For preprocessed coefficients to work, we need our polynomial to be monic (an = 1) and to have its largest degree equal to 1 less than a power of 2 (n = 2k -1 for some k = 1).

9. Matrix multiplication • A matrix is a mathematical structure of numbers arranged in rows and columns that is equivalent to a two-dimensional array. • Two matrices can be added or subtracted element by element if they are the same size. • Two matrices can be multiplied if the number of columns in the first is equal to the number of rows in the second.

10. Matrix multiplication … • If we multiply a 3 x 4 matrix by a 4 x 7 matrix, we will get a 3 x 7 matrix as our answer. • Matrix multiplication is not commutative. • The standard matrix multiplication algorithm will do a * b * c multiplications and a * (b -1) * c additions for two matrices of size a x b and b x c.

11. Winograd multiplication • If you look at each element of the result of a matrix multiplication, you will see that it is nothing more than the dot product of the corresponding row and column of the original matrices. • Consider two of these vectors: V=(v1,v2,v3,v4) and W=(w1,w2,w3,w4). Their dot product is given by V·W.

12. Winograd Multiplication • V=(v1,v2,v3,v4) • W=(w1,w2,w3,w4) • V·W=v1w1+v2w2+v3w3+v4w4 • V·W=(v1+w2)(v2+w1) + (v3+w4)(v4+w3) -v1v2-v3v4-w1w2-w3w4

13. Generalization Compute Once, Use many times

14. Winograd algorithm • Multiplying G a x b and H b x c to get result R a x c. d = b / 2 //calculate rowFactors for G for i = 1 to a do rowFactor[i] = Gi,1 * Gi,2 for j = 2 to d do rowFactor[i] = rowFactor[i] + Gi,2j-1 * Gi,2j end for j end for i //calculate columnFactors for H for i = 1 to c do columnFactor[i] = H1,i * H2,i

15. Winograd algorithm… for j = 2 to d do columnFactor[i] = columnFactor[i] + H2j-1,i * H2j,i end for j end for i //calculate R for i = 1 to a do for j = 1 to c do Ri,j = -rowFactor[i] – columnFactor[j] for k = 1 to d do Ri,j = Ri,j + (Gi,2k-1 + H2k,j) * (Gi,2k + H2k-1,j) end for k end for j end for i

16. Winograd algorithm… //add in terms for odd shared dimension if (2 * (b/2) ≠ b) then for i = 1 to a do for j = 1 to c do Ri,j = Ri,j + Gi,b * Hb,j end for j end for i end if

17. Analysis • Ordinary Matrix Multiply, n3 • Winograd’s Matrix Multiply, (n3/2)+n2 • Additions, n3-n2, v.s. (3/2)n3+2n2-2n • Lower Bound, Best known is n2 • Best known Upper Bound and Best Known Lower Bound are not the same

18. Two By Two Multiplication • c1,1=a1,1b1,1+a1,2b2,1 • c1,2=a1,1b1,2+a1,2b2,2 • c2,1=a2,1b1,1+a2,2b2,1 • c2,2=a2,1b1,2+a2,2b2,2

19. 2x2 Works for Matrices • C1,1=A1,1B1,1+A1,2B2,1 • C1,2=A1,1B1,2+A1,2B2,2 • C2,1=A2,1B1,1+A2,2B2,1 • C2,2=A2,1B1,2+A2,2B2,2 A1,1 A1,2 B1,1 B1,2 C1,1 C1,2 ´ = A2,1 A2,2 B2,1 B2,2 C2,1 C2,2

20. Divide and Conquer? • Assume matrices of size 2n´2n • Multiplications: M(n) = 8M(n/2), M(1)=8 • Additions: A(n) = 8A(n/2)+n2 • M(n) = 8lg n = nlg 8 = n3 • Additions are also Q(n3), but point is moot • Can we reduce the 8 multiplications in the base equations

21. Strassen’s Equations • x1=(a1,1+a2,2)(b1,1+b2,2) • x2=(a2,1+a2,2) b1,1 • x3=a1,1(b1,2-b2,2) • x4=a2,2(b2,1-b1,1) • x5=(a1,1+a1,2) b2,2 • x6=(a2,1-a1,1)(b1,1+b1,2) • x7=(a1,2-a2,2)(b2,1+b2,2)

22. Using The Equations • c1,1 = x1 + x4 - x5 + x7 • c1,2 = x3 + x5 • c2,1 = x2 + x4 • c2,2 = x1 + x3 - x2 + x6

23. Analysis of Strassen • Only 7 multiplications are used • Will work with matrices, because associative property is not used • Multiplications: M(n) = 7M(n/2), M(1) = 7 • Additions: A(n) = 7A(n/2)+18(n2/4) • M(n) = 7lg n = nlg 7 = n2.81 • Additions are also Q(n2.81) • First algorithm to break the n3 “barrier”

24. Gauss-Jordan Method • Any system of linear equations must satisfy one of the following： (1) has no solution. (2) has an unique solution. (3) has an infinite number of solutions.

25. Elementary Row Operations (1)TypeⅠoperation：A’ is obtained by multiplying any row of A by a nonzero scalar c. e.g.

26. Elementary Row Operations (2)Type Ⅱ operation：For some j≠i, let row i of A’=c(row i of A )+row j of A. And let the other rows of A’ be the same as the row of A. e.g.

27. Elementary Row Operations (3)Type Ⅲ operation：Interchange any tow rows of A. e.g. -2(1)+(2) (2)'

28. Continued (1)-(2)” is the unique solution of such problem. • Note: the above 4 systems of linear equation are equivalent.

29. Example [A∣b] = want

30. Example + + +

31. Example + + +

32. Example

33. Usage of type Ⅲ operations ex [A│b]=

34. Special Cases:No Solution ex one row no solution

35. Special Cases: Infinite Solutions ex Homework: one row infinite solutions