Linear Algebra Review (with a Small Dose of Optimization)

1. Linear Algebra Review(with a Small Dose of Optimization) HristoPaskov CS246

2. Outline • Basic definitions • Subspaces and Dimensionality • Matrix functions: inverses and eigenvalue decompositions • Convex optimization

3. Vectors and Matrices • Vector • May also write

4. Vectors and Matrices • Matrix • Written in terms of rows or columns

5. Multiplication • Vector-vector: • Matrix-vector:

6. Multiplication • Matrix-matrix: =

7. Multiplication • Matrix-matrix: • rows of , cols of

8. Multiplication Properties • Associative • Distributive • NOT commutative • Dimensions may not even be conformable

9. Useful Matrices • Identity matrix • Diagonal matrix

10. Useful Matrices • Symmetric • Orthogonal • Columns/ rows are orthonormal • Positive semidefinite: • Equivalently, there exists

11. Outline • Basic definitions • Subspaces and Dimensionality • Matrix functions: inverses and eigenvalue decompositions • Convex optimization

12. Norms • Quantify “size” of a vector • Given , a norm satisfies • Common norms: • Euclidean -norm: • -norm: • -norm:

13. Linear Subspaces

14. Linear Subspaces • Subspace satisfies • If and , then • Vectors span if

15. Linear Independence and Dimension • Vectors are linearlyindependent if • Every linear combination of the is unique • if span and are linearly independent • If span then • If then are NOT linearly independent

16. Linear Independence and Dimension

17. Matrix Subspaces • Matrix defines two subspaces • Column space • Row space • Nullspace of : • Analog for column space

18. Matrix Rank • gives dimensionality of row and column spaces • If has rank , can decompose into product of and matrices =

19. Properties of Rank • For • has fullrank if • If rows not linearly independent • Same for columns if

20. Outline • Basic definitions • Subspaces and Dimensionality • Matrix functions: inverses and eigenvalue decompositions • Convex optimization

21. Matrix Inverse • is invertible iff • Inverse is unique and satisfies • If is invertible then is invertible and

22. Systems of Equations • Given wish to solve • Exists only if • Possibly infinite number of solutions • If is invertible then • Notational device, do notactually invert matrices • Computationally, use solving routines like Gaussian elimination

23. Systems of Equations • What if ? • Find that gives closest to • is projection of onto • Also known as regression • Assume Invertible Projection matrix

24. Systems of Equations

25. Eigenvalue Decomposition • Eigenvalue decomposition of symmetric is • contains eigenvalues of • is orthogonal and contains eigenvectors of • If is not symmetric but diagonalizable • is diagonal by possibly complex • not necessarily orthogonal

26. Characterizations of Eigenvalues • Traditional formulation • Leads to characteristic polynomial • Rayleigh quotient (symmetric )

27. Eigenvalue Properties • For with eigenvalues • When is symmetric • Eigenvalue decomposition is singular value decomposition • Eigenvectors for nonzero eigenvalues give orthogonal basis for

28. Simple Eigenvalue Proof • Why ? • Assume is symmetric and full rank • If , eigenvalue of is 0 • Since is product of eigenvalues, one of the terms is , so product is 0

29. Outline • Basic definitions • Subspaces and Dimensionality • Matrix functions: inverses and eigenvalue decompositions • Convex optimization

30. Convex Optimization • Find minimum of a function subject to solution constraints • Business/economics/ game theory • Resource allocation • Optimal planning and strategies • Statistics and Machine Learning • All forms of regression and classification • Unsupervised learning • Control theory • Keeping planes in the air!

31. Convex Sets • A set is convex if and • Line segment between points in also lies in • Ex • Intersection of halfspaces • balls • Intersection of convex sets

32. Convex Functions • A real-valued function is convex if is convex and and • Graph of upper bounded by line segment between points on graph

33. Gradients • Differentiable convex with • Gradient at gives linear approximation

34. Gradients • Differentiable convex with • Gradient at gives linear approximation

35. Gradient Descent • To minimize move down gradient • But not too far! • Optimum when • Given , learning rate , starting point Do until

36. Stochastic Gradient Descent • Many learning problems have extra structure • Computing gradient requires iterating over all points, can be too costly • Instead, compute gradient at single training example

37. Stochastic Gradient Descent • Given , learning rate , starting point Do until nearly optimal For in random order • Finds nearly optimal

38. Minimize

39. Learning Parameter