1 / 139

Prelude

Prelude. A pattern of activation in a NN is a vector A set of connection weights between units is a matrix Vectors and matrices have well-understood mathematical and geometric properties Very useful for understanding the properties of NNs. Operations on Vectors and Matrices. Outline.

amina
Download Presentation

Prelude

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Prelude • A pattern of activation in a NN is a vector • A set of connection weights between units is a matrix • Vectors and matrices have well-understood mathematical and geometric properties • Very useful for understanding the properties of NNs

  2. Operations on Vectors and Matrices

  3. Outline • The Players: Scalars, Vectors and Matrices • Vectors, matrices and neural nets • Geometric Analysis of Vectors • Multiplying Vectors by Scalars • Multiplying Vectors by Vectors • The inner product (produces a scalar) • The outer product (produces a matrix) • Multiplying Vectors by Matrices • Multiplying Matrices by Matrices

  4. Scalars, Vectors and Matrices • Scalar: A single number (integer or real) • Vector: An ordered list of scalars [ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ]

  5. Scalars, Vectors and Matrices • Scalar: A single number (integer or real) • Vector: An ordered list of scalars [ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ] [ 12 10 ] ≠ [ 10 12 ]

  6. Scalars, Vectors and Matrices • Scalar: A single number (integer or real) • Vector: An ordered list of scalars [ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ] Row vectors

  7. 1 2 3 4 5 1.5 0.3 6.2 12.0 17.1 Scalars, Vectors and Matrices • Scalar: A single number (integer or real) • Vector: An ordered list of scalars [ 1 2 3 4 5 ] [ 0.4 1.2 0.07 8.4 12.3 ] [ 12 10 ] [ 2 ] Row vectors Column Vectors

  8. Scalars, Vectors and Matrices Scalar: A single number (integer or real) Vector: An ordered list of scalars Matrix: An ordered list of vectors: 1 2 6 1 7 8 2 5 9 0 0 3 3 1 5 7 6 3 2 7 9 3 3 1

  9. Scalars, Vectors and Matrices Scalar: A single number (integer or real) Vector: An ordered list of scalars Matrix: An ordered list of vectors: 1 2 6 1 7 8 2 5 9 0 0 3 3 1 5 7 6 3 2 7 9 3 3 1 Row vectors

  10. Scalars, Vectors and Matrices Scalar: A single number (integer or real) Vector: An ordered list of scalars Matrix: An ordered list of vectors: 1 2 6 1 7 8 2 5 9 0 0 3 3 1 5 7 6 3 2 7 9 3 3 1 Column vectors

  11. Scalars, Vectors and Matrices Scalar: A single number (integer or real) Vector: An ordered list of scalars Matrix: An ordered list of vectors: 1 2 6 1 7 8 2 5 9 0 0 3 3 1 5 7 6 3 2 7 9 3 3 1 Matrices are indexed (row, column) M =

  12. Scalars, Vectors and Matrices Scalar: A single number (integer or real) Vector: An ordered list of scalars Matrix: An ordered list of vectors: 1 2 6 1 7 8 2 5 9 0 0 3 3 1 5 7 6 3 2 7 9 3 3 1 Matrices are indexed (row, column) M(1,3) = 6 (row 1, column 3) M =

  13. Scalars, Vectors and Matrices Scalar: A single number (integer or real) Vector: An ordered list of scalars Matrix: An ordered list of vectors: 1 2 6 1 7 8 2 5 9 0 0 3 3 1 5 7 6 3 2 7 9 3 3 1 Matrices are indexed (row, column) M(1,3) = 6 (row 1, column 3) M(3,1) = 3 (row 3, column 1) M =

  14. Variable Naming Conventions • Scalars: Lowercase, italics x, y, z… • Vectors: Lowercase, bold u, v, w… • Matrices: Uppercase, bold M, N, O … • Constants: Greek , , , ,  …

  15. Transposing Vectors If u is a row vector… u = [ 1 2 3 4 5 ] …then u’ (“u-transpose”) is a column vector 1 2 3 4 5 … and vice-versa. u’ =

  16. Transposing Vectors If u is a row vector… u = [ 1 2 3 4 5 ] …then u’ (“u-transpose”) is a column vector 1 2 3 4 5 … and vice-versa. Why in the world would I care?? u’ = You

  17. Transposing Vectors If u is a row vector… u = [ 1 2 3 4 5 ] …then u’ (“u-transpose”) is a column vector 1 2 3 4 5 … and vice-versa. Answer: It’ll matter when we come to vector multiplication. u’ =

  18. Transposing Vectors If u is a row vector… u = [ 1 2 3 4 5 ] …then u’ (“u-transpose”) is a column vector 1 2 3 4 5 … and vice-versa. OK. u’ =

  19. Vectors, Matrices & Neural Nets

  20. Vectors, Matrices & Neural Nets j1 j2 j3 Input units, j

  21. Vectors, Matrices & Neural Nets i1 i2 Output units, i j1 j2 j3 Input units, j

  22. Vectors, Matrices & Neural Nets i1 i2 Output units, i Connection weights, wij w23 w11 w13 w12 w22 w21 j1 j2 j3 Input units, j

  23. Vectors, Matrices & Neural Nets i1 i2 Output units, i Connection weights, wij w23 w11 w13 w12 w22 w21 0.2 0.9 0.5 Input units, j The activations of the input nodes can be represented as a 3-dimensional vector: j = [ 0.2 0.9 0.5 ]

  24. Vectors, Matrices & Neural Nets 1.0 0.0 Output units, i Connection weights, wij w23 w11 w13 w12 w22 w21 j1 j2 j3 Input units, j The activations of the output nodes can be represented as a 2-dimensional vector: i = [ 1.0 0.0 ]

  25. Vectors, Matrices & Neural Nets i1 i2 Output units, i Connection weights, wij w23 w11 0.1 0.2 w13 w12 1.0 w22 w21 j1 j2 j3 Input units, j The weights leading into any output node can be represented as a 3-dimensional vector: w1j = [ 0.1 1.0 0.2 ]

  26. Vectors, Matrices & Neural Nets i1 i2 Output units, i Connection weights, wij w23 -0.9 w11 0.1 0.2 w13 w12 1.0 w22 0.1 w21 1.0 j1 j2 j3 Input units, j The complete set of weights can be represented as a 3 (row) X 2 (column) matrix: 0.1 1.0 0.21.0 0.1 -0.9 W =

  27. Vectors, Matrices & Neural Nets i1 i2 Output units, i Connection weights, wij Why in the world would I care?? w23 -0.9 w11 0.1 0.2 w13 w12 1.0 w22 0.1 w21 1.0 j1 j2 j3 Input units, j The complete set of weights can be represented as a 2 (row) X 3 (column) matrix: 0.1 1.0 0.21.0 0.1 -0.9 W =

  28. Vectors, Matrices & Neural Nets Because the mathematics of vectors and matrices is well-understood. Because vectors have a very useful geometric interpretation. Because Matlab “thinks” in vectors and matrices. Because you are going to have to learn to think in Matlab. Why in the world would I care?? W

  29. Vectors, Matrices & Neural Nets Because the mathematics of vectors and matrices is well-understood. Because vectors have a very useful geometric interpretation. Because Matlab “thinks” in vectors and matrices. Because you are going to have to learn to think in Matlab. OK.

  30. Geometric Analysis of Vectors Dimensionality: The number of numbers in a vector

  31. Geometric Analysis of Vectors Dimensionality: The number of numbers in a vector

  32. Geometric Analysis of Vectors Dimensionality: The number of numbers in a vector

  33. Geometric Analysis of Vectors • Implications for neural networks • Auto-associative nets • State of activation at time t is a vector (a point in a space) • As activations change, vector moves through that space • Will prove invaluable in understanding Hopfield nets • Layered nets (“perceptrons”) • Input vectors activate output vectors • Points in input space map to points in output space • Will prove invaluable in understanding perceptrons and back-propagation learning

  34. Multiplying a Vector by a Scalar 4 5 [ 5 4 ] * 2 =

  35. Multiplying a Vector by a Scalar 8 4 10 5 [ 5 4 ] * 2 = [ 10 8 ] Lengthens the vector but does not change its orientation

  36. Adding a Vector to a Scalar 4 5 [ 5 4 ] + 2 =

  37. Adding a Vector to a Scalar 4 5 [ 5 4 ] + 2 = NAN Is Illegal.

  38. Adding a Vector to a Vector 6 4 5 3 [ 5 4 ] + [ 3 6 ] =

  39. Adding a Vector to a Vector 10 6 4 5 3 8 [ 5 4 ] + [ 3 6 ] = [ 8 10 ] Forms a parallelogram.

  40. Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”) If u and v are both row vectors of the same dimensionality… u = [ 1 2 3 ] v = [ 4 5 6 ]

  41. Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”) If u and v are both row vectors of the same dimensionality… u = [ 1 2 3 ] v = [ 4 5 6 ] … then the product u ·v =

  42. Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”) If u and v are both row vectors of the same dimensionality… u = [ 1 2 3 ] v = [ 4 5 6 ] … then the product u ·v = NAN Is undefined.

  43. Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”) If u and v are both row vectors of the same dimensionality… u = [ 1 2 3 ] v = [ 4 5 6 ] … then the product u ·v = NAN Is undefined. Huh?? Why?? That’s BS!

  44. Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”) I told you you’d eventually care about transposing vectors… ?

  45. Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”) • The Mantra: “Rows by Columns” • Multiply rows (or row vectors) by columns (or column vectors)

  46. Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”) • The Mantra: “Rows by Columns” • Multiply rows (or row vectors) by columns (or column vectors) u = [ 1 2 3 ] v = [ 4 5 6 ]

  47. Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”) • The Mantra: “Rows by Columns” • Multiply rows (or row vectors) by columns (or column vectors) 4 5 6 u = [ 1 2 3 ] v = [ 4 5 6 ] v’ =

  48. Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”) • The Mantra: “Rows by Columns” • Multiply rows (or row vectors) by columns (or column vectors) 4 5 6 u = [ 1 2 3 ] v’ = u ·v’ = 32

  49. Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”) • The Mantra: “Rows by Columns” • Multiply rows (or row vectors) by columns (or column vectors) v’ Imagine rotating your row vector into a (pseudo) column vector… 4 5 6 1 2 3 u = [ 1 2 3 ] u ·v’ = 32

  50. Multiplying a Vector by a Vector 1:The Inner Product (aka “Dot Product”) • The Mantra: “Rows by Columns” • Multiply rows (or row vectors) by columns (or column vectors) v’ Now multiply corresponding elements and add up the products… 4 5 6 1 2 3 4 u = [ 1 2 3 ] u ·v’ = 32

More Related