1 / 16

Backpropagation

Backpropagation. Семинар 3. Общая схема обучени я. 1. prepare batch. 2. forward pass. softmax. non-linear activation. 4. update weights. 3. backward pass. Computational graph. x. *. w 0. *-1. exp. +1. +. w 1. Example 1. Sigmoid. The Chain Rule. x. *. w 0. *-1. exp. +1. +.

lburnham
Download Presentation

Backpropagation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Backpropagation Семинар 3

  2. Общая схема обучения 1. prepare batch 2. forward pass softmax non-linear activation ... 4. update weights 3. backward pass

  3. Computational graph x * w0 *-1 exp +1 + w1

  4. Example1. Sigmoid The Chain Rule x * w0 *-1 exp +1 + w1

  5. Example 1. Sigmoid The Chain Rule x * w0 *-1 exp +1 + w1

  6. Staged computation x * w0 + w1

  7. Gradient checking • Проверка корректности производной: • Если – вектор:

  8. Patterns x * + *3 y max z

  9. Patterns x * + *3 y max z

  10. Example 2. Матрицы ? - +

  11. Example 2. Матрицы • Если Z = XY,  то зная dZ имеем: • далее dx = df / dx • dY= XTdZ • dX = dZYT • Dimensionanalysis • X и dX должны иметь одинаковую размерность • Неплохое объяснение • http://cs231n.stanford.edu/vecDerivs.pdf

  12. Example 3. Simple NN + + L ? ?

  13. Example 4. Softmax

  14. Example 4. Softmax

  15. На практике backpropдля softmaxи cross-entropy loss обычно считают вместе • В таком случае получается очень простая формула для градиента

  16. Что посмотреть? • Course notes стэнфордского курса по данной теме: • http://cs231n.github.io/optimization-2/ • http://cs231n.github.io/neural-networks-3/ • Классная видео-визуализация с объяснениями:https://www.youtube.com/watch?v=Ilg3gGewQ5U • Еще одна неплохая визуализация:https://google-developers.appspot.com/machine-learning/crash-course/backprop-scroll/

More Related