CS231n Winter 2016: Lecture 4: Backpropagation, Neural Networks 1
Why Data Science Stack Exchange recommends it
Data Science Stack Exchange users link it when someone is confused about how backpropagation and gradients work through a computation graph.
Recommended by 2 people across 2 different months, 2017 to 2023.
What it is
Lecture 4 of Stanford's CS231n (Winter 2016), taught by Andrej Karpathy, covers backpropagation and the first part of neural networks. It builds up computation graphs, the chain rule and gradient flow for training networks.
Cited in 1 Wikipedia article, including Backpropagation.
What people said
Recommends n1k31t4 · 2018-05 · Data Science Stack Exchange · on “Gradient of a sum”
It would help to draw out the computation graph (or view it using Tensorboard if you are working with Tensorflow/Keras). Have a look at some good examples in the CS231n course website [LINK] . Perhaps watch lecture 4 of the course [this video] .
Recommends WizardOfRobots · 2023-06 · Data Science Stack Exchange · on “Neural Nets: Difference between activation and activation function, error on Wikipedia?”
I think you might have misinterpreted "...as the derivatives (f^l)'(evaluated at z^l)..." to mean that z^l is the derivative. There is no $X$ in the article, so please refrain from using it. a^(l-1) is probably what you want. the total derivative in this case is a matrix, maybe this lecture will help [this video]
Cites tony · 2017-01 · Data Science Stack Exchange · on “Basic backpropagation question”
To make our life easier, let's omit optimization methods here (e.g. Learning Rate, Momentum) and focus on vanilla Backpropagation. As you know, backprop consists of computing the partial derivative with respect to each weight in order to penalize each weight for its role in the global output error; such that: \begin{equation} \frac{\partial E}{\partial W_{ij}} = e_{i}y_{j} \end{equation} With $E$ the global output error to minimize, $W$ the weight to update, $y$ the output value, and $e$ the local error. The local error $e$ is the unknown we want to compute! :) Using the chain rule, the gradient can be expressed as: \begin{equation} \frac{\partial E}{\partial W_{ij}} = \frac{\partial E}{\par