CS231n Winter 2016: Lecture 5: Neural Networks Part 2

What people said

Recommends Antonios Sarikas · 2023-06 · Cross Validated Stack Exchange · on “Why ReLU halves the variance in He initialization?”

I am reading the paper Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification [LINK] and I can't understand why (see p.4): $E[x^2_l] = \frac{1}{2}Var[y_{l-1}]$ Let $w_{l-1}$ (weights of layer $l-1$ ) have a symmetric distribution around zero. Let $b_{l-1} = 0$ (the biases of layer $l-1$ ). Then, $$y_{l-1} = w_{l-1}\cdot x_{l-1} + b_{l-1}$$ has zero mean and has a symmetric distribution around zero. What I don't get is why after passing $y_{l-1}$ through ReLU, the output $x_l = \max(0, y_{l-1})$ satisfies the quoted equation? I checked the following Lecture [this video] and at 48:07 Andrej Karpathy says that ReLU halves the variance. Any ideas?

Cites ivanbgd · 2018-02 · Cross Validated Stack Exchange · on “Why is weight initialized as 1/sqrt(# of hidden nodes) in neural networks”

For a shallow network, of only a few layers, standard deviation of the Normal distribution with a mean of 0.0, can be 0.01, or even 1.0. But, that kind of a "fixed" standard deviation is not okay for deep neural networks (NN), that have many layers. If you want to use the Uniform distribution, then it translates to the range of [-0.01, 0.01], or [-1.0, 1.0]. I'll use Normal distribution in this answer. Let's take $\tanh$ as activation function. What you'd get in a deep NN with that initialization, in the case of $\sigma = 0.01$, is the so-called "vanishing gradients" problem. Deep layers would have all activations close to zero, and that's not good for backward pass. Namely, deep layers woul

Cites Molem7b5 · 2021-12 · Cross Validated Stack Exchange · on “What is the difference between architecture selection and hyperparameter tuning in the context of neural networks”

I've already seen the response to this questions here [LINK] , but it doesn't really contain, what I feel, is an answer to my specific query. It seems to unclear in a lot of the literature (that I've seen) the clear distinction between what is considered model/architecture selection, and what is hyperparameter tuning. If I have a simple ConvNet I feel it's well established that things like kernal size, number of filters in a layer, stride etc can all be considered as hyperparameters. But what about things like the number of convolutional layers, the number of neurons in the FC hidden layer, the number of hidden FC layers, the number of convolutional layers that come before a batch_norm layer