Lecture 6 | Training Neural Networks I

What people said

Recommends user9947 · 2019-02 · Artificial Intelligence Stack Exchange · on “Does leaky relu help learning if the final output needs negative values?”

The answer depends on a case to case basis. It may so happen that a dataset performs very well on ReLu but takes more number of iterations to converge on leaky ReLu's or PReLu's or vice-versa. There are 2 arguments to consider here: ReLu is the most non-linear among all other type of ReLu's, and by this not so mathematical term I mean to say that it has the steepest drop in slope at 0, as compared to any other type of modified ReLu's. ReLu's omit negative values which can be a significant problem with context to data normalisation. As this video [this video] (~10:00) from Stanford explains how data normalisation is necessary in context of signs of weight updates, so we can very roughly say any fo

Recommends karthikeyan mg · 2019-03 · Data Science Stack Exchange · on “Need equations for some of weight initializers in tensorflow?”

[LINK] 2 - Not necessarily between 0 an 1. Example: If you see the normal glorot formulae, The sigma_i value will entirely depend on fan_in and fan_out. So, it need not be centred at 0. EDIT 1: [LINK] EDIT 2: [LINK] [LINK] Research paper [LINK] - weight initialization by Xavier Credits: Also, check out this [this video] and this [LINK] Hope it helps.

Recommends amin · 2020-10 · Artificial Intelligence Stack Exchange · on “Smallest possible network to approximate the $sin$ function”

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh network had to start from zero). Generally I would say it's impossible to have a good approximation of sinus with just 3 neurons, but if you want to consider one period of sinus, then you can do something. for clarity look at this picture: [LINK] I've write the code for this task in colab and you can find it here [LINK] , and you can play with it if you want. If you run the network several times you may get different results (because of different initializations) and you can see some of them at the Results section of the link above.

Cites user9947 · 2018-06 · Artificial Intelligence Stack Exchange · on “Why Feature Scaling for skewed contour?”

Feature scaling is making your features have an equal representation or say in the final loss of the function. Intuitively as your picture suggests the contour will be elongated along one axes (feature with higher values). Another example is lets say you have to separate two data clusters, one is (x,2) and another is (x,5) where x is variable. Now if you are using sigmoid activation, sigmoid always gives positive values, but to separate the two classes you need to have negative value for (x,2) in the second last layer, thus the NN will for a longer time have to adjust the weights to be negative (assuming positive weight initialisation) of bias and other features until (x,2) gives negative va

Cites DuttaA · 2018-08 · Data Science Stack Exchange · on “Understanding data normalisation”

Normalisation is a very blurry concept. It is quite misunderstood most of the times. I will take the specific case of Neural Nets and explain it: Purpose: Normalisation is done so that the Neural Network weights converge faster. In CNN's and Deep Neural Nets this is of particular help especially in CNN this helps to prevent exploding/vanishing gradients. The most common explanation for normalisation I have come across is that if you have 2 features, one of them has a significant larger scale than the other e.g house price and house area then the feature with a larger scale will dominate the output. This is quite incorrect according to me, since when you back-propagate through the Neural Net

Cites Churchjm · 2020-10 · Data Science Stack Exchange · on “Why the sigmoid activation function results in sub-optimal gradient descent?”

I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford [this video] . She says that because the output of sigmoid is always positive, that any gradients flowing back from a neuron following a sigmoid will all share the same sign as the upstream gradient flowing into that neuron. She then says that a consequence of these weight updates sharing the same sign is a sub-optimal zigzag gradient descent path. I understand this phenomenon when zoomed in on a single neuron. However, since upstream gradients flowing into a layer can be of different signs, it's still possible to get a healthy mixture of positive and negative weight u

Cites Daviiid · 2021-03 · Artificial Intelligence Stack Exchange · on “Why is it a problem if the outputs of an activation function are not zero-centered?”

In this lecture [this video] , the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that the gradient of our loss w.r.t. the weights $\frac{\partial L}{\partial w}$ which is equal to $\frac{\partial L}{\partial \sigma}\frac{\partial \sigma}{\partial w}$ will always be either negative or positive and we'll have a problem updating our weights as she shows in this slide [this video] , we won't be able to move in the direction of the vector $(1,-1)$ . I don't understand why since she only talks about one component of our gradient and not the whole vector. if the components of the g