Which Activation Function Should I Use?
Why Stack Overflow recommends it
Stack Overflow users link it when answering beginner questions about why ReLU is preferred over sigmoid or tanh in hidden layers, and askers cite it when trying to understand its claims.
Recommended by 2 people across 2 different months, 2017 to 2019.
What it is
A short explainer by Siraj Raval on what activation functions do in neural networks and how sigmoid, tanh, ReLU and leaky ReLU differ. It walks through the common problems (saturation, vanishing gradients, non-zero-centered outputs) and gives rules of thumb for choosing one, with sample code on GitHub.
What people said
Recommends Peter · 2017-06 · Stack Overflow · on “Neural network backpropagation with RELU”
Yes the orginal Relu function has the problem you describe. So they later made a change to the formula, and called it leaky Relu In essence Leaky Relu tilts the horizontal part of the function slightly by a very small amount. for more information watch this : An explantion of activation methods, and a improved Relu on youtube [this video]
Recommends user9477964 · 2018-10 · Stack Overflow · on “Deep learning activation function between hidden layers?”
The output of sigmoid is not 0 centered. It cannot handle negative values efficiently. Tanh can cause the problem of busting gradients. According to latest research ,one should use ReLU function in the hidden layers of deep neural networks ( or leakyReLU if the vanishing gradient is faced ). With this the output layer must have a softmax for classification or a linear activation for regression. I learned it from this [this video] great video.
Cites BetaDev · 2018-12 · Stack Overflow · on “How can I set Bias and change Sigmoid to ReLU function in ANN?”
Your question is too broad and there are lots of concept behind ReLU vs sigmoid. But in short: Sigmoid Saturate and kill gradients (look at Gradient descent [LINK] ) sigmoid are not zero centered because output of sigmoid is 0<output<1 . I can see for sigmoid you are using scipy but for ReLU its easy. Relu is defined by the following function f(x) = max(0,x) This means if the input is greater then zero return input else return 0. And ReLU is prefered for hidden layers and other like softmax for output layers. I would say, look different activation function and why we need activation functions on neural net. How sigmoid kills gradients and why they slow converge. Q1 = How do I convert Sigmoid
Cites Hazzaldo · 2019-02 · Stack Overflow · on “Which Activation Function to use for Neural Networks”
Apologies in advance if this question is not the conventional approach, where a snippet of code or a question about a code is involved. I'm just trying to understand certain specific points on the subject of Neural Networks. I was watching a YouTube video (by Siraj Raval - School of AI) about choosing the best Activation function for a neural network: [this video] 1- I tried to understand his explanation of why Sigmoid is not an ideal Activation function to be used by Neural Networks anymore for the following reasons: Sigmoids saturate and kill gradients. Sigmoids slow convergence. Sigmoids are not zero-centered. OK to use on last layer. First of all, the 1st and 2nd reasons I'm guessing are simi