PyTorch Autograd Explained - In-depth Tutorial
Why Stack Overflow recommends it
Stack Overflow users link it when explaining why gradients are or aren't computed, such as detached tensors, non-parameter tensors, and how summed losses are backpropagated through the computation graph.
Recommended by 3 people across 3 different months, 2019 to 2024.
What it is
A 13-minute tutorial by Elliot Waite explaining how PyTorch's autograd system works. It uses worked examples and diagrams to show how operations on tensors with requires_grad=True build a backward graph that is traversed on backward() to accumulate gradients into leaf tensors.
What people said
Recommends Wasi Ahmad · 2019-10 · Stack Overflow · on “pytorch - gradients not calculated for parameters”
Here, c is a regular tensor, not a parameter, so PyTorch does not compute gradients of them. This tutorial ( [this video] ) may help you.
Recommends julius100 · 2021-06 · Stack Overflow · on “Does it make sense to backpropagate a loss calculated from an earlier layer through the entire network?”
Thank you! Your answer together with this video explaining pytorch autograd ( [this video] ) helped me understand it. For those wondering: When doing the sum of loss1 and loss2 pytorch autograd will remember where the elements of the sum came from. When then calling the backward function on the loss it will traverse all of its parts and individually follow it and calculate the gradients. Whats important to remember from this is that when calculating the loss you should always use your frameworks tensor as it does some bookkeeping in the background.
Recommends tgbaggio · 2024-06 · Stack Overflow · on “the output of sigmoid 's requires_grad is False, while the input 's requires_grad is True”
In your post, you mentioned that In this module, scores has requires_grad=True, but after computation, requires_grad for other parameters is False. Even the simplest threshold with requires_grad is also False. I don't understand why this is the case. However, according to my test case, self.threshold.requires_grad=True . Please check the following code: import torch.nn as nn class MoR(nn.Module): def __init__(self, work=False): super().__init__() self.sparsity = 0 self.scores = nn.Parameter(torch.tensor(-4.0)) self.f = torch.sigmoid self.threshold = self.f(self.scores) print(self.scores.requires_grad, self.threshold.requires_grad) MoR() ## True True Therefore, your statement is wrong. Maybe
Cites harshraj22 · 2022-01 · Stack Overflow · on “Some weights of Actor Critic model not updating”
the bug is in act function def act(self, some_input, state): # mu contains info required for gradient mu, var, state_value = self.model(some_input, state) # mu is detached and now has forgot all the operations performed # in self.action_head mu = mu.data.cpu().numpy() sigma = torch.sqrt(var).data.cpu().numpy() action = np.random.normal(mu, sigma) action = np.clip(action, 0, 1) action = torch.from_numpy(action/1000) return action, state_value for the further process, if loss is calculated using tensor operations performed on action , it can not be traced back to update self.action_head weights, as you detached the tensor mu which removes it from the computation graph [this video] and so you do not