一、禁止計(jì)算局部梯度
torch.autogard.no_grad: 禁用梯度計(jì)算的上下文管理器。
當(dāng)確定不會(huì)調(diào)用Tensor.backward()計(jì)算梯度時(shí)，設(shè)置禁止計(jì)算梯度會(huì)減少內(nèi)存消耗。如果需要計(jì)算梯度設(shè)置Tensor.requires_grad=True
兩種禁用方法：
將不用計(jì)算梯度的變量放在with torch.no_grad()里
>>> x = torch.tensor([1.], requires_grad=True)
>>> with torch.no_grad():
... y = x * 2
>>> y.requires_grad
Out[12]:False
使用裝飾器 @torch.no_gard()修飾的函數(shù)，在調(diào)用時(shí)不允許計(jì)算梯度
>>> @torch.no_grad()
... def doubler(x):
... return x * 2
>>> z = doubler(x)
>>> z.requires_grad
Out[13]:False
二、禁止后允許計(jì)算局部梯度
torch.autogard.enable_grad ：允許計(jì)算梯度的上下文管理器
在一個(gè)no_grad上下文中使能梯度計(jì)算。在no_grad外部此上下文管理器無影響.
用法和上面類似：
使用with torch.enable_grad()允許計(jì)算梯度
>>> x = torch.tensor([1.], requires_grad=True)
>>> with torch.no_grad():
... with torch.enable_grad():
... y = x * 2
>>> y.requires_grad
Out[14]:True
>>> y.backward() # 計(jì)算梯度
>>> x.grad
Out[15]: tensor([2.])
在禁止計(jì)算梯度下調(diào)用被允許計(jì)算梯度的函數(shù)，結(jié)果可以計(jì)算梯度
>>> @torch.enable_grad()
... def doubler(x):
... return x * 2
>>> with torch.no_grad():
... z = doubler(x)
>>> z.requires_grad
Out[16]:True
三、是否計(jì)算梯度
torch.autograd.set_grad_enable()
可以作為一個(gè)函數(shù)使用：
>>> x = torch.tensor([1.], requires_grad=True)
>>> is_train = False
>>> with torch.set_grad_enabled(is_train):
... y = x * 2
>>> y.requires_grad
Out[17]:False
>>> torch.set_grad_enabled(True)
>>> y = x * 2
>>> y.requires_grad
Out[18]:True
>>> torch.set_grad_enabled(False)
>>> y = x * 2
>>> y.requires_grad
Out[19]:False
總結(jié)：
單獨(dú)使用這三個(gè)函數(shù)時(shí)沒有什么，但是若是嵌套，遵循就近原則。
x = torch.tensor([1.], requires_grad=True)
with torch.enable_grad():
torch.set_grad_enabled(False)
y = x * 2
print(y.requires_grad)
Out[20]: False
torch.set_grad_enabled(True)
with torch.no_grad():
z = x * 2
print(z.requires_grad)
Out[21]:False
補(bǔ)充：pytorch局部范圍內(nèi)禁用梯度計(jì)算，no_grad、enable_grad、set_grad_enabled使用舉例
原文及翻譯
Locally disabling gradient computation
在局部區(qū)域內(nèi)關(guān)閉(禁用)梯度的計(jì)算.
The context managers torch.no_grad(), torch.enable_grad(),
and torch.set_grad_enabled() are helpful for locally disabling
and enabling gradient computation. See Locally disabling gradient
computation for more details on their usage. These context
managers are thread local, so they won't work if you send
work to another thread using the threading module, etc.
上下文管理器torch.no_grad()、torch.enable_grad()和
torch.set_grad_enabled()可以用來在局部范圍內(nèi)啟用或禁用梯度計(jì)算.
在Locally disabling gradient computation章節(jié)中詳細(xì)介紹了
局部禁用梯度計(jì)算的使用方式.這些上下文管理器具有線程局部性,
因此,如果你使用threading模塊來將工作負(fù)載發(fā)送到另一個(gè)線程,
這些上下文管理器將不會(huì)起作用.
no_grad Context-manager that disabled gradient calculation.
no_grad 用于禁用梯度計(jì)算的上下文管理器.
enable_grad Context-manager that enables gradient calculation.
enable_grad 用于啟用梯度計(jì)算的上下文管理器.
set_grad_enabled Context-manager that sets gradient calculation to on or off.
set_grad_enabled 用于設(shè)置梯度計(jì)算打開或關(guān)閉狀態(tài)的上下文管理器.
例子1
Microsoft Windows [版本 10.0.18363.1440]
(c) 2019 Microsoft Corporation。保留所有權(quán)利。
C:\Users\chenxuqi>conda activate pytorch_1.7.1_cu102
(pytorch_1.7.1_cu102) C:\Users\chenxuqi>python
Python 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.manual_seed(seed=20200910)
torch._C.Generator object at 0x000001A2E55A8870>
>>> a = torch.randn(3,4,requires_grad=True)
>>> a
tensor([[ 0.2824, -0.3715, 0.9088, -1.7601],
[-0.1806, 2.0937, 1.0406, -1.7651],
[ 1.1216, 0.8440, 0.1783, 0.6859]], requires_grad=True)
>>> b = a * 2
>>> b
tensor([[ 0.5648, -0.7430, 1.8176, -3.5202],
[-0.3612, 4.1874, 2.0812, -3.5303],
[ 2.2433, 1.6879, 0.3567, 1.3718]], grad_fn=MulBackward0>)
>>> b.requires_grad
True
>>> b.grad
__main__:1: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
>>> print(b.grad)
None
>>> a.requires_grad
True
>>> a.grad
>>> print(a.grad)
None
>>>
>>> with torch.no_grad():
... c = a * 2
...
>>> c
tensor([[ 0.5648, -0.7430, 1.8176, -3.5202],
[-0.3612, 4.1874, 2.0812, -3.5303],
[ 2.2433, 1.6879, 0.3567, 1.3718]])
>>> c.requires_grad
False
>>> print(c.grad)
None
>>> a.grad
>>>
>>> print(a.grad)
None
>>> c.sum()
tensor(6.1559)
>>>
>>> c.sum().backward()
Traceback (most recent call last):
File "stdin>", line 1, in module>
File "D:\Anaconda3\envs\pytorch_1.7.1_cu102\lib\site-packages\torch\tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "D:\Anaconda3\envs\pytorch_1.7.1_cu102\lib\site-packages\torch\autograd\__init__.py", line 132, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
>>>
>>>
>>> b.sum()
tensor(6.1559, grad_fn=SumBackward0>)
>>> b.sum().backward()
>>>
>>>
>>> a.grad
tensor([[2., 2., 2., 2.],
[2., 2., 2., 2.],
[2., 2., 2., 2.]])
>>> a.requires_grad
True
>>>
>>>
例子2
Microsoft Windows [版本 10.0.18363.1440]
(c) 2019 Microsoft Corporation。保留所有權(quán)利。
C:\Users\chenxuqi>conda activate pytorch_1.7.1_cu102
(pytorch_1.7.1_cu102) C:\Users\chenxuqi>python
Python 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.manual_seed(seed=20200910)
torch._C.Generator object at 0x000002109ABC8870>
>>>
>>> a = torch.randn(3,4,requires_grad=True)
>>> a
tensor([[ 0.2824, -0.3715, 0.9088, -1.7601],
[-0.1806, 2.0937, 1.0406, -1.7651],
[ 1.1216, 0.8440, 0.1783, 0.6859]], requires_grad=True)
>>> a.requires_grad
True
>>>
>>> with torch.set_grad_enabled(False):
... b = a * 2
...
>>> b
tensor([[ 0.5648, -0.7430, 1.8176, -3.5202],
[-0.3612, 4.1874, 2.0812, -3.5303],
[ 2.2433, 1.6879, 0.3567, 1.3718]])
>>> b.requires_grad
False
>>>
>>> with torch.set_grad_enabled(True):
... c = a * 3
...
>>> c
tensor([[ 0.8472, -1.1145, 2.7263, -5.2804],
[-0.5418, 6.2810, 3.1219, -5.2954],
[ 3.3649, 2.5319, 0.5350, 2.0576]], grad_fn=MulBackward0>)
>>> c.requires_grad
True
>>>
>>> d = a * 4
>>> d.requires_grad
True
>>>
>>> torch.set_grad_enabled(True) # this can also be used as a function
torch.autograd.grad_mode.set_grad_enabled object at 0x00000210983982C8>
>>>
>>> # 以函數(shù)調(diào)用的方式來使用
>>>
>>> e = a * 5
>>> e
tensor([[ 1.4119, -1.8574, 4.5439, -8.8006],
[-0.9030, 10.4684, 5.2031, -8.8257],
[ 5.6082, 4.2198, 0.8917, 3.4294]], grad_fn=MulBackward0>)
>>> e.requires_grad
True
>>>
>>> d
tensor([[ 1.1296, -1.4859, 3.6351, -7.0405],
[-0.7224, 8.3747, 4.1625, -7.0606],
[ 4.4866, 3.3759, 0.7133, 2.7435]], grad_fn=MulBackward0>)
>>>
>>> torch.set_grad_enabled(False) # 以函數(shù)調(diào)用的方式來使用
torch.autograd.grad_mode.set_grad_enabled object at 0x0000021098394C48>
>>>
>>> f = a * 6
>>> f
tensor([[ 1.6943, -2.2289, 5.4527, -10.5607],
[ -1.0836, 12.5621, 6.2437, -10.5908],
[ 6.7298, 5.0638, 1.0700, 4.1153]])
>>> f.requires_grad
False
>>>
>>>
>>>
以上為個(gè)人經(jīng)驗(yàn)，希望能給大家一個(gè)參考，也希望大家多多支持腳本之家。如有錯(cuò)誤或未考慮完全的地方，望不吝賜教。
您可能感興趣的文章:- Pytorch實(shí)現(xiàn)將模型的所有參數(shù)的梯度清0
- Pytorch中的自動(dòng)求梯度機(jī)制和Variable類實(shí)例
- 在pytorch中實(shí)現(xiàn)只讓指定變量向后傳播梯度
- pytorch對(duì)梯度進(jìn)行可視化進(jìn)行梯度檢查教程
- pytorch梯度剪裁方式
- PyTorch的SoftMax交叉熵?fù)p失和梯度用法
- 在pytorch中對(duì)非葉節(jié)點(diǎn)的變量計(jì)算梯度實(shí)例
- pytorch損失反向傳播后梯度為none的問題
