How to initialize the weights and biases (for example, with He or Xavier initialization) in a network in PyTorch?
Single layer
To initialize the weights of a single layer, use a function from torch.nn.init
. For instance:
conv1 = torch.nn.Conv2d(...)
torch.nn.init.xavier_uniform(conv1.weight)
Alternatively, you can modify the parameters by writing to conv1.weight.data
(which is a torch.Tensor
). Example:
conv1.weight.data.fill_(0.01)
The same applies for biases:
conv1.bias.data.fill_(0.01)
nn.Sequential
or custom nn.Module
Pass an initialization function to torch.nn.Module.apply
. It will initialize the weights in the entire nn.Module
recursively.
apply(fn): Applies
fn
recursively to every submodule (as returned by.children()
) as well as self. Typical use includes initializing the parameters of a model (see also torch-nn-init).
Example:
def init_weights(m):
if type(m) == nn.Linear:
torch.nn.init.xavier_uniform(m.weight)
m.bias.data.fill_(0.01)
net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))
net.apply(init_weights)
-
4I found a
reset_parameters
method in the source code of many modules. Should I override the method for weight initialization? – Yang Bo Jun 26 '18 at 6:02 -
1what if I want to use a Normal distribution with some mean and std? – Charlie Parker Jul 4 '18 at 21:16
import torch.nn as nn
# a simple network
rand_net = nn.Sequential(nn.Linear(in_features, h_size),
nn.BatchNorm1d(h_size),
nn.ReLU(),
nn.Linear(h_size, h_size),
nn.BatchNorm1d(h_size),
nn.ReLU(),
nn.Linear(h_size, 1),
nn.ReLU())
# initialization function, first checks the module type,
# then applies the desired changes to the weights
def init_normal(m):
if type(m) == nn.Linear:
nn.init.uniform_(m.weight)
# use the modules apply function to recursively apply the initialization
rand_net.apply(init_normal)
Sorry for being so late, I hope my answer will help.
To initialise weights with a normal distribution
use:
torch.nn.init.normal_(tensor, mean=0, std=1)
Or to use a constant distribution
write:
torch.nn.init.constant_(tensor, value)
Or to use an uniform distribution
write:
torch.nn.init.uniform_(tensor, a=0, b=1) # a: lower_bound, b: upper_bound
You can check other methods of initialising tensors here