As @Jan said here, you can overload nn.Linear
and provide a point-wise mask to mask the interaction you want to avoid having. Remember that a fully connected layer is merely a matrix multiplication with an optional additive bias.
Looking at its source code, we can do:
class MaskedLinear(nn.Linear):
def __init__(self, *args, mask, **kwargs):
super().__init__(*args, **kwargs)
self.mask = mask
def forward(self, input):
return F.linear(input, self.weight, self.bias)*self.mask
Having F
defined as torch.nn.functional
Considering the constraint you have given to the second layer:
the first two neurons of the hidden layer should be connected to the first neuron of the output layer
It seems you are looking for this pattern:
tensor([[1., 0., 0.],
[1., 0., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 0., 1.],
[0., 0., 1.]])
Which can be obtained using torch.block_diag
:
mask = torch.block_diag(*[torch.ones(2,1),]*output_size)
Having this, you can define your network as:
net = nn.Sequential(nn.Linear(input_size, hidden_size),
MaskedLinear(hidden_size, output_size, mask))
If you feel like it, you can even implement it inside the custom layer:
class LocalLinear(nn.Linear):
def __init__(self, *args, kernel_size=2, **kwargs):
super().__init__(*args, **kwargs)
assert self.in_features == kernel_size*self.out_features
self.mask = torch.block_diag(*[torch.ones(kernel_size,1),]*self.out_features)
def forward(self, input):
return F.linear(input, self.weight, self.bias)*self.mask
And defining it like so:
net = nn.Sequential(nn.Linear(input_size, hidden_size),
LocalLinear(hidden_size, output_size))