As hkchengrex's answer points out, the PyTorch documentation does not explain what rule is used by adaptive pooling layers to determine the size and locations of the pooling kernels. (In fact, there is a fixme in the PyTorch code indicating the documentation needs to be improved.)
However, the calculation of the kernel sizes and locations is implemented by this cpp function and the key logic is actually in the calls to the functions start_index
and end_index
, which define the location and offset of the kernels.
I believe this Python code re-implements that code and shows how kernels are calculated:
from typing import List
import math
def kernels(ind,outd) -> List:
"""Returns a List [(kernel_offset_start,kernel_length)] defining all the pooling kernels for a 1-D adaptive pooling layer that takes an input of dimension `ind` and yields an output of dimension `outd`"""
def start_index(a,b,c):
return math.floor((float(a) * float(c)) / b)
def end_index(a,b,c):
return math.ceil((float(a + 1) * float(c)) / b)
results = []
for ow in range(outd):
start = start_index(ow,outd,ind)
end = end_index(ow,outd,ind)
sz = end - start
results.append((start,sz))
return results
def kernel_indexes(ind,out) -> List:
"""Returns a List [[*ind]] containing the indexes of the pooling kernels"""
startsLengths = kernels(ind,out)
return [list(range(start,start+length)) for (start,length) in startsLengths]
Here are the key points to notice.
First, it matters a lot whether the input dimension (ind
) is an integer multiple of the output dimension (outd
).
Second, when this is the case, then the adaptive layer's kernels are equally-sized and non-overlapping, and are exactly what would be produced by defining kernels and a stride based on the following rule:
stride = ind // outd
kernel_size = ind - (outd-1)*stride
padding = 0
In other words, in this case it is possible to reproduce the effect of an adaptive pooling layer by using a non-adaptive pooling layer defined with suitable stride, kernel_size, and padding. (Example further below.)
Finally, when instead it is the case that the input size is not an integer multiple of the output size, then PyTorch's adaptive pooling rule produces kernels which overlap and are of variable size.
Since the non-adaptive pooling API does not allow for variably-sized kernels, in this case it seems to me there is no way to reproduce the effect of adaptive pooling by feeding suitable values into a non-adaptive pooling layer.
Here's an example which shows both cases. This helper function lets us compare what's happening with adapative average pooling layer and an ordinary average pooling layer which uses fixed stride and kernel:
import torch
import torch.nn as nn
def compare1DAdaptivity(ind,outd,inputpattern):
c = 1
padding = 0
input = torch.Tensor(inputpattern).view(1,c,ind)
stride = ind // outd
kernel_size = (ind - (outd-1)*stride)
avg_pool = nn.AvgPool1d(stride=stride,kernel_size=kernel_size,padding=padding)
avg_out = avg_pool(input)
adap_avg_pool = torch.nn.AdaptiveAvgPool1d(outd)
adap_avg_out = adap_avg_pool(input)
try:
equal_output = torch.allclose(avg_out,adap_avg_out)
except:
equal_output = False
print("input.shape: {}".format(input.shape))
print("in_dims: {}".format(ind))
print("out_dims: {}".format(outd))
print("")
print("AAL strides: {}".format(stride))
print("AAL kernel_sizes: {}".format(kernel_size))
print("AAL pad: {}".format(padding))
print("")
print("outputs equal: {}".format(equal_output))
print("")
print("AAL input -> output: {} -> {}".format(input,avg_out))
print("adap input -> output: {} -> {}".format(input,adap_avg_out))
return equal_output
So, to give an example of the first case, where the input dimension is a multiple of the output dimension, we can go from 6 to 3. We can see that the approximate adaptive layer and the true adaptive layer give the same output:
compare1DAdaptivity(6,3,[1,0,0,0,0]) # => Tue
AAL input -> output: tensor([[[1., 0., 0., 0., 0., 0.]]]) -> tensor([[[0.5000, 0.0000, 0.0000]]])
adap input -> output: tensor([[[1., 0., 0., 0., 0., 0.]]]) -> tensor([[[0.5000, 0.0000, 0.0000]]])
However, this no longer works if we go from 5 to 3.
compare1DAdaptivity(5,3,[1,0,0,0,0]) # => False
AAL input -> output: tensor([[[1., 0., 0., 0., 0.]]]) -> tensor([[[0.3333, 0.0000, 0.0000]]])
adap input -> output: tensor([[[1., 0., 0., 0., 0.]]]) -> tensor([[[0.5000, 0.0000, 0.0000]]])
But we can reproduce the result of the adaptive layers by manually computing over the indexes:
t = [1,0,0,0,0]; [sum( [t[x] for x in xs] ) / len(xs) for xs in kernel_indexes(5,3)]
# => [0.5,0.0,0.0]