24

I'm currently working on building a convolutional neural network (CNN) that will work on financial time series data. The input shape is (100, 40) - 100 time stamps by 40 features.

The CNN that I'm using uses asymmetric kernel sizes (i.e. 1 x 2 and 4 x 1) and also asymmetric strides (i.e. 1 x 2 for the 1 x 2 layers and 1 x 1 for the 4 x 1 layers).

In order to maintain the height dimension to stay 100, I needed to pad the data. In my research, I noticed that people who use TensorFlow or Keras simply use padding='same'; but this option is apparently unavailable in PyTorch.

According to some answers in What is the difference between 'SAME' and 'VALID' padding in tf.nn.max_pool of tensorflow?, and also this answer on the PyTorch discussion forum, I can manually calculate how I need to pad my data, and use torch.nn.ZeroPad2d to solve the problem - since apparently normal torch.nn.Conv2d layers don't support asymmetric padding (I believe that the total padding I need is 3 in height and 0 in width).

I tried this code:

import torch
import torch.nn as nn

conv = nn.Conv2d(1, 1, kernel_size=(4, 1))
pad = nn.ZeroPad2d((0, 0, 2, 1)) # Add 2 to top and 1 to bottom.

x = torch.randint(low=0, high=9, size=(100, 40))
x = x.unsqueeze(0).unsqueeze(0)

y = pad(x)

x.shape # (1, 1, 100, 40)
y.shape # (1, 1, 103, 40)

print(conv(x.float()).shape)
print(conv(y.float()).shape)

# Output
# x -> (1, 1, 97, 40)
# y -> (1, 1, 100, 40)

It does work, in the sense that the data shape remains the same. However, is there really no padding='same' option available? Also, how can we decide which side to pad?

3
  • stackoverflow.com/questions/55140554/… have a look at that, it will give you a clue. Oct 9, 2019 at 15:05
  • 1
    Really speaking, that feature is possible in Tensorflow due to its static computation graph. In PyTorch, there is a dynamic computation graph, so it's probably difficult to implement (otherwise they would have already done that). Within nn.Conv2D, as you say, there is only symmetric padding, but different padding can be done along different dimensions.
    – akshayk07
    Oct 9, 2019 at 15:08
  • 1
    I think @akshayk07 is right and the dynamic nature of pytorch makes it hard; Here is a good implementation of 'same' padding in pytorch (for 2d conv): github.com/rwightman/pytorch-image-models/blob/master/timm/…
    – Separius
    Apr 9, 2020 at 6:13

3 Answers 3

9

I had the same issue some time ago, so I implemented it myself using a ZeroPad2d layer as you are trying to do. Here is the right formula:

from functools import reduce
from operator import __add__

kernel_sizes = (4, 1)

# Internal parameters used to reproduce Tensorflow "Same" padding.
# For some reasons, padding dimensions are reversed wrt kernel sizes,
# first comes width then height in the 2D case.
conv_padding = reduce(__add__, 
    [(k // 2 + (k - 2 * (k // 2)) - 1, k // 2) for k in kernel_sizes[::-1]])

pad = nn.ZeroPad2d(conv_padding)
conv = nn.Conv2d(1, 1, kernel_size=kernel_sizes)

print(x.shape) # (1, 1, 103, 40)
print(conv(y.float()).shape) # (1, 1, 103, 40)

Also, as mentioned by @akshayk07 and @Separius, I can confirm that it is the dynamic nature of pytorch that makes it hard. Here is a post about this point from a Pytorch developper.

8

It looks like there is now, in pytorch 1.9.1, according to the docs.

padding='valid' is the same as no padding. padding='same' pads the input so the output has the shape as the input. However, this mode doesn't support any stride values other than 1.

2
  • what value does padding='same' use? 0? same as last pixel? miror? Jun 6 at 13:56
  • 1
    @OfirShifman see also in the docs: padding_mode (str, optional) – 'zeros', 'reflect', 'replicate' or 'circular'. Default: 'zeros'. That means you can choose but by default it will pad with zeroes.
    – lucidbrot
    Jun 6 at 14:39
1

padding='same' and padding='valid' is possible in Pytorch 1.10.0+. However, 'same' and 'valid' for padding is not possible for when stride > 1.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.