9

I built several masks through a network. These masks are stored in a torch.tensor variable. I would like to do a cv2.dilate like operation on every channel of the tensor.

I know there is a way that convert the tensor to numpy.ndarray and then apply cv2.dilate to every channel using a for loop. But since there are about 32 channels, this method might slow down the forward operation in the network.

4 Answers 4

18

I think dilate is essentially conv2d operation in torch. See the code below

import cv2
import numpy as np
import torch

im = np.array([ [0, 0, 0, 0, 0],
                [0, 1, 0, 0, 0],
                [0, 1, 1, 0, 0],
                [0, 0, 0, 1, 0],
                [0, 0, 0, 0, 0] ], dtype=np.float32)
kernel = np.array([ [1, 1, 1],
                    [1, 1, 1],
                    [1, 1, 1] ], dtype=np.float32)
print(cv2.dilate(im, kernel))
# [[1. 1. 1. 0. 0.]
#  [1. 1. 1. 1. 0.]
#  [1. 1. 1. 1. 1.]
#  [1. 1. 1. 1. 1.]
#  [0. 0. 1. 1. 1.]]
im_tensor = torch.Tensor(np.expand_dims(np.expand_dims(im, 0), 0)) # size:(1, 1, 5, 5)
kernel_tensor = torch.Tensor(np.expand_dims(np.expand_dims(kernel, 0), 0)) # size: (1, 1, 3, 3)
torch_result = torch.clamp(torch.nn.functional.conv2d(im_tensor, kernel_tensor, padding=(1, 1)), 0, 1)
print(torch_result)
# tensor([[[[1., 1., 1., 0., 0.],
#           [1., 1., 1., 1., 0.],
#           [1., 1., 1., 1., 1.],
#           [1., 1., 1., 1., 1.],
#           [0., 0., 1., 1., 1.]]]])
4
  • 3
    I insist in the fact that dilate and conv2d are not the same operation!
    – Manza
    May 9, 2021 at 9:15
  • Yep, Check out Manza's answer, guys!
    – Ardiya
    May 10, 2021 at 6:47
  • 1
    @Manza: It is true they are not the same in the general case, but the common case of binary dilation can be implemented with convolution, since any point in the convolution != 0 will have a maximum neighbor of 1, and any point in convolution == 0 must have a maximum neighbor of 1. And convolution is often faster than other implementations.
    – pavon
    Jun 10, 2021 at 23:48
  • @pavon: In the case of binary convolution that works, indeed, but in the case of the input being and image, it doesn't (even in the case of flat morphology)
    – Manza
    Jun 12, 2021 at 8:28
7

Edit:

I recently collaborated with kornia and now the morphological operations work as expected.

Edit:

I have created a library for doing exactly that; the library is called nnMorpho and can be installed via pip install nnMorpho. The principle I use is the one described below (i.e.: using the unfold function from PyTorch). For the moment the library is in an early stage (only basic operation are implemented) but I will try to update it to include a bigger variety of operations and parameters.

Dilation and convd2d are not the same

Dilation and convd2d are not the same at all: roughly, convd2d performs a linear filter (which means that it does a ponderated sum around a pixel) whereas dilation performs a non linear filter (takes the maximum around a pixel).

A way of doing morphology in PyTorch

There is a way to do mathematical morphology operations in PyTorch. The main problem you face when dealing with dilation and erosion is that you have to consider a neighborhood of each pixel to compute the maximum (and potentially the sums and the differences if dealing with greyscale structural elements). This problem is solved by the function unfold from PyTorch; it currently only supports batched image-like tensors (i.e.: 4D tensors with dimensions (B,C,H,W)) but this shouldn't be a problem for your needs. The rest is just normal operations.

I join a code doing the dilation (erosion is analogous) and examples:

import numpy as np
import torch
from torch.nn import functional as f
from scipy.ndimage import grey_dilation as dilation_scipy
import matplotlib.pyplot as plt


# Definition of the dilation using PyTorch
def dilation_pytorch(image, strel, origin=(0, 0), border_value=0):
    # first pad the image to have correct unfolding; here is where the origins is used
    image_pad = f.pad(image, [origin[0], strel.shape[0] - origin[0] - 1, origin[1], strel.shape[1] - origin[1] - 1], mode='constant', value=border_value)
    # Unfold the image to be able to perform operation on neighborhoods
    image_unfold = f.unfold(image_pad.unsqueeze(0).unsqueeze(0), kernel_size=strel.shape)
    # Flatten the structural element since its two dimensions have been flatten when unfolding
    strel_flatten = torch.flatten(strel).unsqueeze(0).unsqueeze(-1)
    # Perform the greyscale operation; sum would be replaced by rest if you want erosion
    sums = image_unfold + strel_flatten
    # Take maximum over the neighborhood
    result, _ = sums.max(dim=1)
    # Reshape the image to recover initial shape
    return torch.reshape(result, image.shape)


# Test image
image = np.zeros((7, 7), dtype=int)
image[2:5, 2:5] = 1
image[4, 4] = 2
image[2, 3] = 3

plt.figure()
plt.imshow(image, cmap='Greys', vmin=image.min(), vmax=image.max(), origin='lower')
plt.title('Original image')

# Structural element square 3x3
strel = np.ones((3, 3))

# Origin of the structural element
origin = (1, 1)

# Scipy
dilated_image_scipy = dilation_scipy(image, size=(3, 3), structure=strel)

plt.figure()
plt.imshow(dilated_image_scipy, cmap='Greys', vmin=image.min(), vmax=image.max(), origin='lower')
plt.title('Dilated image - Scipy')

# PyTorch
image_tensor = torch.tensor(image, dtype=torch.float)
strel_tensor = torch.tensor(strel, dtype=torch.float)
dilated_image_pytorch = dilation_pytorch(image_tensor, strel_tensor, origin=origin, border_value=-1000)

plt.figure()
plt.imshow(dilated_image_pytorch.cpu().numpy(), cmap='Greys', vmin=image.min(), vmax=image.max(), origin='lower')
plt.title('Dilated image - PyTorch')

plt.show()

The original image proposed in Scipy documentation

dilated image by scipy

dilated image by pytorch

Considerations about the origin

The origin is a key parameter in dilations and erosions. It operates shifting the image. If you want your image unshifted, you should place it in the middle (which means having a odd-sizes structural element). I tried to used it in scipy and it is not working very well, since it is the same around all the dimensions (which poses problems when dealing with non square structural elements). The code I showed is taking the origin into account properly.

4
  • You may want to update the answer given your kornia contribution :)
    – old-ufo
    Aug 25, 2021 at 13:49
  • Sure! Sorry for being late ^^
    – Manza
    Aug 26, 2021 at 14:51
  • Using that method, with a input of mask (0 or 1), I ended up having to subtract 1 from the results to get what I wanted. Is it the expected behavior? Sep 15, 2022 at 1:02
  • Are you using nnMorpho or kornia? or did you had the same behaviour in both? the point is that in morphology, in general, you use substraction instead of multiplication (used in convolution) which means that a mask is no longer composed of 0 and 1 but of -infinity and 0. I will add soon the parameter "footprint"/"mask" for using classical masks (0/1) in morphology.
    – Manza
    Sep 16, 2022 at 8:13
5

The dilation and erosion functions for PyTorch tensors are implemented in the Kornia library https://kornia.readthedocs.io/en/latest/morphology.html.

1

Ardiya's solution can be used to do Erosion with a little trick. You just need to flip the input and then output

torch_result_erosion = 1 - torch.clamp(torch.nn.functional.conv2d(1 - im_tensor, kernel_tensor, padding=(1, 1)), 0, 1)
1
  • I guess this can be used as a pytorch substitute of the scipy binary_erosion() function i.e. torch version of the scipy.ndimage.binary_erosion()
    – T.V.
    Dec 22, 2021 at 4:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.