13

How can I add a resizing layer to

model = Sequential()

using

model.add(...)

To resize an image from shape (160, 320, 3) to (224,224,3) ?

6 Answers 6

11

I think you should consider using tensorflow's resize_images layer.

https://www.tensorflow.org/api_docs/python/tf/image/resize_images

It appears keras does not include this, and perhaps because the feature does not exist in theano. I have written a custom keras layer that does the same. It's a quick hack, so it might not work well in your case.

import keras
import keras.backend as K
from keras.utils import conv_utils
from keras.engine import InputSpec
from keras.engine import Layer
from tensorflow import image as tfi

class ResizeImages(Layer):
    """Resize Images to a specified size

    # Arguments
        output_size: Size of output layer width and height
        data_format: A string,
            one of `channels_last` (default) or `channels_first`.
            The ordering of the dimensions in the inputs.
            `channels_last` corresponds to inputs with shape
            `(batch, height, width, channels)` while `channels_first`
            corresponds to inputs with shape
            `(batch, channels, height, width)`.
            It defaults to the `image_data_format` value found in your
            Keras config file at `~/.keras/keras.json`.
            If you never set it, then it will be "channels_last".

    # Input shape
        - If `data_format='channels_last'`:
            4D tensor with shape:
            `(batch_size, rows, cols, channels)`
        - If `data_format='channels_first'`:
            4D tensor with shape:
            `(batch_size, channels, rows, cols)`

    # Output shape
        - If `data_format='channels_last'`:
            4D tensor with shape:
            `(batch_size, pooled_rows, pooled_cols, channels)`
        - If `data_format='channels_first'`:
            4D tensor with shape:
            `(batch_size, channels, pooled_rows, pooled_cols)`
    """
    def __init__(self, output_dim=(1, 1), data_format=None, **kwargs):
        super(ResizeImages, self).__init__(**kwargs)
        data_format = conv_utils.normalize_data_format(data_format)
        self.output_dim = conv_utils.normalize_tuple(output_dim, 2, 'output_dim')
        self.data_format = conv_utils.normalize_data_format(data_format)
        self.input_spec = InputSpec(ndim=4)

    def build(self, input_shape):
        self.input_spec = [InputSpec(shape=input_shape)]

    def compute_output_shape(self, input_shape):
        if self.data_format == 'channels_first':
            return (input_shape[0], input_shape[1], self.output_dim[0], self.output_dim[1])
        elif self.data_format == 'channels_last':
            return (input_shape[0], self.output_dim[0], self.output_dim[1], input_shape[3])

    def _resize_fun(self, inputs, data_format):
        try:
            assert keras.backend.backend() == 'tensorflow'
            assert self.data_format == 'channels_last'
        except AssertionError:
            print "Only tensorflow backend is supported for the resize layer and accordingly 'channels_last' ordering"
        output = tfi.resize_images(inputs, self.output_dim)
        return output

    def call(self, inputs):
        output = self._resize_fun(inputs=inputs, data_format=self.data_format)
        return output

    def get_config(self):
        config = {'output_dim': self.output_dim,
                  'padding': self.padding,
                  'data_format': self.data_format}
        base_config = super(ResizeImages, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))
6
  • In my case I still got error: TypeError: Output tensors to a Model must be Keras tensors. Found: Tensor("ResizeBilinear_1:0", shape=(?, 224, 224, 3), dtype=float32)
    – Vadym B.
    Oct 20, 2017 at 10:22
  • @VadymB. Which backend are you using, and which version of Keras? Are you sure you are feeding the layer with a correct input?
    – KeithWM
    Oct 21, 2017 at 18:41
  • keras 2.0.6. I already figured out an issue that I was using tf.image.resize_images and I had to put it into keras.layers.Lambda.
    – Vadym B.
    Oct 26, 2017 at 9:11
  • @VadymB. Not to necromance this thread, but when you put this in the lambda layer, were you able to successfully call 'fit' or 'fit_generator' on it? Can you post how you did it? I've been working on a similar problem off and on for days now, and continue to have errors during fitting... Thanks!
    – Decker
    Jul 25, 2018 at 23:33
  • @Decker, It used to work, but unfortunately I don't have working code using this function. For my use case I eventually switched to tf.depth_to_space, while still using Lambda layer. Here is pseudocode: ``` def custom_fun(input, k=2, **kwargs): import tensorflow as tf return tf.depth_to_space(input, block_size=k) input = Input(...) ... out = Lambda(custom_fun)(input) ```
    – Vadym B.
    Jul 26, 2018 at 12:15
11

The accepted answer uses the Reshape layer, which works like NumPy's reshape, which can be used to reshape a 4x4 matrix into a 2x8 matrix, but that will result in the image loosing locality information:

0 0 0 0
1 1 1 1    ->    0 0 0 0 1 1 1 1
2 2 2 2          2 2 2 2 3 3 3 3
3 3 3 3

Instead, image data should be rescaled / "resized" using, e.g., Tensorflows image_resize. But beware about the correct usage and the bugs! As shown in the related question, this can be used with a lambda layer:

model.add( keras.layers.Lambda( 
    lambda image: tf.image.resize_images( 
        image, 
        (224, 224), 
        method = tf.image.ResizeMethod.BICUBIC,
        align_corners = True, # possibly important
        preserve_aspect_ratio = True
    )
))

In your case, as you have a 160x320 image, you also have to decide whether to keep the aspect ratio, or not. If you want to use a pre-trained network, then you should use the same kind of resizing that the network was trained for.

1
  • 2
    This is a good approach, but the problem with the accepted answer isn't that it "loses locality information"--it's that it's doing the wrong thing completely, and "losing locality information" is just a symptom of that.
    – tsbertalan
    Jan 12, 2021 at 20:44
4

I thought I should post an updated answer, since the accepted answer is wrong and there are some major updates in the recent Keras release.

To add a resizing layer, according to documentation:

tf.keras.layers.experimental.preprocessing.Resizing(height, width, interpolation="bilinear", crop_to_aspect_ratio=False, **kwargs)

For you, it should be:

from tensorflow.keras.layers.experimental.preprocessing import Resizing

model = Sequential()
model.add(Resizing(224,224))
0

A modification of @KeithWM 's answer, adding output_scale, e.g. output_scale=2 means the output is 2 times the input shape :)

class ResizeImages(Layer):
    """Resize Images to a specified size
    https://stackoverflow.com/questions/41903928/add-a-resizing-layer-to-a-keras-sequential-model

    # Arguments
        output_dim: Size of output layer width and height
        output_scale: scale compared with input
        data_format: A string,
            one of `channels_last` (default) or `channels_first`.
            The ordering of the dimensions in the inputs.
            `channels_last` corresponds to inputs with shape
            `(batch, height, width, channels)` while `channels_first`
            corresponds to inputs with shape
            `(batch, channels, height, width)`.
            It defaults to the `image_data_format` value found in your
            Keras config file at `~/.keras/keras.json`.
            If you never set it, then it will be "channels_last".

    # Input shape
        - If `data_format='channels_last'`:
            4D tensor with shape:
            `(batch_size, rows, cols, channels)`
        - If `data_format='channels_first'`:
            4D tensor with shape:
            `(batch_size, channels, rows, cols)`

    # Output shape
        - If `data_format='channels_last'`:
            4D tensor with shape:
            `(batch_size, pooled_rows, pooled_cols, channels)`
        - If `data_format='channels_first'`:
            4D tensor with shape:
            `(batch_size, channels, pooled_rows, pooled_cols)`
    """

    def __init__(self, output_dim=(1, 1), output_scale=None, data_format=None, **kwargs):
        super(ResizeImages, self).__init__(**kwargs)
        data_format = normalize_data_format(data_format)  # does not have
        self.naive_output_dim = conv_utils.normalize_tuple(output_dim,
                                                           2, 'output_dim')
        self.naive_output_scale = output_scale
        self.data_format = normalize_data_format(data_format)
        self.input_spec = InputSpec(ndim=4)

    def build(self, input_shape):
        self.input_spec = [InputSpec(shape=input_shape)]
        if self.naive_output_scale is not None:
            if self.data_format == 'channels_first':
                self.output_dim = (self.naive_output_scale * input_shape[2],
                                   self.naive_output_scale * input_shape[3])
            elif self.data_format == 'channels_last':
                self.output_dim = (self.naive_output_scale * input_shape[1],
                                   self.naive_output_scale * input_shape[2])
        else:
            self.output_dim = self.naive_output_dim

    def compute_output_shape(self, input_shape):
        if self.data_format == 'channels_first':
            return (input_shape[0], input_shape[1], self.output_dim[0], self.output_dim[1])
        elif self.data_format == 'channels_last':
            return (input_shape[0], self.output_dim[0], self.output_dim[1], input_shape[3])

    def _resize_fun(self, inputs, data_format):
        try:
            assert keras.backend.backend() == 'tensorflow'
            assert self.data_format == 'channels_last'
        except AssertionError:
            print("Only tensorflow backend is supported for the resize layer and accordingly 'channels_last' ordering")
        output = tf.image.resize_images(inputs, self.output_dim)
        return output

    def call(self, inputs):
        output = self._resize_fun(inputs=inputs, data_format=self.data_format)
        return output

    def get_config(self):
        config = {'output_dim': self.output_dim,
                  'padding': self.padding,
                  'data_format': self.data_format}
        base_config = super(ResizeImages, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))
-1

To resize the given input image to a target size(in this case 224x224x3):

Use Lambda layer in conventional Keras:

from keras.backend import tf as ktf

inp = Input(shape=(None, None, 3))

[Ref: https://www.tensorflow.org/api_docs/python/tf/keras/backend/resize_images] :

-6

Normally you would use the Reshape layer for this:

model.add(Reshape((224,224,3), input_shape=(160,320,3))

but since your target dimensions don't allow to hold all the data from the input dimensions (224*224 != 160*320), this won't work. You can only use Reshape if the number of elements does not change.

If you are fine with losing some data in your image, you can specify your own lossy reshape:

model.add(Reshape(-1,3), input_shape=(160,320,3))
model.add(Lambda(lambda x: x[:50176])) # throw away some, so that #data = 224^2
model.add(Reshape(224,224,3))

That said, often these transforms are done before applying the data to the model because this is essentially wasted computation time if done in every training step.

5
  • I dont get your statement with regard to computation time. Reshaping has to be done once per image. Whats difference does it make if its done before rather than as part of the model pipeline?
    – Oblomov
    Jan 28, 2017 at 7:35
  • It does not make a difference if you only ever train your model once (if we do not take into account whether these operations are faster on a GPU). But as soon as you train your model twice and compute the reshaping more than once you would have benefited from doing it once, caching the result into a file (numpy save or pickle) and simply loading it.
    – nemo
    Jan 29, 2017 at 19:24
  • 1
    Question is about resize, not reshape. In reshape the total array size must stay the same which is not true for resize. Frome Keras function comments:"Raises a ValueError if the total array size of the output_shape is different then the input_shape, or more then one unknown dimension is specified."
    – thanos.a
    Apr 8, 2017 at 16:20
  • 2
    I am quite sure this is a bad idea. The reshape as proposed will loose all the spatial structure in the input data. Also, the loss incurred will be great if the input and output layers are very different in size (total number of elements).
    – KeithWM
    Jul 4, 2017 at 7:58
  • 2
    I downvoted this, because the question is almost certainly looking for image downsampling/upsampling/interpolation, and not a tensor reshape as you suggested.
    – N. McA.
    Apr 26, 2018 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.