9

I am trying to change the activation function of the last layer of a keras model without replacing the whole layer. In this case, only the softmax function

import keras.backend as K
from keras.models import load_model
from keras.preprocessing.image import load_img, img_to_array
import numpy as np

model = load_model(model_path)  # Load any model
img = load_img(img_path, target_size=(224, 224))
img = img_to_array(img)
print(model.predict(img))

My output:

array([[1.53172877e-07, 7.13159451e-08, 6.18941920e-09, 8.52070968e-07,
    1.25813088e-07, 9.98970985e-01, 1.48254022e-08, 6.09538893e-06,
    1.16236095e-07, 3.91888688e-10, 6.29304608e-08, 1.79565995e-09,
    1.75571788e-08, 1.02110009e-03, 2.14380114e-09, 9.54465733e-08,
    1.05938483e-07, 2.20544337e-07]], dtype=float32)

Then I do this to change the activation:

model.layers[-1].activation = custom_softmax
print(model.predict(test_img))

and the output I got is exactly the same. Any ideas how to fix? Thanks!

You could try to use the custom_softmax below:

def custom_softmax(x, axis=-1):
"""Softmax activation function.
# Arguments
    x : Tensor.
    axis: Integer, axis along which the softmax normalization is applied.
# Returns
    Tensor, output of softmax transformation.
# Raises
    ValueError: In case `dim(x) == 1`.
"""
ndim = K.ndim(x)
if ndim >= 2:
    return K.zeros_like(x)
else:
    raise ValueError('Cannot apply softmax to a tensor that is 1D')
5
  • Are you sure the original model doesn't also already end with a softmax? Those existing outputs are already very close to summing up to 1 (they add up to 1.00000002685) Commented Mar 24, 2018 at 12:49
  • @DennisSoemers I am trying to implement a customize softmax, so it would be a bit different than the normal one. Commented Mar 24, 2018 at 13:44
  • Can you share the code of this customized softmax? Just to make sure that it wouldn't happen to generate the same output as the original one? Or, alternatively, stick a print statement into your custom activation function's code. If you see those prints appearing, you know that your activation function is being called. Commented Mar 24, 2018 at 13:46
  • @DennisSoemers Ive added the function. I expect it to output zeros if you call predict method. Commented Mar 24, 2018 at 13:59
  • 2
    The problem is that the Tensorflow graph is not updated, regardless of the fact that the keras layer is updated. The change doesn't take into effect even recompiling with a new model. The only successful solution I have seen is github.com/raghakot/keras-vis/blob/master/vis/utils/utils.py utils.apply_modifications and it is a bit of a clunky way.
    – layser
    Commented Jun 27, 2018 at 16:28

1 Answer 1

5

At the current state of things there's no official, clean way to do that. As pointed by @layser in the comments, the Tensorflow graph isn't being updated - which results in the lack of change in your output. One option is to use keras-vis' utils. My recommendation is to isolate that in your own utils.py, like so:

from vis.utils.utils import apply_modifications

def update_layer_activation(model, activation, index=-1):
    model.layers[index].activation = activation
    return apply_modifications(model)

Which would lead to a similar use:

model = update_layer_activation(model, custom_softmax)

If you follow the given link, you'll see what they do is quite simple: they save the model to a temporary path, then load it back and return, finally deleting the temp file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.