0

I have a model that needs customized inference hence I modified the predict_step method of the tf.keras.Model class. I want the inference to be modified depending on certain parameters, is there a simple way to have the predict method receive parameters and pass them to the predict_step function?

Something like:

class SimpleModel(tf.keras.Model):
    def __init__(self):
        super().__init__()
        self.threshold = None

    def call(self, inputs, training=None, mask=None):
        return inputs

    def predict(self, x, threshold=0.5, *args, **kwargs):
        self.threshold = threshold
        return super().predict(x, *args, **kwargs)

    def predict_step(self, data):
        return tf.greater(self(data, training=False), self.threshold)


if __name__ == "__main__":
    x = tf.convert_to_tensor([0.0, 0.55, 0.85, 0.9])
    model = SimpleModel()
    model.predict(x, threshold=0.5)
    model.predict(x, threshold=0.75)

The problem with the approach is that since the predict_step has already been created the threshold does not change.

Update 1:

This seems to work, not sure if it is the best way though:

class SimpleModel(tf.keras.Model):
    def __init__(self):
        super().__init__()
        self.threshold = None

    def call(self, inputs, training=None, mask=None):
        return inputs

    def predict(self, x, threshold=0.5, *args, **kwargs):
        self.threshold = threshold
        self.predict_function = None
        return super().predict(x, *args, **kwargs)

    def predict_step(self, data):
        return tf.greater(self(data, training=False), self.threshold)


if __name__ == "__main__":
    x = tf.convert_to_tensor([0.0, 0.55, 0.85, 0.9])
    model = SimpleModel()
    pred = model(x)
    pred_1 = model.predict(x, threshold=0.5)
    pred_2 = model.predict(x, threshold=0.75)
    print(pred, pred_1, pred_2, sep="\n")

Update 2: Following the question I posted here about the predict_step function running in graph mode it seems other way to solve the problem is setting the self.run_eagerly = True of the model.

class SimpleModel(tf.keras.Model):
    def __init__(self):
        super().__init__()
        self.run_eagerly = True
        self.threshold = None

    def call(self, inputs, training=None, mask=None):
        return inputs

    def predict(self, x, threshold=0.5, *args, **kwargs):
        self.threshold = threshold
        return super().predict(x, *args, **kwargs)

    def predict_step(self, data):
        return tf.greater(self(data, training=False), self.threshold)


if __name__ == "__main__":
    x = tf.convert_to_tensor([0.0, 0.55, 0.85, 0.9])
    model = SimpleModel()
    pred_1 = model.predict(x, threshold=0.5)
    pred_2 = model.predict(x, threshold=0.75)
    print(pred_1, pred_2, sep="\n")

It now works without the use of tf.Variable (may run slower because of eager mode).

4
  • Your predict can call predict_step if it wants to (you could use some guard default parameters, i.e. set to None, or just forward some kwargs if you want some customizability). Maybe you want to use decorators? I don't entirely follow the problem. Mar 22, 2021 at 16:59
  • What do you want to do? Mar 22, 2021 at 17:03
  • I updated the example, it's basically to pass parameters to the predict method and have custom inference without changing it too much.
    – Ramon
    Mar 22, 2021 at 17:07
  • @AndrewHolmgren Maybe the update could give you a better idea!
    – Ramon
    Mar 22, 2021 at 17:25

2 Answers 2

1

I have a slightly better idea on what you're after. See this toy example to see if it's kind of what you're after.

class SimpleModel(tf.keras.Model):
    def __init__(self):
        super().__init__()

    def call(self, inputs, training=None, mask=None):
        return inputs

    def custom_predict(func):
        def threshold_handler(self, x, threshold=None, *args, **kwargs):
            if threshold is None:
                return func(self, x, *args, **kwargs)
            else:
                vals = func(self, x, *args, **kwargs)
                return list(filter(lambda x: x > threshold, vals))
        return threshold_handler
    
    # fancy way of saying predict = custom_predict(predict)
    # really, it's running custom_predict masquerading as predict
    @custom_predict
    def predict(self, x, *args, **kwargs):
        return super().predict(x, *args, **kwargs)

x = tf.convert_to_tensor([0.0, 0.55, 0.85, 0.9])
model = SimpleModel()
pred = model(x)
pred_0 = model.predict(x, steps=1)
pred_1 = model.predict(x, threshold=0.5, steps=1)
pred_2 = model.predict(x, threshold=0.75, steps=1)
print(pred, pred_0, pred_1, pred_2, sep="\n")

Of course, the decorator is total overkill when you could have handled the logic in your own predict function, but maybe the higher level idea will get your own ideas flowing on how you want to handle. Another option for customizability is to use callbacks (e.g. see fastai or Pytorch Lightning).

2
  • I find very interesting the decorator approach, It keeps the predict function very clean! I think what I am going for is a combination of this answer and Ender's one. With the decorator I can add the parameters I want to modify for inference and update a tf.Variable per param which will be used on the predict_step function. The real use case much more complicated.
    – Ramon
    Mar 22, 2021 at 20:01
  • 1
    @Ramon I like using decorators in a couple cases: 1) to cleanly wrap a function that came from somewhere else, so that I can extend the function without directly modifying it (like here), and 2) to clearly demark a function template that applies to multiple functions. Callbacks are good when you want to hot-swap between multiple functions at runtime (e.g. maybe you have some prediction mode that thresholds, another that builds a max, another that concatenates, etc.). Each approach is almost like a different style of copy-paste. Mar 22, 2021 at 20:39
1

Since I am not allowed to comment, I'm left with straight up providing an answer which could be very off.

My understanding of your question isn't so much how to call predict_step() multiple times but rather how to make threshold changeable. My suggestion is making the self.threshold a [1 x 1], untrainable variable.

I'm thinking of adding something like

threshold = tf.Variable(.65,trainable=False, dtype='float32')

How I think you could implement it is as follows,

class SimpleModel(tf.keras.Model):
    def __init__(self,threshold=.5):
        super().__init__()
        self.threshold = tf.Variable(threshold,trainable=False, dtype='float32')

    def call(self, inputs, training=None, mask=None):
        return inputs

    def predict(self, x, threshold=0.5, *args, **kwargs):
        self.threshold.assign(tf.convert_to_tensor(threshold,dtype='float32'))
        self.predict_function = None
        return super().predict(x, *args, **kwargs)

    def predict_step(self, data):
        out = tf.greater(self(data, training=False), self.threshold)
        # self.threshold.assign( <Calculate new threshold here as a float32 tensor>)
        return out


if __name__ == "__main__":
    x = tf.convert_to_tensor([0.0, 0.55, 0.85, 0.9])
    model = SimpleModel()
    pred = model(x)
    pred_1 = model.predict(x, threshold=0.5)
    pred_2 = model.predict(x, threshold=0.75)
    print(pred, pred_1, pred_2, sep="\n")

Things changed:

  • threshold argument in __init__() (might not be necessary)
    • If the above is undesired you will have to move the instantiation of the variable from __init__() to predict()
  • Created tf.variable self.threshold in __init__()
  • No longer immediately returning the calculation in predict_step()
  • Added comment placeholder for your threshold recalculation step in predict_step

This code compiled, but because I'm addressing my own interpretation of your query, I might be far from the thing for which you are looking.

1
  • Yes, the question is more on how to extend the predict function to handle parameters that may modify how the inference is done. The threshold in the init is not necessary since I want to decouple inference logic from the model (the model should not worry about the threshold). Using tf.Variable is a great idea! As a matter of fact you can remove self.predict_function = None and will get the same results!
    – Ramon
    Mar 22, 2021 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.