3

I have a keras model that works perfectly in unit tests and in local flask app (flask run). However, the moment I launch the flask app in uwsgi, it gets stuck from the second request on, killing the entire app. Is this because uwsgi spawns multiple processes? How do I get around this problem? Thanks.

4
  • You should post the code responsible for the model, the code which gets stuck.
    – nyanpasu64
    Aug 8, 2018 at 3:31
  • Same was happening with me as well because of memory allocation issue on server. Have you checked logs? If it is throwing any kind of warnings then Kindly post them as well. Aug 8, 2018 at 5:35
  • Make sure that you hold the keras model in memory and not reload it for every request
    – sdcbr
    Aug 8, 2018 at 6:35
  • I still have this issue. Did you find the solution ?
    – DrGeneral
    Oct 2, 2018 at 3:52

3 Answers 3

10

I am having the same issue, this may not be ideal work around. But I believe this is an Keras issue running in multiple processes.

I am running uWSGI with lazy-apps = true.

uwsgi --http 0.0.0.0:5000 --wsgi-file your_flask_server.py --callable app --processes 2 --threads 2 --stats 127.0.0.1:9191 --lazy-apps

Note: By lazy loading apps this will consume more memory like double the memory since it is loading the whole app again in each thread/process

here are some use full links: Similar Issue, Similar Issue

Example deployment guide

2
  • It solves my (similar) issue too, but I get a new one: my different Flask routes return 400 once in a while, apparently randomly...
    – jul
    Nov 19, 2019 at 14:12
  • Thanks much Manoj. This still proved to be very useful / helpful with Tensorflow 2.1.1 as well. Oct 23, 2020 at 19:00
3

I had a similar issue: in a flask app inside a docker environment, we would find that the neural network would hang on predict after the first prediction. to get around this I created a neural network class which had

def __init__(self):
    self.session = tf.Session()
    self.graph = tf.get_default_graph()
    self.model = self.__load_model()
    with self.graph.as_default():
        with self.session.as_default():
            logging.info("neural network initialised")

The last 3 lines seemed to properly initialise the graph and the session which for some reason wasn't happening at the correct place. My predict function was then simply:

def predict(self, x):
    with self.graph.as_default():
        with self.session.as_default():
            y = self.model.predict(x)
    return y

This seems to have stopped the hanging (I don't know why I need a session and a graph, but I added both while I was debugging this and now I'm to afraid to remove either)

3

I was able to get my model predicting with the help of multiprocessing.lock

from multiprocessing import Lock

class LockedPredictor:
    def __init__(self):
        self.mutex = Lock()
        self.model = load_model()

    def predict(self, input):
        with self.mutex: 
          return self.model.predict(input)

Inspiration from here: https://dref360.github.io/keras-web/ . However the solution failed when I had two competing models. I tried applying Manager solution described here but failed.

A proper solution would be based on redis queue as proposed here, which is recommended on the Keras website.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.