11

I have developed a rest API using Flask to expose a Python Keras Deep Learning model (CNN for text classification). I have a very simple script that loads the model into memory and outputs class probabilities for a given text input. The API works perfectly locally.

However, when I git push heroku master, I get Compiled slug size: 588.2M is too large (max is 500M). The model is 83MB in size, which is quite small for a Deep Learning model. Notable dependencies include Keras and its tensorflow backend.

I know that you can use GBs of RAM and disk space on Heroku. But the bottleneck seems to be the slug size. Is there a way to circumvent this? Or is Heroku just not the right tool for deploying Deep Learning models?

2
  • any contribution more constructive than a silent downvote?
    – Antoine
    Commented Feb 17, 2018 at 17:42
  • Have you tried PCF (pivotal.io/platform)?
    – TYZ
    Commented Mar 1, 2018 at 17:36

7 Answers 7

7

Heroku is a very good cloud platform to deploy your apps but if you have a Deep Learning model i.e. an app that needs to predict using large CNN / Deep Learning models then this cloud is not suitable. You can try other cloud platforms like AWS, Amazon Sagemaker, MS Azure, IBM Watson.

I was facing the same issue and after spending several days I came to know it was tensorflow library that was causing this slug overhead.

I solved it using 1 line in the requirements.txt file:

tensorflow-cpu==2.5.0

Instead of

tensorflow==2.5.0

You can use any updated tensorflow library version. Read more about tensorflow-cpu here

2
  • This answer deserves a lot of up-votes. Commented Nov 7, 2021 at 16:58
  • Thanks a lot for appreciating :) Commented Nov 8, 2021 at 18:09
5
+50

The first thing I would check, as suggested by others, is to find out why your repo is so big given that the model size is only 83MB.

Given that you cannot reduce the size there is the option of offloading parts of the repo, but to do this you will still need an idea of which files are taking up the space. Offloading is suggested in the heroku docs. Slug size is limited to 500MB as stated here: https://devcenter.heroku.com/articles/slug-compiler#slug-size and I believe this has to do with the time it takes to spin up a new instance if a change in resources is needed. However, you can use offloading if you have particularly large files. More info on offloading here: https://devcenter.heroku.com/articles/s3

4
  • +1 thanks for the suggestions. 1) I think the dependencies (especially tensorflow) takes a lot of space. 2) my model is the largest file. Do you suggest offloading it to s3? Then, I would need to download it from s3 every time the app is started?
    – Antoine
    Commented Mar 5, 2018 at 9:20
  • It would make sense to offload the model and any other large files to S3. Heroku actually runs on AWS EC2. I believe that since both S3 and EC2 are AWS services the Heroku app running on EC2 should be able to directly access the S3 storage. Commented Mar 6, 2018 at 17:58
  • Another thought, it sounds that your app is deployment of a pre-trained network. If this is the case, have you used tensorflow's deployment tools for reducing the size of the network and library? The tensorflow libs include overhead for training that you don't need for deployment. These are just some links I found after quickly Googling for tensorflow deployment tensorflow.org/deploy and tensorflow.org/mobile/prepare_models Commented Mar 6, 2018 at 18:02
  • marked the answer as accepted because offloading (dropbox) did solve my problem. However, the RAM on Heroku dynos (up to Standard1X) is of only 500M... So I still have a problem if my model uses more than 500M when loaded into memory.
    – Antoine
    Commented Mar 16, 2018 at 9:25
5

You can reduce the model size and use tensorflow-cpu which has a smaller size (144MB with Python 3.8)

pip install tensorflow-cpu

https://pypi.org/project/tensorflow-cpu/#files

3

This answer assumes that your model is only 83MB and the total size of your repository directory is smaller (likely much smaller) than 500MB.

There could be a few issues, but the obvious thing you need to do is reduce your git repository to less than 500MB.

First, try commands like the following to reduce the size of your repo (see this blog post for reference):

heroku plugins:install heroku-repo
heroku repo:gc --app your-app-name
heroku repo:purge_cache --app your-app-name

These might solve your issue.

Another potential issue is that you have at some point committed another (large size) model and removed it from your repo in a subsequent commit. The git repo now includes a version of that model in your .git folder and git history. There are a few fixes for this, but if you don't need your commit history you can copy the repo to another folder and create a fresh git repo with git init. Commit everything with something like "Initial commit" and then try pushing this repo with only one commit to Heroku. Likely that will be a much smaller repo size.

1
  • I tried the first solution without success. However, I indeed had previously committed another large size model (~65MB) which I removed (to replace it with the new model) - so I'm gonna try your second solution
    – Antoine
    Commented Mar 4, 2018 at 9:26
3

A lot of these answers are great for reducing slug size but if anyone still has problems with deploying a deep learning model to heroku it is important to note that for whatever reason tensorflow 2.0 is ~500MB whereas earlier versions are much smaller. Using an earlier version of tensorflow can greatly reduce your slug size.

1
  • 1
    Thanks! This reduced my heroku app size by ~400MB. Tensorflow 1.* is ~ 110MB much less than tensorflow 2.0 ~500MB
    – jsdbt
    Commented May 24, 2020 at 1:33
2

As a resource, you can visit the Heroku Slug Compiler help page.

Having an 83MB model size doesn't mean that it is 83MB all the way. Since packages are being compiled when being pushed to Heroku, this will obviously eat up more slug space so that the packages can be ready for use by the application. The best solution is probably to put large assets to a container like AWS S3 or any other good counterpart. Or worst is to use a different cloud service.

3
  • isn't it possible to slice the app into smaller chunks when pushing to heroku? I don't understand why they enforce a limit of 500MB when pushing while they give gigabytes of RAM for running your application once it is on their platform
    – Antoine
    Commented Feb 28, 2018 at 18:06
  • S3 won't work in my case. I don't need a file storage solution - I need to be able to load the deep learning model on the RAM on the same machine where my flask app is running
    – Antoine
    Commented Feb 28, 2018 at 18:11
  • In that case you might just need to use a service other than Heroku. Commented Mar 1, 2018 at 4:18
2

I would say that Heroku is not the right tool for deploying the deep learning model itself. For that, you could consider using a Platform as a Service dedicated to Deep Learning, such as Floydhub. You could deploy your Flask REST API on Floydhub too.

1
  • 1
    the rest API option on Floydhub is still a feature under development. and a given script can only run for 7 consecutive days, while I need my API to be always up. It seems to me Floydhub is more a tool for training Deep Learning models rather than exposing them right now...
    – Antoine
    Commented Feb 28, 2018 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.