27

I have a Flask app, served by Nginx and Gunicorn with 3 workers. My Flask app is a API microservice designed for doing NLP stuff and I am using the spaCy library for it.

My problem is that they are taking huge number of RAM as loading the spaCy pipeline spacy.load('en') is very memory-intensive and since I have 3 gunicorn workers, each will take about 400MB of RAM.

My question is, is there a way to load the pipeline once and share it across all my gunicorn workers?

6
  • 1
    Maybe you can use preload_app of gunicorn. see stackoverflow.com/a/27242874/7724457
    – tell k
    Apr 18, 2017 at 8:40
  • 2
    Have you found a solution? Apr 17, 2018 at 5:58
  • @Lee, anything you have found out in this area?
    – MichaelR
    Dec 30, 2018 at 9:39
  • How do you use your gunicorn worker, i.e., thread or process? If process can you use redis?
    – JUNPA
    Mar 13, 2021 at 11:15
  • This question is almost as if it's posted by me. I have the exact same setup!
    – thethiny
    Jun 21, 2021 at 14:30

4 Answers 4

1

I need to share Gigabytes of data among instances and use a memory mapped file (https://docs.python.org/3/library/mmap.html). If the amount of data you need to retrieve per request from the pool is small this works fine. Otherwise you can mount a ramdisk where you locate the mounted file.

As I am not familiar with SpaCy I am not sure if this helps. I would have one worker for actually processing the data while loading (spacy.load?) and writing the resulting doc (pickling/marshalling) to the mmf where the other workers can read from it.

To get a better feel of mmap have a look at https://realpython.com/python-mmap/

0

One workaround is, you can load the spaCy pipeline before-hand, pickle (or any comfortable way of serializing) the resultant object and store it in a DB or file system. Each worker can just fetch the serialized object, and simply deserialize it.

2
  • Would pickling it in a module work? I am thinking to load the module in a script, import it, then launch the workers. If doing this would the workers pick up the import?
    – OneLiner
    Feb 18, 2021 at 15:24
  • 1
    When you load the module in a script and import, the import functionality will work for sure. But you will still face the problem of high memory consumption because each worker will load it independently again. You have to pickle it separately before launching the Flask app and store the serialized file in a place accessible by the workers. Feb 19, 2021 at 9:20
0

Sharing the pipeline in memory between workers may help you.

Please check gc.freeze

I think, just do this, in your app.py:

  1. freeze the gc
  2. load pipeline or any other resources that is going to use a big amount of memory
  3. unfreeze gc

and,

  • make sure your worker will not modify (directly or indirectly) any object created during freezing
  • pass the app.py to gunicorn

When fork happens, those memory page holding big resources will not be truly copied by os, because you make sure there are no write operations on it.

If you do not freeze gc, the memory pages will still be written, because gc is writing object reference counts. That why freeze matters.

I just know this way but I didn't try it.

2
  • By the way, gc freezing is firstly added in python3.7 Jan 20, 2021 at 9:15
  • freeze > load > unfreeze results in unresponsive workers when workers restart after serving max_requests. load followed by freeze is what worked for us. Nov 16, 2022 at 11:42
0

This is an answer that works in 2021 using Python3.6 and 3.9 (tested both). I had the same setup as you, using flask to deploy a Spacy NLU API. The solution was to simply was to append --preload to the gunicorn command like so: gunicorn src.main:myFlaskApp --preload. This would cause the fork to happen after the entire src/main.py file has been executed, and not after the myFlaskApp = Flask(__name__).

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.