I have a script that scrapes data by URLslist.
This script is executing in a docker container.
I would like to run it in multiple instances, for example, 20.
For that, I wanted to use docker-compose scale worker=20
and to pass the INDEX to each instance so that the script knows which URLs should be scraped .
Example.
ID, URL
0 https://example.org/sdga2
1 https://example.org/fsdh34
2 https://example.org/fs4h35
3 https://example.org/f1h36
4 https://example.org/fs4h37
...
If there are 3 instances, 1st instance of script should process a url whose ID equals to 0, 3, 6, 9 i.e. ID = INDEX + INSTANCES_NUM * k.
I don't know how to pass INDEX to script running in Docker container. Of course, I can duplicate services in docker-compose.yml with different INDEX in environment vars. But if instances number is greater 10 or even 50 it will be a very bad solution)
Does anyone know how do this?