5

I have a script that scrapes data by URLslist. This script is executing in a docker container. I would like to run it in multiple instances, for example, 20. For that, I wanted to use docker-compose scale worker=20 and to pass the INDEX to each instance so that the script knows which URLs should be scraped .

Example.

ID, URL
0 https://example.org/sdga2
1 https://example.org/fsdh34
2 https://example.org/fs4h35
3 https://example.org/f1h36
4 https://example.org/fs4h37
...

If there are 3 instances, 1st instance of script should process a url whose ID equals to 0, 3, 6, 9 i.e. ID = INDEX + INSTANCES_NUM * k.

I don't know how to pass INDEX to script running in Docker container. Of course, I can duplicate services in docker-compose.yml with different INDEX in environment vars. But if instances number is greater 10 or even 50 it will be a very bad solution)

Does anyone know how do this?

1
  • 1
    Please include your compose file.
    – BMitch
    May 8, 2019 at 12:07

2 Answers 2

5

With docker-compose, I don't believe there's any support for this. However, with swarm mode, which can use a similar compose file, you can pass {{.Task.Slot}} as an environment variable using service templates. E.g.

version: '3'
services:
  test:
    image: busybox
    command: /bin/sh -c "echo My task number is $$task_id && tail -f /dev/null"
    environment:
      task_id: "{{.Task.Slot}}"
    deploy:
      replicas: 5

Instead of docker-compose up, I deploy with docker stack deploy -c docker-compose.yml test. My local swarm cluster is just a single node created with docker swarm init.

Then, reviewing each of these running containers:

$ docker ps --filter label=com.docker.swarm.service.name=test_test
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS               NAMES
ccd0dbebbcbe        busybox:latest      "/bin/sh -c 'echo My…"   About a minute ago   Up About a minute                       test_test.3.i3jg6qrg09wjmntq1q17690q4
bfaa22fa3342        busybox:latest      "/bin/sh -c 'echo My…"   About a minute ago   Up About a minute                       test_test.5.iur5kg6o3hn5wpmudmbx3gvy1
a372c0ce39a2        busybox:latest      "/bin/sh -c 'echo My…"   About a minute ago   Up About a minute                       test_test.4.rzmhyjnjk00qfs0ljpfyyjz73
0b47d19224f6        busybox:latest      "/bin/sh -c 'echo My…"   About a minute ago   Up About a minute                       test_test.1.tm97lz6dqmhl80dam6bsuvc8j
c968cb5dbb5f        busybox:latest      "/bin/sh -c 'echo My…"   About a minute ago   Up About a minute                       test_test.2.757e8evknx745120ih5lmhk34

$ docker ps --filter label=com.docker.swarm.service.name=test_test -q | xargs -n 1 docker logs
My task number is 3
My task number is 5
My task number is 4
My task number is 1
My task number is 2
0
-1

why don't you use a database? mysql or redis.

each container can fetch urls from the database and you can mark fetched urls as complete, always fetch not completed urls from each container. This can scale.

3
  • Look at BMitch's solution.
    – Daler
    May 8, 2019 at 12:44
  • I’d lean towards a dedicated job queue like RabbitMQ, but same idea.
    – David Maze
    May 8, 2019 at 13:24
  • There's a lot of answers as to "why not" depending on the code itself. This is a relatively generic requirement and can be reflected onto a lot of different problems. The primary issue with this suggestion is that failed jobs make a big mess. You must handle the situation of starting a new task when you're not certain really dead, or just busy. On the other hand docker / swarm / kubernetes etc., these are all aware when a job is completely gone (process is dead) and can smoothly handle restart logic without the risk of running two at a time. Jan 26, 2021 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.