1

I am new to django, docker and scrapy and I am trying to run a django app that also uses scrapy (I basically create a django app that is also a scrapy app and try to call a spider from a django view). Despite specifying this scrapy in requirements.txt and running pip from the Dockerfile, the dependencies are not installed in the container prior to running python manage.py runserver 0.0.0.0:8000 and the django app fails during the system checks, resulting in the web container exiting because of the following exception:

 | Exception in thread django-main-thread:
web_1  | Traceback (most recent call last):
web_1  |   File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
web_1  |     self.run()
web_1  |   File "/usr/local/lib/python3.7/threading.py", line 870, in run
web_1  |     self._target(*self._args, **self._kwargs)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 54, in wrapper
web_1  |     fn(*args, **kwargs)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 117, in inner_run
web_1  |     self.check(display_num_errors=True)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 390, in check
web_1  |     include_deployment_checks=include_deployment_checks,
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 377, in _run_checks
web_1  |     return checks.run_checks(**kwargs)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/checks/registry.py", line 72, in run_checks
web_1  |     new_errors = check(app_configs=app_configs)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/checks/urls.py", line 40, in check_url_namespaces_unique
web_1  |     all_namespaces = _load_all_namespaces(resolver)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/checks/urls.py", line 57, in _load_all_namespaces
web_1  |     url_patterns = getattr(resolver, 'url_patterns', [])
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/utils/functional.py", line 80, in __get__
web_1  |     res = instance.__dict__[self.name] = self.func(instance)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/urls/resolvers.py", line 579, in url_patterns
web_1  |     patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/utils/functional.py", line 80, in __get__
web_1  |     res = instance.__dict__[self.name] = self.func(instance)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/urls/resolvers.py", line 572, in urlconf_module
web_1  |     return import_module(self.urlconf_name)
web_1  |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
web_1  |     return _bootstrap._gcd_import(name[level:], package, level)
web_1  |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
web_1  |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
web_1  |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
web_1  |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
web_1  |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
web_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
web_1  |   File "/code/composeexample/urls.py", line 21, in <module>
web_1  |     path('scrapy/', include('scrapy_app.urls')),
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/urls/conf.py", line 34, in include
web_1  |     urlconf_module = import_module(urlconf_module)
web_1  |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
web_1  |     return _bootstrap._gcd_import(name[level:], package, level)
web_1  |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
web_1  |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
web_1  |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
web_1  |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
web_1  |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
web_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
web_1  |   File "/code/scrapy_app/urls.py", line 4, in <module>
web_1  |     from scrapy_app import views
web_1  |   File "/code/scrapy_app/views.py", line 1, in <module>
web_1  |     from scrapy.crawler import CrawlerProcess
web_1  | ModuleNotFoundError: No module named 'scrapy'

I tried using pip3 instead of pip, pip install --no-cache-dir -r requirements.txt, changing the order of the statements in the Dockerfile and I also checked that Scrapy==1.7.3 appears in requirements.txt. Nothing seems to work.

This is my Dockerfile:

FROM python:3
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/

And this is my docker-compose.yml:

version: '3'

services:
  db:
    image: postgres
  web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - "8000:8000"
    depends_on:
      - db

4
  • 1
    Please show requirements.txt
    – Alasdair
    Commented Aug 17, 2019 at 20:59
  • ``` (...) chardet==3.0.4 constantly==15.1.0 cryptography==2.7 cssselect==1.1.0 Django==2.2.4 hyperlink==19.0.0 idna==2.8 incremental==17.5.0 lxml==4.4.1 parsel==1.5.2 psycopg2==2.8.3 pyasn1==0.4.6 pyasn1-modules==0.2.6 pycparser==2.19 PyDispatcher==2.0.5 PyHamcrest==1.9.0 pyOpenSSL==19.0.0 python-scrapyd-api==2.1.2 pytz==2019.2 queuelib==1.5.0 requests==2.22.0 Scrapy==1.7.3 scrapy-djangoitem==1.1.1 scrapyd==1.2.1 service-identity==18.1.0 six==1.12.0 sqlparse==0.3.0 Twisted==19.7.0 urllib3==1.25.3 w3lib==1.21.0 zope.interface==4.6.0 ```
    – Iuli
    Commented Aug 18, 2019 at 11:41
  • Could you please show you code?
    – ozlevka
    Commented Aug 18, 2019 at 14:44
  • 1
    Which command are you using to bring the container up? Are you sure the image it's running on is up to date and that the requirements file in it has Scrapy in it? Commented Aug 18, 2019 at 18:12

2 Answers 2

3

A little late, but I came across this issue and eventually figured it out (and am putting this here for anyone else who has the same issue).

The first time I tried building my docker image, my requirements.txt didn't contain the required modules. Of course, I added the required modules but nothing seemed to happen, this is because we need to rebuild the container from scratch, otherwise we are just attempting to build the same version again and again.

To rebuild the container with our updated files, we write:

docker-compose rm -f
docker-compose pull
docker-compose up

If that doesn't work, try the same but replace final line with docker-compose up --build -d.

I got this from this answer.

1

It seems like you are lacking scrapy in your requirements.txt !

I tried to build a minimal version with all of your components. Hope it helps.

test.py

import scrapy
from time import sleep


def main():
    while True:
        print(scrapy)
        sleep(1)


if __name__ == "__main__":
    main()

requirements.txt

Scrapy==1.7.3

Dockerfile

FROM python:3

ENV PYTHONUNBUFFERED 1

WORKDIR /code

COPY requirements.txt .
RUN pip3 install -r requirements.txt

COPY . ./

CMD [ "python3", "test.py" ]

docker-compose.yml

version: '3'

services:
  db:
    image: postgres
  web:
    build: .
    ports:
      - "8000:8000"
    depends_on:
      - db
4
  • I also refactored the docker part a bit ^^
    – Urhengulas
    Commented Aug 17, 2019 at 21:24
  • I do have Scrapy :( I checked
    – Iuli
    Commented Aug 18, 2019 at 11:35
  • 1
    @Iuliana Voinea How did you check? Can you check again by going into the docker container via docker exec -it web_1 /bin/bash then doing a pip show to check for Scrapy installation?
    – pangyuteng
    Commented Aug 18, 2019 at 19:12
  • yes i did and indeed it is not installed in the container, but that is not my problem, my problem is why it does not execute pip install -r requirements.txt specified in the Dockerfile. I also checked the contents of requirements.txt in the container and they are correct (contains Scrapy).
    – Iuli
    Commented Aug 19, 2019 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.