3

Here is my folder layout:

  • my_django_project
    • project
      • project
        • __init__.py
        • settings.py
        • urls.py
        • wsgi.py
      • people
        • management
          • __init__.py
          • commands
            • __init__.py
            • scrapy.py
        • migrations
        • __init__.py
        • admin.py
        • models.py
        • tests.py
        • views.py
      • scrapy_project
        • scrapy_project
          • spiders
            • __init__.py
            • my_scraper.py
          • __init__.py
          • items.py
          • pipelines.py
          • settings.py

And what I have been doing is following this tutorial: Access django models inside of Scrapy

And I followed everything and I still can't do much about it. Every time I run python manage.py scrapy crawl my_scraper it gives me the same error:

ImportError: No module named scrapy_project.settings

Now I do import the settings in Django's settings.py:

import os

os.environ['SCRAPY_SETTINGS_MODULE'] = 'scrapy_project.settings'

I have tried different approaches. Now I have been able to connect scrapy with Django models, but I want to be able to run the scrapy project from Django.

Here is my scrapy.py in the commands folder:

from __future__ import absolute_import
from django.core.management.base import BaseCommand

class Command(BaseCommand):
    def run_from_argv(self, argv):
        self._argv = argv
        self.execute()

    def handle(self, *args, **options):
        from scrapy.cmdline import execute
        execute(self._argv[1:])

1 Answer 1

4

As far as I understand, you need to move the inner scrapy_project package one level up.

1
  • 1
    Basically I moved what was in the first scrapy_project folder to the top so that it is under the second project folder. Thank You!
    – Nazariy
    Jan 23, 2015 at 3:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.