1

I have got a basic Django web application running on Heroku. I would like to add a spider to crawl some webs (e.g with Scrapy) based on a scheduled task ( e.g. via APScheduler ) to get some tables of Django databases loaded with collected data.

Does anybody know of documentation or examples for the basis to achieve this kind of integration? I find it very hard to figure it out.

1 Answer 1

2

I have not used Scrapy at all, but I'm actually working with APScheduler and it's very simple to use. So my first guess would be to use a BackgroundScheduler (inside your Django app) and add a job to it that would execute a callable "spider" periodically.

The thing here is how could you embed a Scrapy project inside your Django app so you can access one of its "spiders" and effectively use it as a callable in your scheduled job.

I'm maybe not helping much, but I'm just trying to give you some kickstart orientation. I'm pretty sure that if you carefully read the Scrapy's documentation you'll make your way.

Best.

4
  • Yep, BackgroundScheduler is usually the way to go. Jul 24, 2014 at 19:32
  • Vote up my answer :)! haha you're welcome, glad to help a little bit at least!. Jul 25, 2014 at 18:54
  • Hi, could you tell if can add the BackgroudSceduler at my wsgi.py? Dec 14, 2016 at 18:54
  • I'm not sure @JoabMendes, wish I could be of more help Jan 3, 2017 at 3:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.