0

I have a script which which when run adds rss feed parsing tasks to some celery queues. Now I have implemented apscheduler to run the script every 2 hours to get new data from the feeds.

My implementation looks like this:

#!/usr/bin/env python
import atexit
import logging
import os
from logging import getLogger

from apscheduler.schedulers.blocking import BlockingScheduler


logger = getLogger('scheduled_parser')
PARSER_SCHEDULER = 'parser_scheduler'


def main():
   scheduler = BlockingScheduler(job_defaults={'coalesce': True})
   scheduler.add_jobstore('sqlalchemy',alias='scheduler_config', url=os.environ.get("DATABASE_URL"))
   scheduler.add_job(run_parser, 'interval', seconds=int(os.environ.get("SCHEDULER_RUN_FREQUENCY")),
                     id=PARSER_SCHEDULER, replace_existing=True)
   scheduler.start()
   atexit.register(lambda: scheduler.shutdown())

def run_parser():
   < code to add items to queues>

if __name__ == "__main__":
   logging.basicConfig()
   logger.setLevel(logging.INFO)
   main()

My code is deployed on heroku and I have following in my procfile

clock: python scheduled_parser
<celery worker processes>

I am having following issues:

  1. I am storing the scheduler job in persistant storage and I can even see it in my db, but when I do scheduler.get_job(PARSER_SCHEDULER,'scheduler_config') I get None
  2. Whenever I deploy on heroku, I think the next run is being updated. For example if parser is set to run every 2 hours and next run going to be at 4:00pm and if I deploy on Heroku at 3:00pm then my next run happens at 5:00pm instead of 4:00pm.

1 Answer 1

0

Not sure about your issue #1, but I think issue #2 is that on every deploy, this line is going to replace the job, thus resetting the schedule:

   scheduler.add_job(run_parser, 'interval', seconds=int(os.environ.get("SCHEDULER_RUN_FREQUENCY")),
                 id=PARSER_SCHEDULER, replace_existing=True)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.