1

In Scrapy Cloud, a job can be deleted. At first, its state will change, but the data will be available - items data, job.metadata, items.stats. What's the process of deletion in particular? How soon all of data will be erased?

1 Answer 1

2

Job deletion in Scrapy Cloud is an asynchronous process: at first, the jobs are just marked as deleted with all its data available, and then it's purged on the server side after 7 days (using a cron job). It's implemented in this way to have the last chance to recover jobs deleted by mistake or save some important data.

Also, purging in the context relates only to job data (requests/items/logs/samples), while job metadata and stats are stored indefinitely for now (though you shouldn't rely on that, the policy can be revised at any given moment).

2
  • Purged meaning completely - job with items, metadata, stats? You say 1-2 days, so as I understand after some time a deleted job enters a queue where it's purged? Jun 11, 2019 at 16:44
  • Good clarifications, thanks, I've updated my reply. Also, it turned out to be 7 days instead of 1-2 days, I confirmed it with my colleague. Jun 12, 2019 at 10:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.