In Scrapy Cloud, a job
can be deleted
. At first, its state will change, but the data will be available - items data, job.metadata
, items.stats
.
What's the process of deletion in particular? How soon all of data will be erased?
1 Answer
Job deletion in Scrapy Cloud is an asynchronous process: at first, the jobs are just marked as deleted with all its data available, and then it's purged on the server side after 7 days (using a cron job). It's implemented in this way to have the last chance to recover jobs deleted by mistake or save some important data.
Also, purging in the context relates only to job data (requests/items/logs/samples), while job metadata and stats are stored indefinitely for now (though you shouldn't rely on that, the policy can be revised at any given moment).
-
Purged meaning completely -
job
withitems, metadata, stats
? You say 1-2 days, so as I understand after some time a deleted job enters a queue where it's purged? Jun 11, 2019 at 16:44 -
Good clarifications, thanks, I've updated my reply. Also, it turned out to be 7 days instead of 1-2 days, I confirmed it with my colleague. Jun 12, 2019 at 10:39