1

I didn't find an answer to my question so far so I'm giving it a try here.

Let's assume a Spring Batch application with remote partitioning. There's one master/manager application partitioning the dataset and sending it to Kafka (to multiple partitions) and worker nodes are consuming from different Kafka partitions to run in parallel. So far so good.

The question is, what happens if while the workers are still processing the data and doing their own things, the manager application suddenly crashes.

The obvious answer is that the partitioning job execution will stay as STARTED even though the respective worker jobs have the state COMPLETED.

How can I restart the master node without doing the partitioning again and triggering the workers? The only thing I'd want in this case is to mark that particular job execution as COMPLETED since all the worker steps have completed.

I tried restarting the job with the JobOperator interface but obviously it fails since the job is in STARTED state and not in FAILED.

Caused by: org.springframework.batch.core.UnexpectedJobExecutionException: Illegal state (only happens on a race condition): job execution already running with name=partitioningJob and parameters={}

Any suggestions are welcome. Thanks!

1 Answer 1

0

You can change the status of the job execution from STARTED to FAILED and set its END_TIME to a non null value before restarting the same job instance (you might need to do that for the step execution as well if needed). On restart, the manager should notice that all workers have completed and will complete the execution.

13
  • Hi Mahmoud, nice to have you here. :) You mean manually doing an SQL update or is there an easier API to use (like a Repository or something)? Btw is this something on the Spring Batch roadmap? Generally speaking, is there any doc/suggestions on failure handling both on the master and worker side? Sorry for the long comment. :) Mar 28, 2022 at 20:03
  • Thank you! Yes, I meant changing the status manually in the db as mentioned in the appendix here: docs.spring.io/spring-batch/docs/current/reference/html/…. You can do it programmatically if you manage to detect the failure of the manager. There is no plan to automate this in Spring Batch as this is a business decision (mark the job as failed and restart it or abandon it). The rule of thumb when a partitioned job fails (either the manager of one of the workers) is to restart the same job instance: only failed partitions should be reprocessed. Mar 29, 2022 at 7:50
  • Thanks Mahmoud. I have another one for you. What happens when the workers are still processing the partitions while the manager is restarted? Still the manager should move the partitioning job status to FAILED and then restart the job? In that case the manager will still wait until the workers complete all their partitions? Mar 31, 2022 at 12:21
  • In that case, the manager should wait until all workers finish their partitions (successfully or with a failure) up to a configurable timeout. Mar 31, 2022 at 13:47
  • I'm sorry, I don't want to sound ignorant, just wanna make sure I understand it. So in the case I described above (workers are still processing but the manager got restarted), I still need to transition the job state to FAILED and restart the batch job with a JobOperator even though the workers are still processing. This will instruct Spring Batch to restart the job (since it failed) but since the workers are still processing, it's just going to wait until all of them are done. (?) Mar 31, 2022 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.