Are you sure it never moves to the next step? Maybe it does, but, for instance, in 5 minutes?
I'm asking that because Step Functions has the limitation: even if your Glue job executes in a few seconds, Step Functions polls the results from Glue job once every 5 minutes actually.
A kind of workaround you could implement is to change arn:aws:states:::glue:startJobRun.sync
to arn:aws:states:::glue:startJobRun
— then Glue job task just will trigger the Glue job and will move to the next step.
Most likely, you will need to wait the Glue job finished and get some result out of there. Therefore, you need to wrap the previous state with a few more ones.
- The main purpose is to merely start the Glue job. Apart from that, we need Glue job
RunJobId
.
I don't know if it can be retrieved from Glue job itself, so I've created a Lambda to run the Glue job using boto3 start_job_run
function and then get RunJobId
from the response.
- Create a Lambda which will be grabbing the status (
JobRunState
) of the Glue job (via boto3 get_job_run
function) by RunJobId
from the previous step.
- Using
Wait
Step Functions state type, run the Lambda you created every N seconds.
- Use
Choice
state type to filter Glue job statuses out.
- If
RUNNING
, go back to the Wait
step.
- If
SUCCEEDEED
, then go ahead to the next state.
- If
[FAILED | STOPPED]
, go wherever else.
Finally, it looks something like this.
State machine definition
in aws console)?