1

Some jobs are remaining with pending pending state and I can't cancel them.

How do I cancel the job.

Web console shows like this.

  • "The graph is still being analyzed."
  • All logs are "No entries found matching current filter."
  • Job status: "Starting..." There isn't appered a cancel button yet.

There are no instances in the Compute Engline tab.

What I did is below. I created a streaming job. it was simple template job, Pubsub subscription to BigQuery. I set machineType as e2-micro because it was just a testing.

I also tried to drain and cancel by gcloud but it doesn't work.

$ gcloud dataflow jobs drain --region asia-northeast1 JOBID

Failed to drain job [...]: (...): Workflow modification failed. Causes: (...): 
Operation drain not allowed for JOBID. 
Job is not yet ready for draining. Please retry in a few minutes. 
Please ensure you have permission to access the job and the `--region` flag, asia-northeast1, matches the job's
region.

This is jobs list

$ gcloud dataflow jobs list --region asia-northeast1
JOB_ID  NAME                               TYPE       CREATION_TIME        STATE      REGION
JOBID1  pubsub-to-bigquery-udf4            Streaming  2021-02-09 04:24:23  Pending    asia-northeast1
JOBID2  pubsub-to-bigquery-udf2            Streaming  2021-02-09 03:20:35  Pending    asia-northeast1
...other jobs...

Please let me know how to stop/cancel/delete these streaming jobs.

Job IDs:

  • 2021-02-08_20_24_22-11667100055733179687
  • 2021-02-08_20_24_22-11667100055733179687

WebUI: https://i.stack.imgur.com/B75OX.png

https://i.stack.imgur.com/LzUGQ.png

4

2 Answers 2

0

As per personal experience some time few instance get stuck either they keep on running, or they cannot be canceled or you can not see thr graphical data flow pipelines. Best way to handle this kind of issue is to leave them in thr status, unless it is not impacting your solution by exceeding maximum concurrent runs at a moment. It will be canceled automatically or by Google team, since Dataflow is a google managed.

4
  • Thanks for your answer. I see. I wait it will be canceled by the google team. The jobs are still stuck for 48 hours. Fortunately, the job doesn't seem to cost anything according to my billing. so I just wait.
    – tsbm
    Feb 11, 2021 at 2:05
  • Finally, The jobs were stopped as status "Failed" after 6 days passed. Thank you. imgur.com/a/MbSZqur
    – tsbm
    Feb 16, 2021 at 0:48
  • Does it indicate any failure reason over Job logs ? Feb 16, 2021 at 7:57
  • 1
    It shows "Worker machine type has insufficient memory to support this type of Dataflow job." Yes, I set machine type as e2-micro. Probably, It seems too small. I think GCP DataFlow should reject to start of this machine type.
    – tsbm
    Feb 16, 2021 at 13:06
0

In GCP console Dataflow UI, if you have running Dataflow jobs, you will see the "STOP" button just like the below image.

enter image description here

Press the STOP button. enter image description here

When you successfully stop your job, you will see the status like below. (I was too slow to stop the job with the first try, so I had to test it again. :) ) enter image description here

4
  • Thanks for your answer. I know how to stop the job. I usually stop the job like you show. In my case, After I start the job, it keeps the state "starting..." I mean the job doesn't start properly, it doesn't run and I can't stop the job. The job is stuck in the "starting..." state for more than 1 day. i.stack.imgur.com/LzUGQ.png
    – tsbm
    Feb 10, 2021 at 14:39
  • @tsbm So your jobs doesn't start and get stuck in "pending" state forever. right?
    – Jin Lee
    Feb 11, 2021 at 5:43
  • @tsbm I hope Google will take care of your problems soon. Good luck!
    – Jin Lee
    Feb 11, 2021 at 5:49
  • Yes, jobs get stack in “pending” state forever. I’m sorry for my English. Thank you.
    – tsbm
    Feb 11, 2021 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.