39

My pipelines and schedulers were running smoothly without any problems. After I went out to lunch, I changed the number of epochs a Neural Network would run, save the .yaml file again and leave it in the bucket named "budgetff".

Afterwards, everything stopped working.

No clue

enter image description here

There are the errors and I have 0 clue as to how they are appearing. The code within the components doesn't even seem to start. I've made several different components without any success because they just fail at this step.

If it helps, I installed the kfp --pre and did the imports like this

import kfp.v2.dsl, kfp.v2.compiler
from kfp.v2.dsl import Artifact, Dataset, Input, Metrics, Model, Output

kfp-2.0.0-beta.15 - This is the kfp version running on VertexAi and I'm using Kubeflow with @kfp.v2.dsl.components.

I was trying to just run my pipelines. Forcing a run on the scheduler. When it didn't work, I just tried on the notebook.

9 Answers 9

39

The cause is that the latest version of requests does not support urllib3 2.0.0. This is fixed in kfp-2.0.0b16 (see PR with the change), so you can either upgrade to that, or create a new image that downgrades urllib.

Maybe this is triggered by versions of requests-toolbelt and/or urllib3 that were both released in the past few days (May 1 and May 4, 2023, resp).

I have fixed this by building a new container with the following Dockerfile (I use Python 3.9 but use whatever you want):

FROM python:3.9

RUN pip install urllib3==1.26.15 requests-toolbelt==0.10.1

I recommend to build the image using Cloud Build and specify it as the base image for the component.

2
  • This works, I'm running python3.11. I was forced to downgrade my urllib3 from 2.0.7 to 1.26.15 and include requests-toolbelt==0.10.1 for this to work.
    – ICeZer0
    Commented Dec 21, 2023 at 23:41
  • Now, on 05/21/2024, pip install --upgrade requests-toolbelt works by updating requests-toolbelt from 0.10.1 to 1.0.0. (While pip install --upgrade requests does NOT work!) Maybe you could update the answer?
    – Weekend
    Commented May 21 at 8:27
17

Unrelated to OP's specific issue, but outdated versions of twine are also subject to this same error. pip install --upgrade twine won't fix it, but this will:

pip install --upgrade twine requests-toolbelt
1
  • It works by updating requests-toolbelt from 0.10.1 to 1.0.0 , thanks
    – Weekend
    Commented May 21 at 8:18
14

I ran into the same issue today and was scratching my head. I found adding appengine-python-standard to the packages_to_install argument in the component decorator solved the issue:

@component(base_image="python:3.7", packages_to_install=["appengine-python-standard",...])

For context, I'm using KFP v1.8.20 through Vertex workbench. Fingers crossed, it works for v2.0.0-beta.15.

2
  • 3
    This is the simplest fix now, works for python 3.7 and 3.9
    – pajamas
    Commented May 5, 2023 at 6:52
  • 1
    Thank you so much. This works great if you base a Docker image on python:3.7.16-bullseye, which installs requests-toolbelt-0.9.1 when asking for requests, which in turn produces the appengine dependency error. Just install 'appengine-python-standard' before you install requests and you should be fine. Commented May 6, 2023 at 15:34
7

These issues were fixed in kfp==1.8.21 and kfp==2.0.0b16.

0
3

In my case it has worked when in my dependencies I've updated requests-toolbelt from version 0.10.1 to version 1.0.0.

2

I had the same problem. I updated the pipeline component to use a more recent Python version and now it works:

@component(
    base_image="python:3.11",
2

You should use kfp==1.8.21 ref. https://github.com/kubeflow/pipelines/pull/9323

1

Used a base_image="python:3.11 and all is working now. Thank you so much for the valuable feedback and help! Have a great day

Edit: Was having a lot of compatibility issues, decided to go with the first suggestion of adding appengine-python-standard on the packages to install :)

2
  • 1
    Good to hear! Would be great if you could mark my answer as the accepted one :)
    – mwtmurphy
    Commented May 6, 2023 at 19:19
  • Please mark the answer that helped you out as accepted. Also, please put any edits / updates in the original question, so people can see what changed more easily. Commented May 9, 2023 at 22:11
0

This happened to me in my GitHub Action and the problem was that I was using an old version of poetry. Using poetry 1.4.2 fixed it.

Source of an issue with the same root problem: https://github.com/ionrock/cachecontrol/issues/292#issuecomment-1536120527

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.