0

Hi I am trying to apply prefect to my project which is using the library click in dealing with command-line paras. Below is a demo code snippet:

@click.command()
@click.option(
    "-p",
    "--pages",
    type=int,
    default=0,
    help="...",
)
def main(pages):
    print("Running...")
    if pages > 0:
      a()
    else:
      b()
    print("Finished without errors.")

if __name__ == "__main__":
    main()
    another_method()

The doc of prefect mentioned about the example :

flow = Flow("hello-flow", tasks=[hello_task])
flow.register(project_name="tester")

But what if I need to run the program by let say poetry run main.py -p 10, where I need to give a fixed command-line para and also run with poetry. In that case how should I organize or refactor my code to fit with Prefect?

1 Answer 1

3

In general, with Prefect you don't have to use any of this (neither click nor poetry) in order to be able to run your flow from CLI because Prefect ships with its own CLI. In order to start your flow run from a CLI, you can use:

prefect run -p /path/to/your/flow_file.py

Let's assume you have a flow that looks as follows:

from prefect import task, Flow, Parameter

@task(log_stdout=True)
def hello_world(name):
    print(f"Hello {name}!")

with Flow("mini-example") as flow:
    name = Parameter("name", default="world")
    hw = hello_world(name)

If you want to run it locally with a different parameter value than "world", you can use --param option:

prefect run -p /path/to/your/flow_file.py --param name=Marvin

And then, once you are ready to deploy your project to your Prefect backend, you can register it:

prefect register --project yourprojectname -p /path/to/your/flow_file.py

And then you can even trigger a remote flow run that will be stored in a backend (i.e. Prefect Cloud or Server) using:

prefect run --name mini-example --project yourprojectname
4
  • Many thanks to the suggestion! One more thing is : Apart from being hosted by prefect cloud sever, the program might be run by normal local command such as python a.py -p 10. In this case is it the best practice to have the both CLI interfaces: the click implementation and the Prefect Parameter?
    – chaos
    Dec 13, 2021 at 18:00
  • The easiest would be to use only the Prefect CLI to run any flows locally without using click. Additionally, adding click commands makes it harder to deploy your flow to the backend (Server/Cloud) because you would have to remove those click commands before deploying (i.e. registering) your flow to the backend. The CLI is beneficial because it allows that your flow file specifies ONLY your flow structure: your tasks and flow's metadata, nothing else. Then, the CLI provides commands to run or register the flow with no changes to your flow. Without CLI, you would need to call flow.run() Dec 14, 2021 at 0:42
  • Hi, thanks for the clarification. But I am still confused by what you said about with Prefect you don't have to use any of this (neither click nor poetry). Since right now I run the program by poetry run main.py, where the dependencies are resolved by poetry in this case. If I run with prefect run -p xx.py, it would raise dependency errors. In my understanding if my project is supported by poetry then I have to run the program with poetry isn't? Or how can I bypass this and run with prefect run ...
    – chaos
    Dec 14, 2021 at 11:41
  • I'm not a poetry user so hard to tell. Usually, when you use venv or conda for a virtual environment, you just activate your virtual env in which you have prefect installed and you can use Prefect CLI: prefect run... Dec 14, 2021 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.