0

I work on partitioned data (partitioned parquet or SQL table with a "partition" column). I want Kedro to load and save data from a partition I provide at runtime (e.g. kedro run --params partition:A). The number of partitions is large and dynamic.

I use Spark. Is there a way to load/save data the way I need with SparkDataSet or SparkJDBCDataSet?

1 Answer 1

0

A quick google suggests the Spark JDBCDriver can use a timestamp column for partitioning . All Kedro does behind the scenes is pass the catlaog load_args and save_args to the native driver so this may work.

One another way to use a lifecycle hook like before_pipeline_run, inspect the run parameters and then inject some custom logic at that point as you're able to inspect the --params run arguments easily at that point.

A last thought - if you subclass and extend the SQL dataset you want to use you can easily extend it to partition the way you want it. You won't easily be able to pass run --params but it would be easy to retrieve env variables or custom catalog arguments.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.