I work on partitioned data (partitioned parquet or SQL table with a "partition" column). I want Kedro to load and save data from a partition I provide at runtime (e.g. kedro run --params partition:A). The number of partitions is large and dynamic.
I use Spark. Is there a way to load/save data the way I need with SparkDataSet or SparkJDBCDataSet?