Questions tagged [kedro]
Kedro is an open source Python library that helps you build production-ready data and analytics pipelines
140
questions
0
votes
3
answers
47
views
Kedro using wrong conda environment
I have created a conda environment called Foo. After activating this environment I installed Kedro with pip, since conda was giving me a conflict. Even though I'm inside the Foo environment, when I ...
0
votes
1
answer
31
views
Kedro SunPy - Writing Custom Data Set to S3
I'm currently attempting to define a custom dataset to read/write .fits files to/from S3 as SunPy Maps.
The closest thing to this already in the data catalog is the pillow.ImageDataSet pillow....
0
votes
1
answer
53
views
How to pull data from a paginated JSON API using kedro (APIDataSet)?
The problem: I would like to retrieve data from a paginated API that sends JSON responses.
Using kedro.extras.datasets.api.APIDataSet I can query the API and retrieve the initial response. However if ...
1
vote
2
answers
44
views
How to write to HDFS using kedro
I'm trying to output of my Kedro pipeline to the HDFS file system. But I couldn't see on the internet how to do that and on Kedro documents. If anybody configured kedro in catalog please share a ...
0
votes
0
answers
36
views
How to inject the information about load version into Kedro node?
I need to run a Kedro (v0.17.4) pipeline with a node that is supposed to process data with a different logic depending on the load version of the input.
As a simple and crude example assuming there is ...
0
votes
1
answer
36
views
Setup a base dir for the Data Catalog in Kedro
I'm working on a project that, because of the company's compliance rules, the data has to stay in a shared directory, that is synchronized among the programmers. The project's code on the other hand ...
0
votes
1
answer
38
views
Mono repo Kedro project
I started a Kedro project a while ago and started to build different parts of the pipeline which only tangentially interact with each other. In some cases not much at all.
As a consequence, as the ...
0
votes
1
answer
28
views
Versioned Datasets in Kedro
Situation:
I have monthly snapshots that should look like this
snapshot-2021-10.parquet
snapshot-2021-11.parquet
snapshot-2021-12.parquet
snapshot-2022-01.parquet
snapshot-2022-02.parquet
In the ...
0
votes
0
answers
21
views
VersionNotFoundError when writing S3 bucket from a Custom Dataset
I have a custom dataset which I am using to write a popmon report to S3 bucket
class ReportDataset(AbstractVersionedDataSet):
def __init__(self, filepath: str, version: Version = None, credentials: ...
0
votes
0
answers
61
views
Kedro documentation does not show up all functions after compiling kedro build-docs
I tried documenting my project using kedro build-docs command.
.
├── docs
└── src
├── setup.py
├── tests
├── <project>
│ ├── __init__.py
│ ├── __main__.py
│ ├── ...
1
vote
1
answer
37
views
jsonschema 4.4.0 does not provide the extra 'isoduration'
So I'm trying to run some piece of code and keep getting the following error:
File "/opt/conda/lib/python3.8/site-packages/pkg_resources/__init__.py", line 770, in resolve
raise ...
0
votes
3
answers
634
views
Python Kedro PySpark : py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext
it's my first project using kedro with Pyspark and I have an issue. I work with the new Mac (M1). When I do spark-shell in the terminal, spark is successfully installed and I have the right output (...
0
votes
1
answer
29
views
How to plot on kedro mlflow ui x1=array/list/dict and y1=array/list/dict?
I am new to kedro, and I don't know if I am asking the right question here.
Is it possible on kedro mlflow ui to plot x and y lists?
I am running kedro pipeline with mlflow. I have catalog.yaml which ...
-1
votes
1
answer
57
views
Pickling a file
I using kedro in conjunction with databricks-connect to run my machine learning model. I am trained and tested the model using databricks notebook and saved the pickle file of the model to azure blob ...
0
votes
1
answer
54
views
Protobuf compatibility error when running Kedro pipeline
I have a Kedro pipeline that I want to run through a Python script, I think I have the minimum necessary code to do this, but everytime I try to run the pipeline through the script, I get a ...
0
votes
2
answers
51
views
how do I run a local script using github actions
Hello I am using kedro (a pipeline tool) and want to use github actions to trigger a kedro command (kedro run) whenever I make a push to my github repo.
Since I have all the data in my local repo, I ...
0
votes
1
answer
22
views
How to fetch complex MongoDB Data from Kedro?
I'm attempting to get hands on Kedro, but don't understand how to build my Data Fetcher (that I used before).
My Data is stored in a MongoDB instance over multiple “Tables”. One table are my usernames....
0
votes
1
answer
109
views
How to save keras model in kedro
I am able to save DNN Model in h5 format on s3. but when I import it in inference pipeline of kedro tool, I am getting blank?no predictions.
I made following changes in catalog.yml file:
model:
...
0
votes
0
answers
69
views
How to install new package with conda without breaking the existing Kedro installation?
I have a working conda environment with Kedro installed. The .yml file is available by the link 1. My kedro pipelines work fine in this environment. However, when I try to install matplotlib package ...
1
vote
1
answer
133
views
Facing kedro-airflow validation error from pydantic when running in docker
I am new to kedro and airflow. I am trying to deploy a kedro pipeline in airflow by using docker.
But while executing my DAG I get this error:
2022-01-27 16:17:19,659 - airflow.task - ERROR - Task ...
0
votes
1
answer
120
views
Kedro pipeline on partitioned data
I work on partitioned data (partitioned parquet or SQL table with a "partition" column). I want Kedro to load and save data from a partition I provide at runtime (e.g. kedro run --params ...
0
votes
2
answers
107
views
How to save kedro dataset in azure and still have it in memory
I want to save Kedro memory dataset in azure as a file and still want to have it in memory as my pipeline will be using this later in the pipeline. Is this possible in Kedro. I tried to look at ...
3
votes
2
answers
367
views
AttributeError: Object ParquetDataSet cannot be loaded from kedro.extras.datasets.pandas
I'm quite new using Kedro and after installing kedro in my conda environment, I'm getting the following error when trying to list my catalog:
Command performed: kedro catalog list
Error:
kedro.io....
0
votes
1
answer
49
views
Why doesn't my Kedro starter prompt for input?
I would like to create my own Kedro starter. I have tried to replicate the relevant portions of the pandas iris starter. I have a cookiecutter.json file with what I believe are appropriate mappings, ...
1
vote
2
answers
159
views
Is there a package in R that mimics KEDRO as a modular collaborative framework for development?
I currently work with Kedro (from quantum black https://kedro.readthedocs.io/en/stable/01_introduction/01_introduction.html) as a framework for deployment oriented framework to code collaboratively. ...
1
vote
1
answer
343
views
How to access environment name in kedro pipeline
Is there any way to access the kedro pipeline environment name? Actually below is my problem.
I am loading the config paths as below
conf_paths = ["conf/base", "conf/local"]
...
1
vote
0
answers
178
views
Use an Azure ML compute cluster to run Kedro + Mlflow pipeline
I want to use an Azure Machine Learning compute cluster as a compute target to run a Kedro pipeline integrated with Mlflow.
Here's the code snippet (hooks.py) that integrates experiment tracking using ...
0
votes
2
answers
94
views
Kedro run pointing to a previously used Azure Data Lake
I'm trying to read from / write another one of my ADLS Gen2 storage accounts. Until now, it worked perfectly with an old one.
I updated the credentials.yml with the new account name and key but it ...
1
vote
1
answer
214
views
kedro DataSetError while loading PartitionedDataSet
I am using PartitionedDataSet to load multiple csv files from azure blob storage. I defined my data set in the datacatalog as below.
my_partitioned_data_set:
type: PartitionedDataSet
...
0
votes
1
answer
154
views
how to make a kedro pipeline take configurable input dataframes?
I have created a workflow in kedro made of different data science processing pipelines. These pipelines are tested independently.
When i run a particular kedro pipeline in stand alone fashion, the ...
0
votes
1
answer
92
views
azure datasource throwing error in Kedro datacatalog
I am facing error when configuring azure blob storage dataset in kedro datacatalog.
I have the dataset defined in my catalog.yml as below:
brand_dataset:
type: pandas.CSVDataSet
filepath: "...
-1
votes
2
answers
145
views
How do I add multiple csv files to the catalog in kedro
I have 4 csv files in Azure blob storage, with same metadata, that i want to process. How can i add them to the datacatalog with a single name in Kedro.
I checked this question
https://stackoverflow....
0
votes
1
answer
367
views
kedro context and catalog missing from ipython session
I launched ipython session and trying to load a dataset.
I am running
df = catalog.load("test_dataset")
Facing the below error
NameError: name 'catalog' is not defined
I also tried %...
0
votes
0
answers
204
views
Run experiments on Azure ML with Kedro and Mlflow
I'm trying to run the whole Kedro pipeline as an Azure ML experiment. I had two options here. The first one was to use the built-in logging feature of Azure ML and the second one was to use the ...
0
votes
1
answer
94
views
Logging the git_sha as a parameter on Mlflow using Kedro hooks
I would like to log the git_sha parameter on Mlflow as shown in the documentation. What appears to me here, is that simply running the following portion of code should be enough to get git_sha logged ...
0
votes
1
answer
192
views
Azure Data Lake Storage Gen2 (ADLS Gen2) as a data source for Kedro pipeline
According to Kedro's documentation, Azure Blob Storage is one of the available data sources. Does this extend to ADLS Gen2 ?
Haven't tried Kedro yet, but before I invest some time on it, I wanted to ...
-1
votes
1
answer
498
views
Adding parameters in Kedro Pipeline
I am trying to write test cases for Kedro pipeline. I have params:lr as input for my model training node. Its not being loaded from the parameters of the training pipeline, nor from parameters.yml.
...
2
votes
2
answers
193
views
Kedro - Memory management
I am working on a Kedro 0.17.2 project that is running on out-of-memory issues and I'm trying to reduce the memory footprint.
I'm doing the profiling by using mprof from the memory-profiler library ...
1
vote
0
answers
234
views
Kedro cannot find run
As a part of upgrading Kedro from 0.16.2 to 0.17.3 in our organization, I've made changes to Kedro related files in our codebase based on Kedro starter pyspark-iris on 0.17.3.
Now I get an error of ...
0
votes
1
answer
81
views
Want to run Specific node or group of nodes and capture the output into a variable in kedro jupyter lab
I am new to kedro, I am trying to run Spaceflights tutorial. I want to run the complete data_processing_pipeline 'dp', and capture the output in a dataframe. I am running it on Jupyter Lab. I used the ...
0
votes
1
answer
191
views
How to dynamically pass save_args to kedro catalog?
I'm trying to write delta tables in Kedro. Changing file format to delta makes the write as delta tables with mode as overwrite.
Previously, a node in the raw layer (meta_reload) creates a dataset ...
2
votes
1
answer
272
views
Running a kedro pipeline with inputs and outputs defined through the command line
I would like to run a kedro pipeline using different inputs and saving the results in an output folder where inputs paths and outputs paths are provided through the command line
I sow the possibility ...
0
votes
1
answer
120
views
How to let kedro execute nodes in sequence
I am trying to use kedro to run a workflow. Following figure is my workflow(node 1-3 is sequential and nodes 31, 32 and 33 is three branches which from node 3). You can see the kedro is running ...
0
votes
1
answer
218
views
Saving data with DataCatalog
I was looking at iris project example provided by kedro. Apart from logging the accuracy I also wanted to save the predictions and test_y as a csv.
This is the example node provided by kedro.
def ...
0
votes
1
answer
189
views
Building an autoencoder with Keras and Kedro
I'm trying to build an autoencoder, which I'm sure I'm doing something wrong. I tried separating the creation of the model from the actual training but this is not really working out for me and is ...
1
vote
2
answers
173
views
How to read/write/sync data on cloud with Kedro
In short: how can I save a file both locally AND on cloud, similarly how to set to read from local.
Longer description: There are two scenario, 1) building model 2) serving model through API. In ...
2
votes
1
answer
168
views
Waiting for nodes to finish in Kedro
I have a pipeline in Kedro that looks like this:
from kedro.pipeline import Pipeline, node
from .nodes import *
def foo():
return Pipeline([
node(a, inputs=["train_x", "...
1
vote
1
answer
344
views
Kedro: ValueError: Pipeline does not contain nodes named ['preprocess_companies_node']
Similar to the question described earlier, I followed the spaceflights tutorial, at create pipeline step, I got the following error when running kedro run --node=preproces_companies_node
ValueError: ...
0
votes
1
answer
177
views
How to use SQL Server Bulk Insert in Kedro Node?
I am managing a data pipeline using Kedro and at the last step I have a huge csv file stored in a S3 bucket and I need to load it back to SQL Server.
I'd normally go about that with a bulk insert, but ...
0
votes
1
answer
88
views
Kedro can not find SQL Server table
I have these two datasets defined:
flp_test_query:
type: pandas.SQLQueryDataSet
credentials: dw_dev_credentials
sql: select numero from dwdb.dwschema.flp_tst
load_args:
index_col: [numero]
...