Questions tagged [kedro]

Kedro is an open source Python library that helps you build production-ready data and analytics pipelines

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

Login time out error for pyodbc connection string in kedro

I'm trying to build up connection string in kedro for SQLQueryDataset to set the connection with MSSQL and Pyodbc in Azure Databricks but encountered with an error: (pyodbc.OperationalError) ('HYT00',...
  • 1
0 votes
1 answer
16 views

Interpolate sql in SQLDataset in catalog.yml

Is there a way to interpolate a SQLDataset query in catalog.yml passing some argument\parameter. Example: person: type: pandas.SQLQueryDataSet sql: "SELECT * FROM public.people WHERE id = ${...
  • 590
0 votes
0 answers
16 views

Adding multiple data catalog into one in kedro

I am dynamically creating pipeline, nodes and catalog for a project. After creating the them, I want to add pipelines and the catalogs. Adding the pipelines is possible by sum function, but it's not ...
0 votes
0 answers
36 views

Colorful notebook output with rich library

May someone tell me how to set again that colorful output from jupyter notebook without using rich.print? I use VSCode. I've got this feature with kedro=0.18.4 and lost with kedro=0.18.5. Kedro ...
  • 58
0 votes
1 answer
66 views

Parametrize input datasets in kedro

I'm trying to move my project into a kedro pipeline but I'm struggling with the following step: my prediction pipeline is being run by a scheduler. The scheduler supplies all the necessary parameters (...
  • 322
0 votes
0 answers
53 views

kedro dynamic catalog creation only for specific nodes before their run

I have several thousands of files to be processed of the different types. I am using dynamic catalog creation with hooks. I used first after_catalog_created hook but it is too early in and I need ...
  • 99
2 votes
0 answers
42 views

Kedro catalog fails when overwriting a GeoJson dataset even though the driver is supported

I have the following catalog item in my kedro project suggested_routes_table@geopandas: type: geopandas.GeoJSONDataSet filepath: data/04_feature/routes_suggestions_table.geojson load_args: driver: &...
  • 21
0 votes
1 answer
44 views

Does kedro data catalog accept .arrow files?

While using Kedro I want to load some data and work with it. To do that, one has to register the data in a conf/base/catalog.yml file. The Kedro Documentation of the Data Catalog explains how one can ...
1 vote
1 answer
1k views

Python: kedro viz SQLAlchemy DeprecationWarning

I tried to work with kedro and started with the spaceflight tutorial. I installed the src/requirements.txt in a .venv. When running kedro viz (or kedro run or even kedro --version), I get lets of ...
  • 135
0 votes
1 answer
59 views

kedro jupyter notebook in command prompt returns kedro.framework.cli.jupyter.single kernelspec manager' could not be imported"

I have been trying to activate jupyter notebooks in a kedro context for over 24 hours now and I receive the same error all the time. I have searched around and no one seems to be able to solve this ...
  • 1
0 votes
1 answer
103 views

Kedro (Python) DeprecationWarning: `np.bool8`

When I try to create a new kedro project or run an existing one, I get the following deprecation warning (see also screenshot below). As far as I understand the warning is neglebile, however, as I am ...
  • 135
0 votes
0 answers
33 views

Kedro register dataset from a board from pins package

For my project I want to use a combination of kedro for the pipeline orchestration and pins for data and model versioning. I have some data which I stored on a board from the pins package. As I have ...
  • 137
0 votes
2 answers
238 views

How to use Kedro with Great-expectations?

I am using Kedro to create a pipeline for ETL purposes and column specific validations are being done using Great-Expectations. There is a hooks.py file listed in Kedro documentation here. This hook ...
0 votes
0 answers
63 views

Kedro, running inference on user input

I have a pipeline with the model I want to use. Outside of the project, I have an app.py file where I'm going to create the UI/UX for my users to run my model. Right now I'm just using a sample string ...
0 votes
0 answers
53 views

how to call kedro pipline or nodes in Django framework

I have to call Kedro nodes or pipelines in Django API, and need to use Kedro pipelines or nodes output as input in Django API. Not getting any solution. Please suggest some solution for this. How to ...
0 votes
0 answers
25 views

Integrate kedro debugger with docker container

I'm currently trying to set up the debugger for kedro in VScode adding the following launch.json as the documentation suggests. The thing is, I have 2 different docker containers that I would like to ...
0 votes
1 answer
31 views

Grouping raw datasets in a kedro visualization

I am looking for a way to group all of the raw datasets in a kedro pipeline visualization into one collapsible/expandable "node", similar to the way that namespaces are collapsible/...
0 votes
2 answers
187 views

Kedro - Getting path to item in the datacatalog

I'm training an nlp model using spacy. I have the preprocessing steps all written as a pipeline, and now I need to do the training. According to spacy's documentation I need to run the following ...
0 votes
1 answer
254 views

How can I register kedro data catalog programmatically in Kedro 0.18?

For various reasons (mainly ability to dynamically construct file paths) I like to define the data catalog programatically, and not use yaml file to define datasets e.g. DataCatalog( {"...
1 vote
1 answer
196 views

Kedro - How to update a dataset in a Kedro pipeline given that a dataset cannot be both input and output of a node (only DAG)?

In a Kedro project, I have a dataset in catalog.yml that I need to increment by adding a few lines each time I call my pipeline. #catalog.yml my_main_dataset: type: pandas.SQLTableDataSet ...
0 votes
1 answer
29 views

Can't run KedroSession with `from_inputs` parmeter: ValueError: Pipeline does not contain data_sets named [...]

In jupyter notebook, when I run session.run(pipeline_name='sim', from_inputs=['measurements', 'params:simulation']), passing datasets & params specified in catalog.yaml, everything works fine. ...
  • 79
0 votes
0 answers
62 views

How to generate kedro pipelines automatically (like DataEngineerOne does)?

Having seen the video of DataEngineerOne: How To Use a Parameter Range to Generate Pipelines Automatically I want to automate a pipeline that simulates an electronic circuit. I want to do a grid ...
  • 79
1 vote
2 answers
132 views

How to show plotly chart in kedro

I am trying to use data science tool kedro according to this tutorial. I followed the instruction(write config.yaml, node.py and pipeline.py etc) and do exactly the same as the documentation) and ...
0 votes
2 answers
57 views

How can I run `catalog.load` in a non-IPython context?

In IPython I can run data = catalog.load('my_dataset') in order to load a dataset specified as 'my_dataset' in the catalog.yml file. What's the equivalent command in a pthon script? What do I need to ...
  • 79
0 votes
1 answer
115 views

Include Quarto rendering in kedro pipeline and pass it inputs/outputs

I am using kedro to make some comparative analysis. I am using the quarto python package providing a wrapper to the quarto cli through the render function. This function will take a qmd file as input ...
  • 1,335
0 votes
2 answers
132 views

kedro PartitionedDataSet lazy saving to spare memory?

I am working with PartionedDataSet in kedro. One of the data set is of type pillow.ImageDataSet: raw_images: type: PartitionedDataSet <<: *data_path_on_disk dataset: ...
  • 1,335
0 votes
0 answers
36 views

Kedro template configuration does not load globals.yml configuration into catalog.yml FOR Jupyter Lab

It works for the CLI ( kedro run ... ) but not for Jupyter Lab. I have just recently upgraded from 0.17.1 to 0.18.3. Have made changes to settings.py which uses the Templated Config Loader ( just like ...
0 votes
1 answer
120 views

Kedro on Databricks: Cannot import SparkDataset

Cannot import SparkDataset in Databricks using; from kedro.extras.datasets.spark import SparkDataSet
0 votes
1 answer
548 views

import fsspec throws error (AttributeError: 'EntryPoints' object has no attribute 'get')

import fsspec throws error (AttributeError: 'EntryPoints' object has no attribute 'get') []
1 vote
0 answers
124 views

Is there a way to include an Azure Databricks Lakehouse query as a DataCatalog dataset in kedro?

We want to use kedro to control our ML pipelines in Azure Databricks. We are querying (and joining) relatively large tables in Databricks' Lakehouse. Therefore, we would like to include those joins in ...
  • 1,478
0 votes
1 answer
181 views

How to change the kedro configuration environment in jupyter notebook?

I want to run a kedro pipeline in the base env using jupyter notebook. I do this the following way: %reload_kedro --env=base session.run(pipeline_name='dpfm1') Doing this, the %reload_kedro command ...
  • 79
0 votes
3 answers
439 views

How to run a kedro pipeline interactively like a function

I would like to run kedro pipelines in jupyter notebook with different inputs, so something like this: data = catalog.load('my_dataset') params = catalog.load('params:my_params') pipelines['...
  • 79
0 votes
2 answers
107 views

DataSetError in Docker Kedro deployment

I try to deploy example Kedro starter project (pandas-iris). I successfuly run it locally (kedro run), and then, having kedro-docker install, init a Docker, build image and push it to my registry. ...
0 votes
0 answers
44 views

How to do SQL like querying the parquet files in kedro

I'm new to kedro, I'm just wondering if I could do SQL like querying the parquet files instead of using Dataframe API's. Please help me out if there is a way. Thanks in advance!
  • 5,927
1 vote
1 answer
129 views

How to use generators with kedro?

Thanks to David Beazley's slides on Generators I'm quite taken with using generators for data processing in order to keep memory consumption minimal. Now I'm working on my first kedro project, and my ...
  • 79
0 votes
2 answers
383 views

dynamic parameters on datasets in Kedro

I would like to call an API to enrich an existing dataset. The existing dataset is a CSVDataSet configured in the catalog. Now I would like to create a Node, that enriches the CSVDataSet with data ...
  • 683
0 votes
1 answer
442 views

ModuleNotFoundError: No module named 'kedro.versioning'

i have upgraded my kedro to latest version.But i have used kedro.versioning in my project.And latest kedro has no module of this name.Can anyone please suggest anything
  • 1
0 votes
0 answers
140 views

how to use kedro.versioning in latest version of kedro?

I have previously used kedro version 0.17.6 in my project. Now i have upgraded my version to 0.18.2. But in latest version of kedro there is no module of kedro.versioning. So i am getting a error that ...
  • 1
0 votes
1 answer
271 views

Kedro template configuration does not load/parse variables

A follow up to this question. I am using Kedro v0.18.2. I am trying use the TemplateConfig so I have created a globals.yml under conf/base, which looks like this: paths: base_path: s3://...
1 vote
2 answers
365 views

Kedro template configuration does not load globals.yml configuration into catalog.yml

I am using Kedro to create some data pipelines, for which I have various file formats, as well as a custom one. According to the documentation, I have created a globals.yml under config/base where I ...
0 votes
1 answer
162 views

Looking for the right way to make a kedro node output lazily two partitioned datasets

I've built a node in Kedro loading lazily an input partitioned dataset, and saving lazily two partitioned datasets as an output (following recommendations found in the Kedro community: using a lambda +...
2 votes
1 answer
353 views

Load existing data catalog programmatically

I want to write pytest unit test in Kedro 0.17.5. They need to perform integrity checks on dataframes created by the pipeline. These dataframes are specified in the catalog.yml and already persisted ...
0 votes
2 answers
133 views

Access configuration in the pipelines definition (not only nodes)

I want to dynamically alter pipelines based on the configuration provided. Is there possibility to pass configuration based on the environment to the register_pipelines() or to the create_pipeline() ...
  • 325
0 votes
1 answer
57 views

Rate limiting Kedro API requests

I have a few datasets from the government dataset that I'm using on my ML model, the problem is, their server is not that great to put it nicely. Whenever I run my pipeline, when I pull from their API ...
0 votes
2 answers
373 views

Kedro 0.16.3 and kedro[spark.SparkDataSet] pip libraries cannot be installed together on databricks cluster

Till last week both kedro and kedro[spark.SparkDataSet] pip libraries were installed on the cluster. But since last 3-4 days they wont be installed together on the cluster. It shows that its a ...
  • 1
0 votes
3 answers
209 views

Kedro using wrong conda environment

I have created a conda environment called Foo. After activating this environment I installed Kedro with pip, since conda was giving me a conflict. Even though I'm inside the Foo environment, when I ...
0 votes
2 answers
92 views

Getting Kedro Custom Dataset for SunPy Maps to write to/from S3

I'm currently attempting to define a custom dataset to read/write .fits files to/from S3 as SunPy Maps. The closest thing to this already in the data catalog is the pillow.ImageDataSet pillow....
0 votes
1 answer
149 views

How to pull data from a paginated JSON API using kedro (APIDataSet)?

The problem: I would like to retrieve data from a paginated API that sends JSON responses. Using kedro.extras.datasets.api.APIDataSet I can query the API and retrieve the initial response. However if ...
1 vote
2 answers
179 views

How to write to HDFS using kedro

I'm trying to output of my Kedro pipeline to the HDFS file system. But I couldn't see on the internet how to do that and on Kedro documents. If anybody configured kedro in catalog please share a ...
0 votes
0 answers
272 views

How to inject the information about load version into Kedro node?

I need to run a Kedro (v0.17.4) pipeline with a node that is supposed to process data with a different logic depending on the load version of the input. As a simple and crude example assuming there is ...
  • 33