Questions tagged [kedro]

Kedro is an open source Python library that helps you build production-ready data and analytics pipelines

Filter by
Sorted by
Tagged with
1 vote
1 answer
25 views

Your Kedro project version 0.17.6 does not match Kedro package version

when running kedro ipython into a cloned repo, I get: _version_mismatch_error(metadata_dict["kedro_init_version"])) ValueError: Your Kedro project version 0.17.6 does not match Kedro package ...
gaut's user avatar
  • 5,825
1 vote
1 answer
35 views

KEDRO - How to specify an arbitrary binary file in catalog.yml?

I'm currently working on a datascience project using LLMs (Large language models). Weights for models usually come in different formats, most frequently .bin or .gguf, and I'd like to keep it that way....
Quakumei's user avatar
1 vote
1 answer
72 views

Kedro viz blank page

I created a sample pipeline that works correctly with kedro run, however when I try to visualise it with kedro viz I'm getting basically a blank page, even though the terminal doesn't show a single ...
Fatafim's user avatar
  • 297
1 vote
2 answers
62 views

Kedro simple project run error: ModuleNotFoundError followed by ValueError

Full code available on: https://github.com/braKhAbid/kedro_mnist I am trying to setup a simple python ML project in Kedro. It is very simple, one pipeline of three nodes, a data loading node, a model ...
Brahim Khalil Abid's user avatar
0 votes
1 answer
6 views

How to import Custom json encoder class into data cataloge

I’ve a df which stores lists in a column. I am saving the df with all columns in json using config_new:   type: json.JSONDataSet   filepath: data/01_raw/new_config.json   save_args:     indent: 6   It’...
SSR's user avatar
  • 1
1 vote
1 answer
74 views

in kedro / pyspark how to use MemoryDataset

I am trying to use a MemoryDataset with kedro, in order to not save the intermeiate result to disk. # nodes.py def preprocess_format_tracksessions(tracksess: DataFrame, userid_profiles:pd.DataFrame , ...
gaut's user avatar
  • 5,825
0 votes
1 answer
37 views

Define column names when reading a spark dataset in kedro

With kedro, how can I define the column names when reading a spark.SparkDataSet? below my catalog.yaml. user-playlists: type: spark.SparkDataSet file_format: csv filepath: data/01_raw/lastfm-...
gaut's user avatar
  • 5,825
0 votes
1 answer
33 views

in Kedro, how to handle tar.gz archives from the web

I have a tar.gz file that I am downloading from this link: http://ocelma.net/MusicRecommendationDataset/lastfm-1K.html What is the best way to fully integrate this TSV data into kedro, perhaps with an ...
gaut's user avatar
  • 5,825
1 vote
1 answer
215 views

Getting Kedro globals.yml to work with OmegaConfigLoader

I just started evaluating Kedro for use, and I began with a small project where I read data from a MS-SQL Server. The pipeline will run with a few months in between, and with new date ranges every ...
Magnus Persson's user avatar
2 votes
1 answer
106 views

logging in python and kedro, how to log only DEBUG info to a file and INFO to console

I'm trying to configure logging so that INFO level messages go to the console and DEBUG level messages go to a file instead. So far, I am able to get working INFO to console and DEBUG to file, the ...
Emilio's user avatar
  • 33
0 votes
0 answers
68 views

how to create kedro catalog entry + custom DataSet init method with values from credentials.yml

I'm trying to create a custom DataSet class within the kedro framework. I need some help understanding how to combine values from the credentials.yml file. what is the kedro way of handling the '...
Emilio's user avatar
  • 33
1 vote
1 answer
202 views

how do kedro catalog properties get passed to Datasets when pipeline is executed

I'm trying to create two custom kedro DataSets and I'm not quite sure how to configure the __init__ methods. The first DataSet is simple, its a readonly, fetch rss items from a static URL. catalog ...
Emilio's user avatar
  • 33
0 votes
2 answers
92 views

How can I find kedro_cli.py in my project?

I installed Kedro using pip install, and I made a new project using kedro new. I was able to run (kedro run --pipeline myPipelineName) from terminal in PyCharm or from cmd when I make cd to the ...
sama acm's user avatar
3 votes
1 answer
280 views

Is there a way to have files in the Kedro Catalog, that are missing?

I have a kedro pipeline which generates a file that is used again for the next run of that same pipeline. However, when the pipeline runs for the first time, that file does not exist, and it is ...
Nandha Kumar's user avatar
1 vote
1 answer
173 views

How to combine multiple similar datasets into one in Kedro dynamically?

I use Kedro to run pipelines for multiple models that each generate a CSV file with the same schema. However, I need to combine each of the generated CSV files into one and do some post processing ...
Nandha Kumar's user avatar
0 votes
1 answer
155 views

Access Kedro MemoryDataSet when running packaged Kedro pipeline in a script

I want to be able to access a MemoryDataSet results dataframe from a kedro pipeline that I've imported into a script after packaging the pipeline into a python package. I have a kedro pipeline written ...
trevinator's user avatar
2 votes
1 answer
103 views

S3FS is not recognised in lambda when using torch

I have a Kedro Pipeline Node that on AWS Lambda that accesses s3. It runs if I'm not using torch but fails with Install s3fs to access S3 if I add torch as a dependency. I have a Kedro Pipeline I want ...
Julius Hetzel's user avatar
1 vote
3 answers
231 views

Unable to Save Data to HDF5 File using fsspec in Python

I'm trying to save data (3D numpy array) to an HDF5 file using fsspec in Python, but I'm encountering issues and I am unable to successfully write the data to the file. The bigger picture is I am ...
twinsten's user avatar
1 vote
1 answer
116 views

Kedro - how to set max_workers when running pipelines with ParallelRunner?

I'm using Kedro version 0.18.7 and python 3.9 in WSL2. I'd like to run nodes of my pipeline in parallel by running the command kedro run --pipeline <pipeline_name> --runner ParallelRunner. ...
Jenny_L's user avatar
  • 13
1 vote
0 answers
91 views

How to call the data object from another namespace in already namespaced kedro pipeline?

I have the following namespaced pipelines in kedro: def create_pipeline(**kwargs) -> Pipeline: pipeline_instance = pipeline( [ node( func=split_data, ...
Andrzej Wodecki's user avatar
0 votes
1 answer
66 views

kedro: Used azure key and scope, then pull data from snowflake via databricks cluster

I'm currently running into a requirement where we are using a scope/key credential to access/grab data from our Snowflake instance, then load it in our blob storage, while using our Databricks cluster....
Christianne Rio Ortega's user avatar
1 vote
2 answers
544 views

Kedro Pipeline contains no nodes after applying all provided filters

ValueError: Pipeline contains no nodes after applying all provided filters src/pipelines/data_processing<br> src/pipelines/data_science<br> src/pipelines/data_project<br> src/...
Eduardo_Eng_Dados's user avatar
2 votes
1 answer
161 views

Where is my kedro output when using the databricks extension in vscode

I am using kedro together with the databricks extension for vscode to access databricks server on Azure. Everything works pretty well but I don't see any output when executing the file locally. The ...
YoniPrv's user avatar
  • 73
0 votes
1 answer
82 views

Login time out error for pyodbc connection string in kedro

I'm trying to build up connection string in kedro for SQLQueryDataset to set the connection with MSSQL and Pyodbc in Azure Databricks but encountered with an error: (pyodbc.OperationalError) ('HYT00',...
saloni's user avatar
  • 21
0 votes
1 answer
33 views

Interpolate sql in SQLDataset in catalog.yml

Is there a way to interpolate a SQLDataset query in catalog.yml passing some argument\parameter. Example: person: type: pandas.SQLQueryDataSet sql: "SELECT * FROM public.people WHERE id = ${...
Nikola's user avatar
  • 620
0 votes
1 answer
49 views

Adding multiple data catalog into one in kedro

I am dynamically creating pipeline, nodes and catalog for a project. After creating the them, I want to add pipelines and the catalogs. Adding the pipelines is possible by sum function, but it's not ...
Raghunandan Pareek's user avatar
0 votes
0 answers
231 views

Colorful notebook output with rich library

May someone tell me how to set again that colorful output from jupyter notebook without using rich.print? I use VSCode. I've got this feature with kedro=0.18.4 and lost with kedro=0.18.5. Kedro ...
matt91t's user avatar
  • 93
0 votes
1 answer
547 views

Parametrize input datasets in kedro

I'm trying to move my project into a kedro pipeline but I'm struggling with the following step: my prediction pipeline is being run by a scheduler. The scheduler supplies all the necessary parameters (...
w_sz's user avatar
  • 332
0 votes
0 answers
269 views

kedro dynamic catalog creation only for specific nodes before their run

I have several thousands of files to be processed of the different types. I am using dynamic catalog creation with hooks. I used first after_catalog_created hook but it is too early in and I need ...
AHR's user avatar
  • 99
2 votes
0 answers
72 views

Kedro catalog fails when overwriting a GeoJson dataset even though the driver is supported

I have the following catalog item in my kedro project suggested_routes_table@geopandas: type: geopandas.GeoJSONDataSet filepath: data/04_feature/routes_suggestions_table.geojson load_args: driver: &...
Natalio's user avatar
  • 31
0 votes
1 answer
88 views

Does kedro data catalog accept .arrow files?

While using Kedro I want to load some data and work with it. To do that, one has to register the data in a conf/base/catalog.yml file. The Kedro Documentation of the Data Catalog explains how one can ...
Jamal Rnjbal's user avatar
1 vote
1 answer
2k views

Python: kedro viz SQLAlchemy DeprecationWarning

I tried to work with kedro and started with the spaceflight tutorial. I installed the src/requirements.txt in a .venv. When running kedro viz (or kedro run or even kedro --version), I get lets of ...
SysRIP's user avatar
  • 179
0 votes
1 answer
165 views

kedro jupyter notebook in command prompt returns kedro.framework.cli.jupyter.single kernelspec manager' could not be imported"

I have been trying to activate jupyter notebooks in a kedro context for over 24 hours now and I receive the same error all the time. I have searched around and no one seems to be able to solve this ...
MDLA's user avatar
  • 1
0 votes
1 answer
299 views

Kedro (Python) DeprecationWarning: `np.bool8`

When I try to create a new kedro project or run an existing one, I get the following deprecation warning (see also screenshot below). As far as I understand the warning is neglebile, however, as I am ...
SysRIP's user avatar
  • 179
0 votes
0 answers
45 views

Kedro register dataset from a board from pins package

For my project I want to use a combination of kedro for the pipeline orchestration and pins for data and model versioning. I have some data which I stored on a board from the pins package. As I have ...
Mischa's user avatar
  • 137
2 votes
2 answers
672 views

How to use Kedro with Great-expectations?

I am using Kedro to create a pipeline for ETL purposes and column specific validations are being done using Great-Expectations. There is a hooks.py file listed in Kedro documentation here. This hook ...
Dhaval Thakkar's user avatar
1 vote
0 answers
137 views

Kedro, running inference on user input

I have a pipeline with the model I want to use. Outside of the project, I have an app.py file where I'm going to create the UI/UX for my users to run my model. Right now I'm just using a sample string ...
João Areias's user avatar
  • 1,212
0 votes
0 answers
99 views

how to call kedro pipline or nodes in Django framework

I have to call Kedro nodes or pipelines in Django API, and need to use Kedro pipelines or nodes output as input in Django API. Not getting any solution. Please suggest some solution for this. How to ...
Pallavi Kumari's user avatar
1 vote
0 answers
157 views

Integrate kedro debugger with docker container

I'm currently trying to set up the debugger for kedro in VScode adding the following launch.json as the documentation suggests. The thing is, I have 2 different docker containers that I would like to ...
panky_is's user avatar
0 votes
1 answer
62 views

Grouping raw datasets in a kedro visualization

I am looking for a way to group all of the raw datasets in a kedro pipeline visualization into one collapsible/expandable "node", similar to the way that namespaces are collapsible/...
Ben Smith's user avatar
0 votes
3 answers
681 views

Kedro - Getting path to item in the datacatalog

I'm training an nlp model using spacy. I have the preprocessing steps all written as a pipeline, and now I need to do the training. According to spacy's documentation I need to run the following ...
João Areias's user avatar
  • 1,212
2 votes
1 answer
810 views

How can I register kedro data catalog programmatically in Kedro 0.18?

For various reasons (mainly ability to dynamically construct file paths) I like to define the data catalog programatically, and not use yaml file to define datasets e.g. DataCatalog( {"...
gants_kuhelgarten's user avatar
3 votes
1 answer
678 views

Kedro - How to update a dataset in a Kedro pipeline given that a dataset cannot be both input and output of a node (only DAG)?

In a Kedro project, I have a dataset in catalog.yml that I need to increment by adding a few lines each time I call my pipeline. #catalog.yml my_main_dataset: type: pandas.SQLTableDataSet ...
Jean-Baptiste's user avatar
0 votes
1 answer
74 views

Can't run KedroSession with `from_inputs` parmeter: ValueError: Pipeline does not contain data_sets named [...]

In jupyter notebook, when I run session.run(pipeline_name='sim', from_inputs=['measurements', 'params:simulation']), passing datasets & params specified in catalog.yaml, everything works fine. ...
ilja's user avatar
  • 121
0 votes
1 answer
442 views

How to generate kedro pipelines automatically (like DataEngineerOne does)?

Having seen the video of DataEngineerOne: How To Use a Parameter Range to Generate Pipelines Automatically I want to automate a pipeline that simulates an electronic circuit. I want to do a grid ...
ilja's user avatar
  • 121
1 vote
1 answer
365 views

How to show plotly chart in kedro

I am trying to use data science tool kedro according to this tutorial. I followed the instruction(write config.yaml, node.py and pipeline.py etc) and do exactly the same as the documentation) and ...
Ryo Matsuzaka's user avatar
0 votes
2 answers
259 views

How can I run `catalog.load` in a non-IPython context?

In IPython I can run data = catalog.load('my_dataset') in order to load a dataset specified as 'my_dataset' in the catalog.yml file. What's the equivalent command in a pthon script? What do I need to ...
ilja's user avatar
  • 121
0 votes
1 answer
247 views

Include Quarto rendering in kedro pipeline and pass it inputs/outputs

I am using kedro to make some comparative analysis. I am using the quarto python package providing a wrapper to the quarto cli through the render function. This function will take a qmd file as input ...
Oneira's user avatar
  • 1,405
1 vote
2 answers
416 views

kedro PartitionedDataSet lazy saving to spare memory?

I am working with PartionedDataSet in kedro. One of the data set is of type pillow.ImageDataSet: raw_images: type: PartitionedDataSet <<: *data_path_on_disk dataset: ...
Oneira's user avatar
  • 1,405
1 vote
1 answer
389 views

Kedro on Databricks: Cannot import SparkDataset

Cannot import SparkDataset in Databricks using; from kedro.extras.datasets.spark import SparkDataSet
Abhijit Das's user avatar

1
2 3 4 5