Questions tagged [kedro]
Kedro is an open source Python library that helps you build production-ready data and analytics pipelines
204
questions
1
vote
1
answer
25
views
Your Kedro project version 0.17.6 does not match Kedro package version
when running kedro ipython into a cloned repo, I get:
_version_mismatch_error(metadata_dict["kedro_init_version"]))
ValueError: Your Kedro project version 0.17.6 does not match Kedro package ...
1
vote
1
answer
35
views
KEDRO - How to specify an arbitrary binary file in catalog.yml?
I'm currently working on a datascience project using LLMs (Large language models). Weights for models usually come in different formats, most frequently .bin or .gguf, and I'd like to keep it that way....
1
vote
1
answer
72
views
Kedro viz blank page
I created a sample pipeline that works correctly with kedro run, however when I try to visualise it with kedro viz I'm getting basically a blank page, even though the terminal doesn't show a single ...
1
vote
2
answers
62
views
Kedro simple project run error: ModuleNotFoundError followed by ValueError
Full code available on: https://github.com/braKhAbid/kedro_mnist
I am trying to setup a simple python ML project in Kedro. It is very simple, one pipeline of three nodes, a data loading node, a model ...
0
votes
1
answer
6
views
How to import Custom json encoder class into data cataloge
I’ve a df which stores lists in a column. I am saving the df with all columns in json using
config_new:
type: json.JSONDataSet
filepath: data/01_raw/new_config.json
save_args:
indent: 6
It’...
1
vote
1
answer
74
views
in kedro / pyspark how to use MemoryDataset
I am trying to use a MemoryDataset with kedro, in order to not save the intermeiate result to disk.
# nodes.py
def preprocess_format_tracksessions(tracksess: DataFrame, userid_profiles:pd.DataFrame , ...
0
votes
1
answer
37
views
Define column names when reading a spark dataset in kedro
With kedro, how can I define the column names when reading a spark.SparkDataSet? below my catalog.yaml.
user-playlists:
type: spark.SparkDataSet
file_format: csv
filepath: data/01_raw/lastfm-...
0
votes
1
answer
33
views
in Kedro, how to handle tar.gz archives from the web
I have a tar.gz file that I am downloading from this link: http://ocelma.net/MusicRecommendationDataset/lastfm-1K.html
What is the best way to fully integrate this TSV data into kedro, perhaps with an ...
1
vote
1
answer
215
views
Getting Kedro globals.yml to work with OmegaConfigLoader
I just started evaluating Kedro for use, and I began with a small project where I read data from a MS-SQL Server. The pipeline will run with a few months in between, and with new date ranges every ...
2
votes
1
answer
106
views
logging in python and kedro, how to log only DEBUG info to a file and INFO to console
I'm trying to configure logging so that INFO level messages go to the console and DEBUG level messages go to a file instead. So far, I am able to get working INFO to console and DEBUG to file, the ...
0
votes
0
answers
68
views
how to create kedro catalog entry + custom DataSet init method with values from credentials.yml
I'm trying to create a custom DataSet class within the kedro framework. I need some help understanding how to combine values from the credentials.yml file.
what is the kedro way of handling the '...
1
vote
1
answer
202
views
how do kedro catalog properties get passed to Datasets when pipeline is executed
I'm trying to create two custom kedro DataSets and I'm not quite sure how to configure the __init__ methods.
The first DataSet is simple, its a readonly, fetch rss items from a static URL.
catalog ...
0
votes
2
answers
92
views
How can I find kedro_cli.py in my project?
I installed Kedro using pip install, and I made a new project using kedro new. I was able to run (kedro run --pipeline myPipelineName) from terminal in PyCharm or from cmd when I make cd to the ...
3
votes
1
answer
280
views
Is there a way to have files in the Kedro Catalog, that are missing?
I have a kedro pipeline which generates a file that is used again for the next run of that same pipeline. However, when the pipeline runs for the first time, that file does not exist, and it is ...
1
vote
1
answer
173
views
How to combine multiple similar datasets into one in Kedro dynamically?
I use Kedro to run pipelines for multiple models that each generate a CSV file with the same schema. However, I need to combine each of the generated CSV files into one and do some post processing ...
0
votes
1
answer
155
views
Access Kedro MemoryDataSet when running packaged Kedro pipeline in a script
I want to be able to access a MemoryDataSet results dataframe from a kedro pipeline that I've imported into a script after packaging the pipeline into a python package.
I have a kedro pipeline written ...
2
votes
1
answer
103
views
S3FS is not recognised in lambda when using torch
I have a Kedro Pipeline Node that on AWS Lambda that accesses s3. It runs if I'm not using torch but fails with Install s3fs to access S3 if I add torch as a dependency.
I have a Kedro Pipeline I want ...
1
vote
3
answers
231
views
Unable to Save Data to HDF5 File using fsspec in Python
I'm trying to save data (3D numpy array) to an HDF5 file using fsspec in Python, but I'm encountering issues and I am unable to successfully write the data to the file.
The bigger picture is I am ...
1
vote
1
answer
116
views
Kedro - how to set max_workers when running pipelines with ParallelRunner?
I'm using Kedro version 0.18.7 and python 3.9 in WSL2.
I'd like to run nodes of my pipeline in parallel by running the command kedro run --pipeline <pipeline_name> --runner ParallelRunner. ...
1
vote
0
answers
91
views
How to call the data object from another namespace in already namespaced kedro pipeline?
I have the following namespaced pipelines in kedro:
def create_pipeline(**kwargs) -> Pipeline:
pipeline_instance = pipeline(
[
node(
func=split_data,
...
0
votes
1
answer
66
views
kedro: Used azure key and scope, then pull data from snowflake via databricks cluster
I'm currently running into a requirement where we are using a scope/key credential to access/grab data from our Snowflake instance, then load it in our blob storage, while using our Databricks cluster....
1
vote
2
answers
544
views
Kedro Pipeline contains no nodes after applying all provided filters
ValueError: Pipeline contains no nodes after applying all provided filters
src/pipelines/data_processing<br>
src/pipelines/data_science<br>
src/pipelines/data_project<br>
src/...
2
votes
1
answer
161
views
Where is my kedro output when using the databricks extension in vscode
I am using kedro together with the databricks extension for vscode to access databricks server on Azure.
Everything works pretty well but I don't see any output when executing the file locally.
The ...
0
votes
1
answer
82
views
Login time out error for pyodbc connection string in kedro
I'm trying to build up connection string in kedro for SQLQueryDataset to set the connection with MSSQL and Pyodbc in Azure Databricks but encountered with an error:
(pyodbc.OperationalError) ('HYT00',...
0
votes
1
answer
33
views
Interpolate sql in SQLDataset in catalog.yml
Is there a way to interpolate a SQLDataset query in catalog.yml passing some argument\parameter.
Example:
person:
type: pandas.SQLQueryDataSet
sql: "SELECT * FROM public.people WHERE id = ${...
0
votes
1
answer
49
views
Adding multiple data catalog into one in kedro
I am dynamically creating pipeline, nodes and catalog for a project. After creating the them, I want to add pipelines and the catalogs. Adding the pipelines is possible by sum function, but it's not ...
0
votes
0
answers
231
views
Colorful notebook output with rich library
May someone tell me how to set again that colorful output from jupyter notebook without using rich.print? I use VSCode.
I've got this feature with kedro=0.18.4 and lost with kedro=0.18.5. Kedro ...
0
votes
1
answer
547
views
Parametrize input datasets in kedro
I'm trying to move my project into a kedro pipeline but I'm struggling with the following step:
my prediction pipeline is being run by a scheduler. The scheduler supplies all the necessary parameters (...
0
votes
0
answers
269
views
kedro dynamic catalog creation only for specific nodes before their run
I have several thousands of files to be processed of the different types. I am using dynamic catalog creation with hooks. I used first after_catalog_created hook but it is too early in and I need ...
2
votes
0
answers
72
views
Kedro catalog fails when overwriting a GeoJson dataset even though the driver is supported
I have the following catalog item in my kedro project
suggested_routes_table@geopandas: type: geopandas.GeoJSONDataSet filepath: data/04_feature/routes_suggestions_table.geojson load_args: driver: &...
0
votes
1
answer
88
views
Does kedro data catalog accept .arrow files?
While using Kedro I want to load some data and work with it. To do that, one has to register the data in a conf/base/catalog.yml file.
The Kedro Documentation of the Data Catalog explains how one can ...
1
vote
1
answer
2k
views
Python: kedro viz SQLAlchemy DeprecationWarning
I tried to work with kedro and started with the spaceflight tutorial.
I installed the src/requirements.txt in a .venv.
When running kedro viz (or kedro run or even kedro --version), I get lets of ...
0
votes
1
answer
165
views
kedro jupyter notebook in command prompt returns kedro.framework.cli.jupyter.single kernelspec manager' could not be imported"
I have been trying to activate jupyter notebooks in a kedro context for over 24 hours now and I receive the same error all the time. I have searched around and no one seems to be able to solve this ...
0
votes
1
answer
299
views
Kedro (Python) DeprecationWarning: `np.bool8`
When I try to create a new kedro project or run an existing one, I get the following deprecation warning (see also screenshot below). As far as I understand the warning is neglebile, however, as I am ...
0
votes
0
answers
45
views
Kedro register dataset from a board from pins package
For my project I want to use a combination of kedro for the pipeline orchestration and pins for data and model versioning. I have some data which I stored on a board from the pins package.
As I have ...
2
votes
2
answers
672
views
How to use Kedro with Great-expectations?
I am using Kedro to create a pipeline for ETL purposes and column specific validations are being done using Great-Expectations. There is a hooks.py file listed in Kedro documentation here. This hook ...
1
vote
0
answers
137
views
Kedro, running inference on user input
I have a pipeline with the model I want to use. Outside of the project, I have an app.py file where I'm going to create the UI/UX for my users to run my model. Right now I'm just using a sample string ...
0
votes
0
answers
99
views
how to call kedro pipline or nodes in Django framework
I have to call Kedro nodes or pipelines in Django API, and need to use Kedro pipelines or nodes output as input in Django API. Not getting any solution. Please suggest some solution for this.
How to ...
1
vote
0
answers
157
views
Integrate kedro debugger with docker container
I'm currently trying to set up the debugger for kedro in VScode adding the following launch.json as the documentation suggests.
The thing is, I have 2 different docker containers that I would like to ...
0
votes
1
answer
62
views
Grouping raw datasets in a kedro visualization
I am looking for a way to group all of the raw datasets in a kedro pipeline visualization into one collapsible/expandable "node", similar to the way that namespaces are collapsible/...
0
votes
3
answers
681
views
Kedro - Getting path to item in the datacatalog
I'm training an nlp model using spacy. I have the preprocessing steps all written as a pipeline, and now I need to do the training. According to spacy's documentation I need to run the following ...
2
votes
1
answer
810
views
How can I register kedro data catalog programmatically in Kedro 0.18?
For various reasons (mainly ability to dynamically construct file paths) I like to define the data catalog programatically, and not use yaml file to define datasets e.g.
DataCatalog(
{"...
3
votes
1
answer
678
views
Kedro - How to update a dataset in a Kedro pipeline given that a dataset cannot be both input and output of a node (only DAG)?
In a Kedro project, I have a dataset in catalog.yml that I need to increment by adding a few lines each time I call my pipeline.
#catalog.yml
my_main_dataset:
type: pandas.SQLTableDataSet
...
0
votes
1
answer
74
views
Can't run KedroSession with `from_inputs` parmeter: ValueError: Pipeline does not contain data_sets named [...]
In jupyter notebook, when I run session.run(pipeline_name='sim', from_inputs=['measurements', 'params:simulation']), passing datasets & params specified in catalog.yaml, everything works fine. ...
0
votes
1
answer
442
views
How to generate kedro pipelines automatically (like DataEngineerOne does)?
Having seen the video of DataEngineerOne: How To Use a Parameter Range to Generate Pipelines Automatically I want to automate a pipeline that simulates an electronic circuit. I want to do a grid ...
1
vote
1
answer
365
views
How to show plotly chart in kedro
I am trying to use data science tool kedro according to this tutorial.
I followed the instruction(write config.yaml, node.py and pipeline.py etc) and do exactly the same as the documentation) and ...
0
votes
2
answers
259
views
How can I run `catalog.load` in a non-IPython context?
In IPython I can run data = catalog.load('my_dataset') in order to load a dataset specified as 'my_dataset' in the catalog.yml file. What's the equivalent command in a pthon script? What do I need to ...
0
votes
1
answer
247
views
Include Quarto rendering in kedro pipeline and pass it inputs/outputs
I am using kedro to make some comparative analysis.
I am using the quarto python package providing a wrapper to the quarto cli through the render function. This function will take a qmd file as input ...
1
vote
2
answers
416
views
kedro PartitionedDataSet lazy saving to spare memory?
I am working with PartionedDataSet in kedro. One of the data set is of type pillow.ImageDataSet:
raw_images:
type: PartitionedDataSet
<<: *data_path_on_disk
dataset:
...
1
vote
1
answer
389
views
Kedro on Databricks: Cannot import SparkDataset
Cannot import SparkDataset in Databricks using;
from kedro.extras.datasets.spark import SparkDataSet