Questions tagged [kedro]
Kedro is an open source Python library that helps you build production-ready data and analytics pipelines
184
questions
0
votes
0
answers
15
views
Login time out error for pyodbc connection string in kedro
I'm trying to build up connection string in kedro for SQLQueryDataset to set the connection with MSSQL and Pyodbc in Azure Databricks but encountered with an error:
(pyodbc.OperationalError) ('HYT00',...
0
votes
1
answer
16
views
Interpolate sql in SQLDataset in catalog.yml
Is there a way to interpolate a SQLDataset query in catalog.yml passing some argument\parameter.
Example:
person:
type: pandas.SQLQueryDataSet
sql: "SELECT * FROM public.people WHERE id = ${...
0
votes
0
answers
16
views
Adding multiple data catalog into one in kedro
I am dynamically creating pipeline, nodes and catalog for a project. After creating the them, I want to add pipelines and the catalogs. Adding the pipelines is possible by sum function, but it's not ...
0
votes
0
answers
36
views
Colorful notebook output with rich library
May someone tell me how to set again that colorful output from jupyter notebook without using rich.print? I use VSCode.
I've got this feature with kedro=0.18.4 and lost with kedro=0.18.5. Kedro ...
0
votes
1
answer
66
views
Parametrize input datasets in kedro
I'm trying to move my project into a kedro pipeline but I'm struggling with the following step:
my prediction pipeline is being run by a scheduler. The scheduler supplies all the necessary parameters (...
0
votes
0
answers
53
views
kedro dynamic catalog creation only for specific nodes before their run
I have several thousands of files to be processed of the different types. I am using dynamic catalog creation with hooks. I used first after_catalog_created hook but it is too early in and I need ...
2
votes
0
answers
42
views
Kedro catalog fails when overwriting a GeoJson dataset even though the driver is supported
I have the following catalog item in my kedro project
suggested_routes_table@geopandas: type: geopandas.GeoJSONDataSet filepath: data/04_feature/routes_suggestions_table.geojson load_args: driver: &...
0
votes
1
answer
44
views
Does kedro data catalog accept .arrow files?
While using Kedro I want to load some data and work with it. To do that, one has to register the data in a conf/base/catalog.yml file.
The Kedro Documentation of the Data Catalog explains how one can ...
1
vote
1
answer
1k
views
Python: kedro viz SQLAlchemy DeprecationWarning
I tried to work with kedro and started with the spaceflight tutorial.
I installed the src/requirements.txt in a .venv.
When running kedro viz (or kedro run or even kedro --version), I get lets of ...
0
votes
1
answer
59
views
kedro jupyter notebook in command prompt returns kedro.framework.cli.jupyter.single kernelspec manager' could not be imported"
I have been trying to activate jupyter notebooks in a kedro context for over 24 hours now and I receive the same error all the time. I have searched around and no one seems to be able to solve this ...
0
votes
1
answer
103
views
Kedro (Python) DeprecationWarning: `np.bool8`
When I try to create a new kedro project or run an existing one, I get the following deprecation warning (see also screenshot below). As far as I understand the warning is neglebile, however, as I am ...
0
votes
0
answers
33
views
Kedro register dataset from a board from pins package
For my project I want to use a combination of kedro for the pipeline orchestration and pins for data and model versioning. I have some data which I stored on a board from the pins package.
As I have ...
0
votes
2
answers
238
views
How to use Kedro with Great-expectations?
I am using Kedro to create a pipeline for ETL purposes and column specific validations are being done using Great-Expectations. There is a hooks.py file listed in Kedro documentation here. This hook ...
0
votes
0
answers
63
views
Kedro, running inference on user input
I have a pipeline with the model I want to use. Outside of the project, I have an app.py file where I'm going to create the UI/UX for my users to run my model. Right now I'm just using a sample string ...
0
votes
0
answers
53
views
how to call kedro pipline or nodes in Django framework
I have to call Kedro nodes or pipelines in Django API, and need to use Kedro pipelines or nodes output as input in Django API. Not getting any solution. Please suggest some solution for this.
How to ...
0
votes
0
answers
25
views
Integrate kedro debugger with docker container
I'm currently trying to set up the debugger for kedro in VScode adding the following launch.json as the documentation suggests.
The thing is, I have 2 different docker containers that I would like to ...
0
votes
1
answer
31
views
Grouping raw datasets in a kedro visualization
I am looking for a way to group all of the raw datasets in a kedro pipeline visualization into one collapsible/expandable "node", similar to the way that namespaces are collapsible/...
0
votes
2
answers
187
views
Kedro - Getting path to item in the datacatalog
I'm training an nlp model using spacy. I have the preprocessing steps all written as a pipeline, and now I need to do the training. According to spacy's documentation I need to run the following ...
0
votes
1
answer
254
views
How can I register kedro data catalog programmatically in Kedro 0.18?
For various reasons (mainly ability to dynamically construct file paths) I like to define the data catalog programatically, and not use yaml file to define datasets e.g.
DataCatalog(
{"...
1
vote
1
answer
196
views
Kedro - How to update a dataset in a Kedro pipeline given that a dataset cannot be both input and output of a node (only DAG)?
In a Kedro project, I have a dataset in catalog.yml that I need to increment by adding a few lines each time I call my pipeline.
#catalog.yml
my_main_dataset:
type: pandas.SQLTableDataSet
...
0
votes
1
answer
29
views
Can't run KedroSession with `from_inputs` parmeter: ValueError: Pipeline does not contain data_sets named [...]
In jupyter notebook, when I run session.run(pipeline_name='sim', from_inputs=['measurements', 'params:simulation']), passing datasets & params specified in catalog.yaml, everything works fine. ...
0
votes
0
answers
62
views
How to generate kedro pipelines automatically (like DataEngineerOne does)?
Having seen the video of DataEngineerOne: How To Use a Parameter Range to Generate Pipelines Automatically I want to automate a pipeline that simulates an electronic circuit. I want to do a grid ...
1
vote
2
answers
132
views
How to show plotly chart in kedro
I am trying to use data science tool kedro according to this tutorial.
I followed the instruction(write config.yaml, node.py and pipeline.py etc) and do exactly the same as the documentation) and ...
0
votes
2
answers
57
views
How can I run `catalog.load` in a non-IPython context?
In IPython I can run data = catalog.load('my_dataset') in order to load a dataset specified as 'my_dataset' in the catalog.yml file. What's the equivalent command in a pthon script? What do I need to ...
0
votes
1
answer
115
views
Include Quarto rendering in kedro pipeline and pass it inputs/outputs
I am using kedro to make some comparative analysis.
I am using the quarto python package providing a wrapper to the quarto cli through the render function. This function will take a qmd file as input ...
0
votes
2
answers
132
views
kedro PartitionedDataSet lazy saving to spare memory?
I am working with PartionedDataSet in kedro. One of the data set is of type pillow.ImageDataSet:
raw_images:
type: PartitionedDataSet
<<: *data_path_on_disk
dataset:
...
0
votes
0
answers
36
views
Kedro template configuration does not load globals.yml configuration into catalog.yml FOR Jupyter Lab
It works for the CLI ( kedro run ... ) but not for Jupyter Lab. I have just recently upgraded from 0.17.1 to 0.18.3.
Have made changes to settings.py which uses the Templated Config Loader ( just like ...
0
votes
1
answer
120
views
Kedro on Databricks: Cannot import SparkDataset
Cannot import SparkDataset in Databricks using;
from kedro.extras.datasets.spark import SparkDataSet
0
votes
1
answer
548
views
import fsspec throws error (AttributeError: 'EntryPoints' object has no attribute 'get')
import fsspec throws error (AttributeError: 'EntryPoints' object has no attribute 'get')
[]
1
vote
0
answers
124
views
Is there a way to include an Azure Databricks Lakehouse query as a DataCatalog dataset in kedro?
We want to use kedro to control our ML pipelines in Azure Databricks.
We are querying (and joining) relatively large tables in Databricks' Lakehouse. Therefore, we would like to include those joins in ...
0
votes
1
answer
181
views
How to change the kedro configuration environment in jupyter notebook?
I want to run a kedro pipeline in the base env using jupyter notebook. I do this the following way:
%reload_kedro --env=base
session.run(pipeline_name='dpfm1')
Doing this, the %reload_kedro command ...
0
votes
3
answers
439
views
How to run a kedro pipeline interactively like a function
I would like to run kedro pipelines in jupyter notebook with different inputs, so something like this:
data = catalog.load('my_dataset')
params = catalog.load('params:my_params')
pipelines['...
0
votes
2
answers
107
views
DataSetError in Docker Kedro deployment
I try to deploy example Kedro starter project (pandas-iris).
I successfuly run it locally (kedro run), and then, having kedro-docker install, init a Docker, build image and push it to my registry.
...
0
votes
0
answers
44
views
How to do SQL like querying the parquet files in kedro
I'm new to kedro, I'm just wondering if I could do SQL like querying the parquet files instead of using Dataframe API's. Please help me out if there is a way.
Thanks in advance!
1
vote
1
answer
129
views
How to use generators with kedro?
Thanks to David Beazley's slides on Generators I'm quite taken with using generators for data processing in order to keep memory consumption minimal. Now I'm working on my first kedro project, and my ...
0
votes
2
answers
383
views
dynamic parameters on datasets in Kedro
I would like to call an API to enrich an existing dataset.
The existing dataset is a CSVDataSet configured in the catalog.
Now I would like to create a Node, that enriches the CSVDataSet with data ...
0
votes
1
answer
442
views
ModuleNotFoundError: No module named 'kedro.versioning'
i have upgraded my kedro to latest version.But i have used kedro.versioning in my project.And latest kedro has no module of this name.Can anyone please suggest anything
0
votes
0
answers
140
views
how to use kedro.versioning in latest version of kedro?
I have previously used kedro version 0.17.6 in my project. Now i have upgraded my version to 0.18.2. But in latest version of kedro there is no module of kedro.versioning. So i am getting a error that ...
0
votes
1
answer
271
views
Kedro template configuration does not load/parse variables
A follow up to this question. I am using Kedro v0.18.2. I am trying use the TemplateConfig so I have created a globals.yml under conf/base, which looks like this:
paths:
base_path: s3://...
1
vote
2
answers
365
views
Kedro template configuration does not load globals.yml configuration into catalog.yml
I am using Kedro to create some data pipelines, for which I have various file formats, as well as a custom one. According to the documentation, I have created a globals.yml under config/base where I ...
0
votes
1
answer
162
views
Looking for the right way to make a kedro node output lazily two partitioned datasets
I've built a node in Kedro loading lazily an input partitioned dataset, and saving lazily two partitioned datasets as an output (following recommendations found in the Kedro community: using a lambda +...
2
votes
1
answer
353
views
Load existing data catalog programmatically
I want to write pytest unit test in Kedro 0.17.5. They need to perform integrity checks on dataframes created by the pipeline.
These dataframes are specified in the catalog.yml and already persisted ...
0
votes
2
answers
133
views
Access configuration in the pipelines definition (not only nodes)
I want to dynamically alter pipelines based on the configuration provided. Is there possibility to pass configuration based on the environment to the register_pipelines() or to the create_pipeline() ...
0
votes
1
answer
57
views
Rate limiting Kedro API requests
I have a few datasets from the government dataset that I'm using on my ML model, the problem is, their server is not that great to put it nicely. Whenever I run my pipeline, when I pull from their API ...
0
votes
2
answers
373
views
Kedro 0.16.3 and kedro[spark.SparkDataSet] pip libraries cannot be installed together on databricks cluster
Till last week both kedro and kedro[spark.SparkDataSet] pip libraries were installed on the cluster. But since last 3-4 days they wont be installed together on the cluster. It shows that its a ...
0
votes
3
answers
209
views
Kedro using wrong conda environment
I have created a conda environment called Foo. After activating this environment I installed Kedro with pip, since conda was giving me a conflict. Even though I'm inside the Foo environment, when I ...
0
votes
2
answers
92
views
Getting Kedro Custom Dataset for SunPy Maps to write to/from S3
I'm currently attempting to define a custom dataset to read/write .fits files to/from S3 as SunPy Maps.
The closest thing to this already in the data catalog is the pillow.ImageDataSet pillow....
0
votes
1
answer
149
views
How to pull data from a paginated JSON API using kedro (APIDataSet)?
The problem: I would like to retrieve data from a paginated API that sends JSON responses.
Using kedro.extras.datasets.api.APIDataSet I can query the API and retrieve the initial response. However if ...
1
vote
2
answers
179
views
How to write to HDFS using kedro
I'm trying to output of my Kedro pipeline to the HDFS file system. But I couldn't see on the internet how to do that and on Kedro documents. If anybody configured kedro in catalog please share a ...
0
votes
0
answers
272
views
How to inject the information about load version into Kedro node?
I need to run a Kedro (v0.17.4) pipeline with a node that is supposed to process data with a different logic depending on the load version of the input.
As a simple and crude example assuming there is ...