Questions tagged [dagster]
Dagster is an open source system for building modern data applications.
145
questions
0
votes
0
answers
9
views
How to add op_tags to AssetsDefinitions after it has been defined
I can define an asset with op_tags, like
@asset(op_tags={"model": "lgbm"})
def model():
return MODEL
but I would like to use decorators to update/modify an assets tag, like
@...
0
votes
0
answers
23
views
Dagster getting partition using different timezone
I would like some help in scheduling a dagster job that should get s3 folders on a different timezone. For example it brings all data s3:///2023/09/01/00 to 23 (i.e all hours). But what I want is it ...
0
votes
0
answers
19
views
Dagster reload instead of shutdown and restart
when developing dagster ops , should I restart the service on each code changes to see the results? Or is it possible to reload it automatically?
I asked chatgpt information regarding Dagster's ...
0
votes
2
answers
46
views
Dagster: Why do I need to specify ECS cluster name, and separately its CPU/memory requirements?
In Dagster cloud's documentation, for AWS and under Per-deployment configuration, I see that I need to specify both:
Cluster name (Under "cluster")
Requested server resources (Under ...
0
votes
1
answer
14
views
Does Dagster-cloud support autoscaling directly?
I found in Dagster's documentation a https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#per-job-kubernetes-configuration
On auto scaling using Kubernetes.
Does Dagster ...
0
votes
1
answer
30
views
Setting GPU resources when using Dagster
I found in Dagster's documentation a section on customizing the resource requirements for machines orchestrated by that service, in AWS ECS and over Kubernetes:
https://docs.dagster.io/dagster-cloud/...
0
votes
1
answer
18
views
Deploy to different machine profiles using DAGster
Can I specify the machine or machine type I'd like to deploy each task/workflow in Dagster separately?
(e.g. Having an A->B DAG, make step A performed on a machine of type M_a, and have step B ...
0
votes
0
answers
11
views
A schedule running multiple hobs sequentially
I am trying to create a schedule that will fit my model to sensor data every week. Now, I have about 100 sensors to analyse and the model architecture and parameters are the same. I have a job which ...
0
votes
0
answers
30
views
Dagster get partition of job ran before sensor triggered
I want to get the partition of a job that is than before a sensor is trigger. I not sure how to do that in dagster.
I have a sensor looks like this:
@sensor(job=job2)
def run_job2_after_job1:
date= ...
0
votes
0
answers
43
views
How to using Dagster recommended structure with Docker/ECS
Dagster recommends structuring your project list this:
project_fully_featured
├── Makefile
├── README.md
├── dbt_project
├── project_fully_featured
│ ├── __init__.py
│ ├── assets
│ │ ├── ...
1
vote
1
answer
23
views
NameError: name 'topstory_ids' is not defined
I am new to Dagster and trying to follow the tutorial from official documentation, https://docs.dagster.io/tutorial/building-an-asset-graph
After I copied and pasted this code, I got this error. ...
0
votes
0
answers
40
views
How to make @op run in sequence when job invoked from sensor?
can you please help me overcome the below issue
i have an sensor [my_sensor] defined on a job [my_job].
[my_job] executes op [my_op] which requires an input [filename] from sensor which i am passing ...
0
votes
0
answers
37
views
Dagster+dbt : referring a dagster asset as a source in dbt model
my dbt_model’s source is a dasgter asset out of dbt, how can I refer this asset inside a dbt model definiton?
the dagster asset =
@asset(
group_name=". .... ",
io_manager_key=".....
0
votes
1
answer
78
views
DagsterImportError: Encountered ImportError: `cannot import name 'URL' from 'datasets'
I am running dagster dev in a folder created by running dagster project scaffold --name dagster_pipeline:
assets.py imports a constant from datasets.py that I have added:
import requests
from dagster ...
0
votes
0
answers
53
views
Get the previous scheduled parition date in a dagster asset
Is there a way to get the previous scheduled partition key in an asset?
If i use the below in a scheduled asset it will return a partition key even if the should_execute in the schedule is False:
@...
0
votes
1
answer
95
views
Dagster UI not showing on Digital Ocean with nginx
I'm trying to run open-source Dagster on a digital ocean droplet. This droplet is working OK for MLFlow and accessing our database via phpmyadmin.
When I try to access the Dagster UI via www.dev....
1
vote
0
answers
260
views
How to pass parameter to asset from op in dagster
I am new to dagster and having difficult in running the asset for different parameter values when job scheduled.
I have created a pipeline using dagster.
Trying to materialize the outcome of upstream ...
0
votes
1
answer
131
views
How to provide parameters defined at the source asset declaration to the IO Manager?
So my initial task was to create an IO Manager that should connect to a database and return data as pandas dataframe.
(I am using dagster 1.3.10)
Design
IMO, the credentials (ip, port, user, password) ...
0
votes
1
answer
103
views
dagster: run every hour and pass last execution time to asset
I'm new to dagster to please be patient for not fully grasping it yet.
I want to "run an asset" every hour which should be possible with schedules. What i also want is to pass the last ...
1
vote
1
answer
251
views
write test for dagster asset job
I am trying to write a simple test for a dagster job and I can't get it through...
I am using dagster 1.3.6
So I have defined this job using the function dagster.define_asset_job
from dagster import ...
0
votes
1
answer
210
views
Unable to open Dagster UI on local machine when dagster script is running from inside a docker container
I am running into an issue trying to open the Dagster UI from inside a docker container. Furthermore, this is a step towards my larger of goal, of opening the dagster UI from a docker container, thats ...
0
votes
1
answer
292
views
create multiple jobs from similar assets with different partitions and schedules
I have a assets.py with the following:
from dagster import asset, job, Output
from my_modules import some_func
from ..partitions import daily_partitions_def, t1_daily_partitions_def
@asset(...
0
votes
1
answer
167
views
Get Dagster Config Schema in Python
I am trying to programmatically kick off a Dagster job from my Python code. So far I am all set with using the python client to kick off a job with either the default or fully specified config.
The ...
1
vote
0
answers
92
views
Docker AWS ECS integration on Mac NoCredentialProviders error when running "docker compose up"
When I am trying to run docker compose up to deploy my infrastructure to AWS using Docker's ECS integration. I'm getting the following error
NoCredentialProviders: no valid providers in the chain. ...
0
votes
1
answer
76
views
How to create DAGs with Spark?
I'm new to Spark and I've realized that, for the pipeline I'm making, it would be much more convenient to have a DAG to represent the pipeline to improve monitoring, scheduling, etc.
I connected Spark ...
0
votes
0
answers
95
views
Execute ECS task from Dagster
I am relatively new to Dagster. I am trying to execute an ECS task from dagster using this standalone python file as a job:
from dagster import job, op
@op()
def my_op(context):
context.log.info('...
0
votes
2
answers
280
views
How to implement io_manager that have a parameter at asset level?
I am a bit confused about the usage of resources, configuration and how they are linked to a context and an asset.
So I have a parquet io manager that is able to manipulate and partitionned not-...
2
votes
0
answers
222
views
How not to overwrite materialized assets in Dagster
I'm new to Dagster.
According to the docs, when you use the built-in filesystem IO manager:
Subsequent materializations of an asset will overwrite previous materializations of that asset.
https://...
-1
votes
1
answer
119
views
How can I partition dagster assets by year?
Not seeing a built in definition to partition assets by year (just hourly-monthly). Is there a way to manipulate the built in time definitions to accomplish this?
Any help is appreciated!
0
votes
0
answers
805
views
cannot import name 'Definitions' from 'dagster'
I'm trying out this example project: https://github.com/dagster-io/dagster/tree/master/examples/project_fully_featured
I added a workspace.yaml file with the lines:
load_from:
- python_module: ...
0
votes
1
answer
289
views
How do you explicitly name an Output in Dagster?
I have the following asset in Dagster:
@asset
def some_asset():
output = "some_great_output"
return Output(
value=output,
)
As it is above, Dagster will name the ...
0
votes
1
answer
273
views
Reduce a partitioned asset to a single data frame
I have a software defined asset that compares a (very) large set of partitions to a single small data frame. I've made a toy explainer that shows the situation:
partitions = dagster....
0
votes
1
answer
360
views
How to load assets from dbt project? #12893
I'm trying to load assets from my dbt project called "dbtz", but i'm receiving the error
hyprster:dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2)...
0
votes
1
answer
306
views
Dagster is detecting my code but not my sensors?
I have a simple local dagster project. One of its files is as follows:
from dagster import asset, Definitions, build_asset_reconciliation_sensor, AssetSelection
group_name = 'for_testing'
@asset(...
0
votes
0
answers
126
views
Dagster is not recognizing schedule despite it running successfully
I have a local dagter project.
I start the dagit & dagster-deamon via
export DAGSTER_HOME="{path-to-project}/my-dagster-project/"
export PYTHONLEGACYWINDOWSSTDIO="UTF-8"
...
0
votes
0
answers
252
views
Call downstream assets or ops in parallel with Dagster
I build a data pipeline with Dagster using the asset API.
I am looking for a way to explicitly call/execute an asset (or op) for a list of items. While assets are executed in parallel if they are ...
1
vote
1
answer
73
views
Dagster: handle TypedDict as op output
Context
To keep track of the origin of my Dataframes, across my different ops, I defined TaggedDF as
class TaggedDF(TypedDict):
df: DataFrame
ressource_name: str
So I could
@op(retry_policy=...
0
votes
2
answers
330
views
stream data between tasks in pipeline orchestration tool Prefect/Dagster/Airflow
How can I stream data between tasks in a workflow with the help of a data pipeline orchestration tool like Prefect, Dagster or Airflow?
I am looking for a good data pipeline orchestration tool.
I ...
0
votes
0
answers
197
views
Dagster orchestrator - How can I run commands to process dbt models in another Docker container?
I'm playing a bit with Dagster orchestrator and I have some doubts. My intention is use this tool only as a scheduler and runner for commands since I have the business logic developed in other ...
0
votes
1
answer
275
views
How to use dagster with great expectations?
The issue
I'm trying out great expectations with dagster, as per this guide
My pipeline seems to execute correctly until it reaches this block:
expectation = dagster_ge.ge_validation_op_factory(
...
0
votes
0
answers
116
views
Providing the same resource to every op in a job
So I need myresource in every op. How do I do this without having to specify it in every op, but in the job definition instead? I think this is possible, but I cannot find a specification in the ...
1
vote
1
answer
98
views
How do I add the materialization runtime to a software defined asset in Dagster?
I would like to keep track of how long it takes to materialize software defined assets over time (using Dagster).
Ideally I'd add the "duration" to the materialization metadata.
I could do ...
3
votes
1
answer
872
views
dagster can you trigger a job to run via an api?
I have been looking all over for the answer, but can't seem to find what I'm looking for
I want to create an api endpoint that can pass information to the dagster assets and trigger a run. For example,...
0
votes
1
answer
450
views
How do I write a Dagster asset that depends on an earlier partition of itself?
I was using depends_on_past with Airflow. I'm now using Dagster, with software-defined assets, and I was told that the way to get similar functionality is with build_asset_reconciliation_sensor and a ...
4
votes
2
answers
658
views
why does kubernetes delete secrets after helm upgrade?
when performing helm upgrade, I find that secrets that are created upon initial install are deleted. Why is this? The example I am using is dagster. When installing with:
helm install dagster dagster/...
1
vote
1
answer
349
views
Dagster -Execute an @Op only when all parallel executions are finished(DynamicOutput)
I have a problem that in fact I am not able to solve in dagster.
I have the following configuration:
I have step 1 where I get the data from an endpoint
step 2 gets a list of customers dynamically:
...
1
vote
1
answer
398
views
Getting current execution date in a task or asset in dagster
Is there an easier way than what I'm doing to get the current date in an dagster asset, than what I'm currently doing?
def current_dt():
return datetime.today().strftime('%Y-%m-%d')
@asset
def ...
2
votes
1
answer
487
views
how to iterate over a list of values returning from ops to jobs in dagster
I am new to the dagster world and working on ops and jobs concepts. \
my requirement is to read a list of data from config_schema and pass it to @op function and return the same list to jobs. \
The ...
1
vote
2
answers
569
views
How do I pass data to an op in a different module in dagster?
I am new to dagster and am having a difficult time sorting this one out. I have to jobs defined in my dagster pipeline and I want to pass data from an op in one job to an op in another
My setup is as ...
0
votes
0
answers
90
views
How to allow memoization for jobs defined from Dagster assets
I would like to use Memoization for jobs that I am defining from AssetsDefinitions with define_asset_job.
The problem is: define_asset_job does not allow me to set-up a version_strategy.
How can I ...