Questions tagged [dagster]

Dagster is an open source system for building modern data applications.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

How to add op_tags to AssetsDefinitions after it has been defined

I can define an asset with op_tags, like @asset(op_tags={"model": "lgbm"}) def model(): return MODEL but I would like to use decorators to update/modify an assets tag, like @...
pedrovgp's user avatar
  • 767
0 votes
0 answers
23 views

Dagster getting partition using different timezone

I would like some help in scheduling a dagster job that should get s3 folders on a different timezone. For example it brings all data s3:///2023/09/01/00 to 23 (i.e all hours). But what I want is it ...
Srinivas's user avatar
  • 2,020
0 votes
0 answers
19 views

Dagster reload instead of shutdown and restart

when developing dagster ops , should I restart the service on each code changes to see the results? Or is it possible to reload it automatically? I asked chatgpt information regarding Dagster's ...
sahar Jam's user avatar
0 votes
2 answers
46 views

Dagster: Why do I need to specify ECS cluster name, and separately its CPU/memory requirements?

In Dagster cloud's documentation, for AWS and under Per-deployment configuration, I see that I need to specify both: Cluster name (Under "cluster") Requested server resources (Under ...
yuvalm2's user avatar
  • 870
0 votes
1 answer
14 views

Does Dagster-cloud support autoscaling directly?

I found in Dagster's documentation a https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#per-job-kubernetes-configuration On auto scaling using Kubernetes. Does Dagster ...
yuvalm2's user avatar
  • 870
0 votes
1 answer
30 views

Setting GPU resources when using Dagster

I found in Dagster's documentation a section on customizing the resource requirements for machines orchestrated by that service, in AWS ECS and over Kubernetes: https://docs.dagster.io/dagster-cloud/...
yuvalm2's user avatar
  • 870
0 votes
1 answer
18 views

Deploy to different machine profiles using DAGster

Can I specify the machine or machine type I'd like to deploy each task/workflow in Dagster separately? (e.g. Having an A->B DAG, make step A performed on a machine of type M_a, and have step B ...
yuvalm2's user avatar
  • 870
0 votes
0 answers
11 views

A schedule running multiple hobs sequentially

I am trying to create a schedule that will fit my model to sensor data every week. Now, I have about 100 sensors to analyse and the model architecture and parameters are the same. I have a job which ...
Luca's user avatar
  • 10.5k
0 votes
0 answers
30 views

Dagster get partition of job ran before sensor triggered

I want to get the partition of a job that is than before a sensor is trigger. I not sure how to do that in dagster. I have a sensor looks like this: @sensor(job=job2) def run_job2_after_job1: date= ...
Siren O's user avatar
  • 59
0 votes
0 answers
43 views

How to using Dagster recommended structure with Docker/ECS

Dagster recommends structuring your project list this: project_fully_featured ├── Makefile ├── README.md ├── dbt_project ├── project_fully_featured │ ├── __init__.py │ ├── assets │ │ ├── ...
Yepher's user avatar
  • 1,475
1 vote
1 answer
23 views

NameError: name 'topstory_ids' is not defined

I am new to Dagster and trying to follow the tutorial from official documentation, https://docs.dagster.io/tutorial/building-an-asset-graph After I copied and pasted this code, I got this error. ...
Steve's user avatar
  • 3,043
0 votes
0 answers
40 views

How to make @op run in sequence when job invoked from sensor?

can you please help me overcome the below issue i have an sensor [my_sensor] defined on a job [my_job]. [my_job] executes op [my_op] which requires an input [filename] from sensor which i am passing ...
Robinhood's user avatar
0 votes
0 answers
37 views

Dagster+dbt : referring a dagster asset as a source in dbt model

my dbt_model’s source is a dasgter asset out of dbt, how can I refer this asset inside a dbt model definiton? the dagster asset = @asset( group_name=". .... ", io_manager_key=".....
Remis Haroon - رامز's user avatar
0 votes
1 answer
78 views

DagsterImportError: Encountered ImportError: `cannot import name 'URL' from 'datasets'

I am running dagster dev in a folder created by running dagster project scaffold --name dagster_pipeline: assets.py imports a constant from datasets.py that I have added: import requests from dagster ...
Albert Gevorgyan's user avatar
0 votes
0 answers
53 views

Get the previous scheduled parition date in a dagster asset

Is there a way to get the previous scheduled partition key in an asset? If i use the below in a scheduled asset it will return a partition key even if the should_execute in the schedule is False: @...
pyCthon's user avatar
  • 11.7k
0 votes
1 answer
95 views

Dagster UI not showing on Digital Ocean with nginx

I'm trying to run open-source Dagster on a digital ocean droplet. This droplet is working OK for MLFlow and accessing our database via phpmyadmin. When I try to access the Dagster UI via www.dev....
James Bradbury's user avatar
1 vote
0 answers
260 views

How to pass parameter to asset from op in dagster

I am new to dagster and having difficult in running the asset for different parameter values when job scheduled. I have created a pipeline using dagster. Trying to materialize the outcome of upstream ...
Hari's user avatar
  • 299
0 votes
1 answer
131 views

How to provide parameters defined at the source asset declaration to the IO Manager?

So my initial task was to create an IO Manager that should connect to a database and return data as pandas dataframe. (I am using dagster 1.3.10) Design IMO, the credentials (ip, port, user, password) ...
guillaume latour's user avatar
0 votes
1 answer
103 views

dagster: run every hour and pass last execution time to asset

I'm new to dagster to please be patient for not fully grasping it yet. I want to "run an asset" every hour which should be possible with schedules. What i also want is to pass the last ...
beginner_'s user avatar
  • 7,280
1 vote
1 answer
251 views

write test for dagster asset job

I am trying to write a simple test for a dagster job and I can't get it through... I am using dagster 1.3.6 So I have defined this job using the function dagster.define_asset_job from dagster import ...
guillaume latour's user avatar
0 votes
1 answer
210 views

Unable to open Dagster UI on local machine when dagster script is running from inside a docker container

I am running into an issue trying to open the Dagster UI from inside a docker container. Furthermore, this is a step towards my larger of goal, of opening the dagster UI from a docker container, thats ...
Paul Fentress's user avatar
0 votes
1 answer
292 views

create multiple jobs from similar assets with different partitions and schedules

I have a assets.py with the following: from dagster import asset, job, Output from my_modules import some_func from ..partitions import daily_partitions_def, t1_daily_partitions_def @asset(...
pyCthon's user avatar
  • 11.7k
0 votes
1 answer
167 views

Get Dagster Config Schema in Python

I am trying to programmatically kick off a Dagster job from my Python code. So far I am all set with using the python client to kick off a job with either the default or fully specified config. The ...
Jared Segal's user avatar
1 vote
0 answers
92 views

Docker AWS ECS integration on Mac NoCredentialProviders error when running "docker compose up"

When I am trying to run docker compose up to deploy my infrastructure to AWS using Docker's ECS integration. I'm getting the following error NoCredentialProviders: no valid providers in the chain. ...
user3843520's user avatar
0 votes
1 answer
76 views

How to create DAGs with Spark?

I'm new to Spark and I've realized that, for the pipeline I'm making, it would be much more convenient to have a DAG to represent the pipeline to improve monitoring, scheduling, etc. I connected Spark ...
Stan Shunpike's user avatar
0 votes
0 answers
95 views

Execute ECS task from Dagster

I am relatively new to Dagster. I am trying to execute an ECS task from dagster using this standalone python file as a job: from dagster import job, op @op() def my_op(context): context.log.info('...
jsanjayce's user avatar
  • 272
0 votes
2 answers
280 views

How to implement io_manager that have a parameter at asset level?

I am a bit confused about the usage of resources, configuration and how they are linked to a context and an asset. So I have a parquet io manager that is able to manipulate and partitionned not-...
guillaume latour's user avatar
2 votes
0 answers
222 views

How not to overwrite materialized assets in Dagster

I'm new to Dagster. According to the docs, when you use the built-in filesystem IO manager: Subsequent materializations of an asset will overwrite previous materializations of that asset. https://...
Alena Volkova's user avatar
-1 votes
1 answer
119 views

How can I partition dagster assets by year?

Not seeing a built in definition to partition assets by year (just hourly-monthly). Is there a way to manipulate the built in time definitions to accomplish this? Any help is appreciated!
Edan Coben's user avatar
0 votes
0 answers
805 views

cannot import name 'Definitions' from 'dagster'

I'm trying out this example project: https://github.com/dagster-io/dagster/tree/master/examples/project_fully_featured I added a workspace.yaml file with the lines: load_from: - python_module: ...
SerrHenry's user avatar
0 votes
1 answer
289 views

How do you explicitly name an Output in Dagster?

I have the following asset in Dagster: @asset def some_asset(): output = "some_great_output" return Output( value=output, ) As it is above, Dagster will name the ...
Alex's user avatar
  • 3,326
0 votes
1 answer
273 views

Reduce a partitioned asset to a single data frame

I have a software defined asset that compares a (very) large set of partitions to a single small data frame. I've made a toy explainer that shows the situation: partitions = dagster....
Mike Dewar's user avatar
  • 10.9k
0 votes
1 answer
360 views

How to load assets from dbt project? #12893

I'm trying to load assets from my dbt project called "dbtz", but i'm receiving the error hyprster:dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2)...
skulden's user avatar
  • 380
0 votes
1 answer
306 views

Dagster is detecting my code but not my sensors?

I have a simple local dagster project. One of its files is as follows: from dagster import asset, Definitions, build_asset_reconciliation_sensor, AssetSelection group_name = 'for_testing' @asset(...
MYK's user avatar
  • 1,998
0 votes
0 answers
126 views

Dagster is not recognizing schedule despite it running successfully

I have a local dagter project. I start the dagit & dagster-deamon via export DAGSTER_HOME="{path-to-project}/my-dagster-project/" export PYTHONLEGACYWINDOWSSTDIO="UTF-8" ...
MYK's user avatar
  • 1,998
0 votes
0 answers
252 views

Call downstream assets or ops in parallel with Dagster

I build a data pipeline with Dagster using the asset API. I am looking for a way to explicitly call/execute an asset (or op) for a list of items. While assets are executed in parallel if they are ...
Martin Preusse's user avatar
1 vote
1 answer
73 views

Dagster: handle TypedDict as op output

Context To keep track of the origin of my Dataframes, across my different ops, I defined TaggedDF as class TaggedDF(TypedDict): df: DataFrame ressource_name: str So I could @op(retry_policy=...
zar3bski's user avatar
  • 2,783
0 votes
2 answers
330 views

stream data between tasks in pipeline orchestration tool Prefect/Dagster/Airflow

How can I stream data between tasks in a workflow with the help of a data pipeline orchestration tool like Prefect, Dagster or Airflow? I am looking for a good data pipeline orchestration tool. I ...
phobic's user avatar
  • 914
0 votes
0 answers
197 views

Dagster orchestrator - How can I run commands to process dbt models in another Docker container?

I'm playing a bit with Dagster orchestrator and I have some doubts. My intention is use this tool only as a scheduler and runner for commands since I have the business logic developed in other ...
RgG's user avatar
  • 23
0 votes
1 answer
275 views

How to use dagster with great expectations?

The issue I'm trying out great expectations with dagster, as per this guide My pipeline seems to execute correctly until it reaches this block: expectation = dagster_ge.ge_validation_op_factory( ...
Imad's user avatar
  • 2,358
0 votes
0 answers
116 views

Providing the same resource to every op in a job

So I need myresource in every op. How do I do this without having to specify it in every op, but in the job definition instead? I think this is possible, but I cannot find a specification in the ...
user1943569's user avatar
1 vote
1 answer
98 views

How do I add the materialization runtime to a software defined asset in Dagster?

I would like to keep track of how long it takes to materialize software defined assets over time (using Dagster). Ideally I'd add the "duration" to the materialization metadata. I could do ...
MYK's user avatar
  • 1,998
3 votes
1 answer
872 views

dagster can you trigger a job to run via an api?

I have been looking all over for the answer, but can't seem to find what I'm looking for I want to create an api endpoint that can pass information to the dagster assets and trigger a run. For example,...
Nicholas Hansen-Feruch's user avatar
0 votes
1 answer
450 views

How do I write a Dagster asset that depends on an earlier partition of itself?

I was using depends_on_past with Airflow. I'm now using Dagster, with software-defined assets, and I was told that the way to get similar functionality is with build_asset_reconciliation_sensor and a ...
Sandy Ryza's user avatar
4 votes
2 answers
658 views

why does kubernetes delete secrets after helm upgrade?

when performing helm upgrade, I find that secrets that are created upon initial install are deleted. Why is this? The example I am using is dagster. When installing with: helm install dagster dagster/...
pomply's user avatar
  • 93
1 vote
1 answer
349 views

Dagster -Execute an @Op only when all parallel executions are finished(DynamicOutput)

I have a problem that in fact I am not able to solve in dagster. I have the following configuration: I have step 1 where I get the data from an endpoint step 2 gets a list of customers dynamically: ...
Cristiano Cardoso dos Santos's user avatar
1 vote
1 answer
398 views

Getting current execution date in a task or asset in dagster

Is there an easier way than what I'm doing to get the current date in an dagster asset, than what I'm currently doing? def current_dt(): return datetime.today().strftime('%Y-%m-%d') @asset def ...
pyCthon's user avatar
  • 11.7k
2 votes
1 answer
487 views

how to iterate over a list of values returning from ops to jobs in dagster

I am new to the dagster world and working on ops and jobs concepts. \ my requirement is to read a list of data from config_schema and pass it to @op function and return the same list to jobs. \ The ...
Piyush Jiwane's user avatar
1 vote
2 answers
569 views

How do I pass data to an op in a different module in dagster?

I am new to dagster and am having a difficult time sorting this one out. I have to jobs defined in my dagster pipeline and I want to pass data from an op in one job to an op in another My setup is as ...
walkingwithataco's user avatar
0 votes
0 answers
90 views

How to allow memoization for jobs defined from Dagster assets

I would like to use Memoization for jobs that I am defining from AssetsDefinitions with define_asset_job. The problem is: define_asset_job does not allow me to set-up a version_strategy. How can I ...
pedrovgp's user avatar
  • 767