Questions tagged [google-cloud-composer]

Google Cloud Composer is a fully managed workflow orchestration service, built on Apache Airflow, that empowers you to author, schedule, and monitor pipelines that span across clouds and on-premises data centers.

Filter by
Sorted by
Tagged with
0 votes
0 answers
25 views

Is there any way to get the latest S3 file by GCP StorageTransfer?

We're running Airflow/Composer on GCP and trying to get the latest S3 file by any python operator. Currently we're getting S3 file once a day, they can happen to export several files in the same day ...
Sachiko's user avatar
  • 840
1 vote
1 answer
67 views

PyPi package versions compatible with Cloud Composer's `apache-airflow-providers-microsoft-azure`

I'm attempting to load a few packages into an instance of Google Cloud Composer 2. I constantly getting dependency or package issues and am unable to get the versions of the packages that function. ...
user4933's user avatar
  • 1,515
0 votes
0 answers
35 views

How to list all connection & import a json list of connections in Cloud composer 2 Airflow 2

I am trying to list all connections in Cloud composer, view its value via cli command and also import another connections.json to Composer 2. I am not able to find it relevant CLI commands in the gcp ...
mehere's user avatar
  • 1,487
0 votes
0 answers
11 views

how do I enable debug logs for triggers?

I'm debugging an issue now, I have managed to enable debug logs for operators by updating airflow.cfg with [core] logging_level = DEBUG But I don't see any trigger logs in the task view. How can I ...
Dagang Wei's user avatar
  • 25.1k
0 votes
0 answers
28 views

GCP Cloud Composer access Airflow UI with token

Is there a way on GCP Cloud Composer to access Airflow UI without GCP/Google account only with constructing the link with some kind of token. AWS has it implemented this way: #!/bin/bash HOST=...
Nikola's user avatar
  • 1
0 votes
0 answers
26 views

BigQueryInsertJobOperator sometimes fails without logging Job Status

I have some bigquery tasks that follow this structure for the BigQueryInsertJobOperator: for phase in phases: trf_tasks[phase] = [] for entity in phases[phase]['entities']: trf_task ...
fneri's user avatar
  • 1
1 vote
0 answers
48 views

GCP Cloud Composer pods is forbidden: User system:anonymous cannot list resource pods in API group in the namespace \"customnamespace\"

I want to run a DAG on Cloud Composer but error message said kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': '-4fd3-44e8-80c2-...
MG_'s user avatar
  • 11
0 votes
1 answer
58 views

How do I invoke a Dataform workflow using assertions (Data Quality tests) Airflow via GCP Cloud Composer?

I need help to execute a (Data Quality) task with Dataform's assertions to UniqueKey in BigQuery using Cloud Composer's DAG. The task below executes the workflow, but I can't perform Data Quality ...
dsrebechi's user avatar
0 votes
0 answers
39 views

Automatically assign Google Groups to Airflow roles in GCP

I have a Google Cloud Composer instance (v2) whose configuration is managed by Terraform and whose dags are in the format: dags/ foo/ dag_1.py dag_2.py bar/ dag_3.py dag_4.py In ...
Chris BT's user avatar
0 votes
1 answer
27 views

Run DAG after TriggerDagRunOperator fails

I have a DAG which runs continously by using TriggerDagRunOperator.TriggerDagRunOperator triggers the same DAG upon completion. The schedule_interval is specified as 'once' and now when the ...
Aym's user avatar
  • 75
0 votes
1 answer
57 views

KubernetesPodOperator is saying forbidden in composer for default namespace?

Helo I'm receiving the following error, and I don't understand why. I believe it has to do with the service account, but the service account should have editor permission Objective Run an R script ...
moltke_colombia's user avatar
0 votes
1 answer
36 views

Error: ValueError: unknown url type: 'g'. Unable to connect to jdbc driver stored in GCS bucket by Dataflow script triggered by Cloud composer

I am unable to connect to jdbc driver (jar file) stored in GCS bucket by Dataflow script triggered by Cloud composer. I have a composer script & a dataflow script. Composer script: Triggers the ...
Viknesh S K's user avatar
0 votes
0 answers
43 views

Airflow: Rscript: command not found even though Docker image has R installed

so I have a bit of an issue where I'm receiving the error Rscript: command not found even though there was a succesful build of a Docker image involving R. Issue: Rscript: command not found while r-...
moltke_colombia's user avatar
0 votes
0 answers
26 views

How to enable GCP Composer UI access by environment name

I have multiple cloud composer environments in the same GCP project. I want to give a given user access only to one of these environments. I already created a role with the following permissions: ...
GabrielBoehme's user avatar
1 vote
0 answers
41 views

Airflow - Compute Engine SSH Hook creates a new pair of keys everytime it connects to VM

We have Airflow 2.5.3 running on GCP Cloud Composer (2.4.6). Around 200 tasks need to be daily executed on a VM located on the same project and VPC. Thus, we need to use an SSHOperator to execute the ...
WillDCC's user avatar
  • 21
0 votes
1 answer
49 views

How to install AWS CLI on Cloud Composer?

I need to install AWS CLI tool on Google Cloud Composer to be able to use it with BashOperator from Airflow DAGs. AWS CLI documentation page explains how to install it as a package, but Cloud Composer ...
vilozio's user avatar
  • 135
0 votes
1 answer
54 views

CloudSqlExecuteQueryOperator does not return results to xcom

I have an airflow pipeline that does an UPSERT to a postgresql database on GCP cloud sql. Now, before I was using the PostgresOperator on a local database and everything was working fine, but needed ...
CONNOR TRUEX's user avatar
0 votes
4 answers
83 views

Cannot access task instance to get xcom

I am new to using xcoms in Airflow. Here's what my code looks like: with DAG(default_args=default_args, max_active_runs=1, dag_id="my_dag", catchup=False, ...
FrustratedWithFormsDesigner's user avatar
0 votes
0 answers
38 views

.airflowignore file is not able to hide DAG

We have airflow with GCP composer of version 2.2.3. In airflow we have 2K DAG files where only 100 are Parent Dag's and need to be visible. Rest all are part of taskgroup. But on airflow UI all 2K ...
Manoj Sonavane's user avatar
0 votes
0 answers
50 views

Export from AlloyDB in Composer Airflow - SSH connection difficulties

The issue I am encountering is how to create an SSH tunnel using IAP for AlloyDB export in Airflow hosted on Google Composer. I am replacing a CloudSQL connection with AlloyDB. The composer ...
gtcat's user avatar
  • 1
0 votes
1 answer
83 views

Not able to send Composer(airflow job) to dataproc

Im trying to submit an airflow job in composer (gcp) to dataproc. When i do this i get the following error ask failed with exception Traceback (most recent call last): File "/opt/python3.8/lib/...
luckyman80's user avatar
0 votes
0 answers
61 views

raise DuplicateTaskIdFound(f"Task id '{key}' has already been added to the DAG")

def getDate(**kwargs): big_query_task = BigQueryExecuteQueryOperator( dag = dag, task_id='del_data_'+table, sql=''' #standardSQL delete from `{}` where {} ...
Uyyan's user avatar
  • 21
0 votes
0 answers
29 views

Unknown problem with connection to Airflow Database, server closed connection unexpectedly, GCP Cloud Composer

I have daily ingest pipeline as production runs on Cloud Composer, the code is unchanged for a while. One day there is one error that raised from database_health.py that raised "server closed ...
Chanan Chanachon's user avatar
0 votes
0 answers
30 views

Why am i not able to extract data on airflow from mongo?

I am extract a data from mongo collection, which works completely fine on my local system but when i upload it gcp and run my DAG it fails because my collection return zero data, why is it happening? ...
skye's user avatar
  • 67
0 votes
1 answer
70 views

How to pass a nested dictionaries as env variables to Kubernetes Pod Operator

In dag script PARAM = { 'key1' : 'value1', 'key2' : 'value2' } t1 = KubernetesPodOperator( task_id=task_id, name=task_name, cmds=["pipenv", "run", "...
zZzZ's user avatar
  • 145
0 votes
2 answers
168 views

Airflow unable to mount a google cloud bucket using gcsfuse

I want to mount a Google Cloud Bucket into my airflow environment so that I can read and write files on that GCS Bucket. I am using Cloud Composer 2 (composer-2.1.14-airflow-2.5.1 image) In airflow I ...
Virendra's user avatar
  • 2,570
1 vote
0 answers
103 views

Connecting SQL database to Airflow in Google Cloud Composer

hope all are well. I have a DAG running in the Google Cloud Composer. But, I am having an issue connecting sql db for the DAG. One way I tried was to export my local SQLite db file to Google Cloud ...
knk's user avatar
  • 47
0 votes
0 answers
29 views

Cloud Composer meta-database keeps increasing even with cleanup script

I am running a Cloud Composer environment on GCP. I see that the meta-database size keeps increasing. I have followed the documentation's instructions here: https://cloud.google.com/composer/docs/...
Tsh's user avatar
  • 1
0 votes
1 answer
13 views

SpannerQueryDatabaseInstanceOperator how to perform a parametrized query

From the documentation SpannerQueryDatabaseInstanceOperator accepts a query parameter. However there's no something smart like the PostgresOperator that accepts also a parameters parameter to use ...
Momo's user avatar
  • 345
0 votes
0 answers
14 views

Google Cloud Airflow Composer

I am trying to trigger a python script that i upload in GCS bucket, using Airflow composer. The task was sucess but actually it did non run and new data did not created. I tried to use BashOperator or ...
mikenguyen1712's user avatar
0 votes
0 answers
39 views

Most efficient way to parameterize variables in Cloud Composer?

CONTEXT I'm currently using Airflow 2.4.3 in Google Cloud Composer with Python 3.8.12. I have about 100+ DAGs with approximately 4-8 tasks each and almost each task is accessing a value in Airflow ...
pfhermosa's user avatar
2 votes
0 answers
47 views

Pulling XCOM when creating a dataproc cluster with DataprocCreateClusterOperator in airflow

I'm trying to pull an XCOM variable when creating a dataproc cluster in airflow with DataprocCreateClusterOperator. I need it because I need a variable that I am pulling by querying an API and saving ...
Dominno's user avatar
  • 21
1 vote
1 answer
83 views

Airflow task fails after sending notification to PubSub via Airflow Plugin - Error with psycopg2 and SQLAlchemy

General description I'm encountering an issue with my custom Apache Airflow plugin which sends notifications to Google Cloud PubSub upon various task events. After the PubSub notification is sent ...
dadadima's user avatar
  • 938
0 votes
0 answers
207 views

Airflow Worker giving error (Celery command failed on host) after successfully completing task in GCP Cloud Composer

My task completes successfully in the airflow worker. But when I check the logs I always see this error after successful completion. The actual task/dag does not fails, but still It gives following ...
Harbeer Kadian's user avatar
0 votes
1 answer
55 views

Airflow 2.5.3 - invalid source format 'parquet' is not valid value

I'm migrating to Composer version 2.4.1 and AIrflow 2.5.3. Some DAGS that I use the BigQueryCreateExternalTableOperator are returning an error when creating the external table, I am passing the file ...
Gustavo Lima Monteiro's user avatar
0 votes
1 answer
219 views

Google Cloud Composer failed to connect to Cloud SQL PostgreSQL database

I am creating a Google Cloud Composer to use Apache Airflow CloudSQLExecuteQueryOperator to sumbit a simple query to Cloud SQL PostgreSQL database. But it failed with the following error message: ...
Albert's user avatar
  • 11
1 vote
0 answers
210 views

Why Repository could not be reached in Dataform, with connection to Gitlab

I have implement data transformation pipeline on Google Cloud Composer, by using "DataformCreateCompilationResultOperator" to create comilation on Dataform "...
Chanan Chanachon's user avatar
0 votes
0 answers
74 views

Airflow task status is failed but the task is running

One of my DAGs is shown as success but one task is failed. But this task is still running. The task which should run after it is already finished and a success. When the "failed" task is ...
TheMB's user avatar
  • 1
0 votes
0 answers
134 views

Issue with Airflow Composer Upgrade to composer-2.3.5-airflow-2.5.3

I'm currently facing an issue after upgrading my GCP Airflow Composer from version 2.2.4 to 2.5.3. I'm reaching out to seek your assistance in resolving this issue. After the upgrade, I encountered ...
Dư Châu's user avatar
0 votes
1 answer
105 views

Import bigquery_to_postgres on GCP Cloud composer

I would like to use BigQueryToPostgresOperator in my Composer Dag in GCP. I am getting: ImportError: cannot import name 'bigquery_to_postgres' from 'airflow.providers.google.cloud.transfers' (/opt/...
p.magalhaes's user avatar
  • 7,757
0 votes
0 answers
21 views

Can two tasks in Airflow 1.X have the same name?

I have a DAG and a backfill DAG. They have different dag ids but same task names. I found out that once I pushed the backfill DAG to production, Airflow immediately removed the other one from Airflow ...
Marcus Anthony's user avatar
0 votes
1 answer
44 views

Does Composer not support comments in requriements.txt?

This is my requirements.txt # This is a comment python-json-logger==2.0.4 # Here's a private package my_private_pypi_package==1.2.3 I get an error: > gcloud composer environments update myenv --...
red888's user avatar
  • 28.6k
0 votes
0 answers
38 views

How do I check if there are changes in pip dependencies before updating and installing them?

Installing a package like this (or pointing to a requirements file) causes AF to trigger an environment update: gcloud composer environments update myenv \ --location us-central1 \ --project ...
red888's user avatar
  • 28.6k
0 votes
0 answers
23 views

Is there a way to make Public Composer Instances to Private Without Recreating Environments in GCP?

I am currently facing a challenge with our public Composer instances on Google Cloud Platform. We want to make these instances private and restrict public IPs without going through the process of ...
Vamshi_krishna's user avatar
0 votes
1 answer
110 views

How do I programmatically check a composer image version is valid before creating a new environment?

The following image version is not valid: composer-2.3.3-airflow-2.5.1 If you try to create a new environment (gcloud composer environments create) with that image it will hang for 40+ minutes and ...
red888's user avatar
  • 28.6k
0 votes
0 answers
221 views

GCP (Google Cloud) Kubernetes Pod Opeartors (that run DBT models) Error when upgrading Cloud Composer Environment despite same IAM policies

I recently upgraded our GCP cloud composer environment that has DAGs that uses Kubernetes Pods to run DBT models that transform BQ from: composer-2.0.28-airflow-2.2.5 (Image 1) to: composer-2.1.11-...
Cloud Boi's user avatar
0 votes
1 answer
383 views

Cloud Composer GKE coupling broken since composer-2.3.5-airflow-2.5.3

Since upgrading to the latest composer (composer-2.3.5-airflow-2.5.3 ), our all our long-running GKE pods seem to fail. Example error: [2023-08-01, 14:20:23 CEST] {before.py:40} INFO - Starting call ...
Jonny5's user avatar
  • 1,429
0 votes
1 answer
107 views

How can I update multiple configuration fields of a composer environment at once?

When I try to run gcloud composer environments update and specify --update-labels and --update-env-variables I get an error: Composer deployment failed with error: ERROR: (gcloud.composer.environments....
red888's user avatar
  • 28.6k
0 votes
1 answer
574 views

Scaling GCP Cloud Composer environment size

I am using Cloud Composer 2. I understand the purpose of scaling the number of workers, their compute and their memory. I also understand why I would want additional schedulers (mostly resiliency, I'...
Mike Williamson's user avatar
2 votes
1 answer
142 views

Where is the actual documentation for the composer python SDK?

Am I crazy or is this documentation non-existent: https://cloud.google.com/python/docs/reference/composer/latest "Code samples and snippets live in the samples/ folder." What samples folder? ...
red888's user avatar
  • 28.6k

1
2 3 4 5
25