Questions tagged [google-cloud-composer]
Google Cloud Composer is a fully managed workflow orchestration service, built on Apache Airflow, that empowers you to author, schedule, and monitor pipelines that span across clouds and on-premises data centers.
1,238
questions
0
votes
0
answers
25
views
Is there any way to get the latest S3 file by GCP StorageTransfer?
We're running Airflow/Composer on GCP and trying to get the latest S3 file by any python operator.
Currently we're getting S3 file once a day, they can happen to export several files in the same day ...
1
vote
1
answer
67
views
PyPi package versions compatible with Cloud Composer's `apache-airflow-providers-microsoft-azure`
I'm attempting to load a few packages into an instance of Google Cloud Composer 2. I constantly getting dependency or package issues and am unable to get the versions of the packages that function.
...
0
votes
0
answers
35
views
How to list all connection & import a json list of connections in Cloud composer 2 Airflow 2
I am trying to list all connections in Cloud composer, view its value via cli command and also import another connections.json to Composer 2.
I am not able to find it relevant CLI commands in the gcp ...
0
votes
0
answers
11
views
how do I enable debug logs for triggers?
I'm debugging an issue now, I have managed to enable debug logs for operators by updating airflow.cfg with
[core]
logging_level = DEBUG
But I don't see any trigger logs in the task view. How can I ...
0
votes
0
answers
28
views
GCP Cloud Composer access Airflow UI with token
Is there a way on GCP Cloud Composer to access Airflow UI without GCP/Google account only with constructing the link with some kind of token.
AWS has it implemented this way:
#!/bin/bash
HOST=...
0
votes
0
answers
26
views
BigQueryInsertJobOperator sometimes fails without logging Job Status
I have some bigquery tasks that follow this structure for the BigQueryInsertJobOperator:
for phase in phases:
trf_tasks[phase] = []
for entity in phases[phase]['entities']:
trf_task ...
1
vote
0
answers
48
views
GCP Cloud Composer pods is forbidden: User system:anonymous cannot list resource pods in API group in the namespace \"customnamespace\"
I want to run a DAG on Cloud Composer but error message said
kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': '-4fd3-44e8-80c2-...
0
votes
1
answer
58
views
How do I invoke a Dataform workflow using assertions (Data Quality tests) Airflow via GCP Cloud Composer?
I need help to execute a (Data Quality) task with Dataform's assertions to UniqueKey in BigQuery using Cloud Composer's DAG. The task below executes the workflow, but I can't perform Data Quality ...
0
votes
0
answers
39
views
Automatically assign Google Groups to Airflow roles in GCP
I have a Google Cloud Composer instance (v2) whose configuration is managed by Terraform and whose dags are in the format:
dags/
foo/
dag_1.py
dag_2.py
bar/
dag_3.py
dag_4.py
In ...
0
votes
1
answer
27
views
Run DAG after TriggerDagRunOperator fails
I have a DAG which runs continously by using TriggerDagRunOperator.TriggerDagRunOperator triggers the same DAG upon completion. The schedule_interval is specified as 'once' and now when the ...
0
votes
1
answer
57
views
KubernetesPodOperator is saying forbidden in composer for default namespace?
Helo I'm receiving the following error, and I don't understand why. I believe it has to do with the service account, but the service account should have editor permission
Objective Run an R script ...
0
votes
1
answer
36
views
Error: ValueError: unknown url type: 'g'. Unable to connect to jdbc driver stored in GCS bucket by Dataflow script triggered by Cloud composer
I am unable to connect to jdbc driver (jar file) stored in GCS bucket by Dataflow script triggered by Cloud composer. I have a composer script & a dataflow script.
Composer script: Triggers the ...
0
votes
0
answers
43
views
Airflow: Rscript: command not found even though Docker image has R installed
so I have a bit of an issue where I'm receiving the error Rscript: command not found even though there was a succesful build of a Docker image involving R.
Issue: Rscript: command not found while r-...
0
votes
0
answers
26
views
How to enable GCP Composer UI access by environment name
I have multiple cloud composer environments in the same GCP project. I want to give a given user access only to one of these environments. I already created a role with the following permissions:
...
1
vote
0
answers
41
views
Airflow - Compute Engine SSH Hook creates a new pair of keys everytime it connects to VM
We have Airflow 2.5.3 running on GCP Cloud Composer (2.4.6). Around 200 tasks need to be daily executed on a VM located on the same project and VPC. Thus, we need to use an SSHOperator to execute the ...
0
votes
1
answer
49
views
How to install AWS CLI on Cloud Composer?
I need to install AWS CLI tool on Google Cloud Composer to be able to use it with BashOperator from Airflow DAGs.
AWS CLI documentation page explains how to install it as a package, but Cloud Composer ...
0
votes
1
answer
54
views
CloudSqlExecuteQueryOperator does not return results to xcom
I have an airflow pipeline that does an UPSERT to a postgresql database on GCP cloud sql. Now, before I was using the PostgresOperator on a local database and everything was working fine, but needed ...
0
votes
4
answers
83
views
Cannot access task instance to get xcom
I am new to using xcoms in Airflow.
Here's what my code looks like:
with DAG(default_args=default_args,
max_active_runs=1,
dag_id="my_dag",
catchup=False,
...
0
votes
0
answers
38
views
.airflowignore file is not able to hide DAG
We have airflow with GCP composer of version 2.2.3.
In airflow we have 2K DAG files where only 100 are Parent Dag's and need to be visible. Rest all are part of taskgroup.
But on airflow UI all 2K ...
0
votes
0
answers
50
views
Export from AlloyDB in Composer Airflow - SSH connection difficulties
The issue I am encountering is how to create an SSH tunnel using IAP for AlloyDB export in Airflow hosted on Google Composer.
I am replacing a CloudSQL connection with AlloyDB. The composer ...
0
votes
1
answer
83
views
Not able to send Composer(airflow job) to dataproc
Im trying to submit an airflow job in composer (gcp) to dataproc. When i do this i get the following error
ask failed with exception Traceback (most recent call last): File "/opt/python3.8/lib/...
0
votes
0
answers
61
views
raise DuplicateTaskIdFound(f"Task id '{key}' has already been added to the DAG")
def getDate(**kwargs):
big_query_task = BigQueryExecuteQueryOperator(
dag = dag,
task_id='del_data_'+table,
sql='''
#standardSQL
delete from `{}` where {} ...
0
votes
0
answers
29
views
Unknown problem with connection to Airflow Database, server closed connection unexpectedly, GCP Cloud Composer
I have daily ingest pipeline as production runs on Cloud Composer, the code is unchanged for a while. One day there is one error that raised from database_health.py that raised "server closed ...
0
votes
0
answers
30
views
Why am i not able to extract data on airflow from mongo?
I am extract a data from mongo collection, which works completely fine on my local system but when i upload it gcp and run my DAG it fails because my collection return zero data, why is it happening?
...
0
votes
1
answer
70
views
How to pass a nested dictionaries as env variables to Kubernetes Pod Operator
In dag script
PARAM = {
'key1' : 'value1',
'key2' : 'value2'
}
t1 = KubernetesPodOperator(
task_id=task_id,
name=task_name,
cmds=["pipenv", "run", "...
0
votes
2
answers
168
views
Airflow unable to mount a google cloud bucket using gcsfuse
I want to mount a Google Cloud Bucket into my airflow environment so that I can read and write files on that GCS Bucket. I am using Cloud Composer 2 (composer-2.1.14-airflow-2.5.1 image)
In airflow I ...
1
vote
0
answers
103
views
Connecting SQL database to Airflow in Google Cloud Composer
hope all are well. I have a DAG running in the Google Cloud Composer. But, I am having an issue connecting sql db for the DAG.
One way I tried was to export my local SQLite db file to Google Cloud ...
0
votes
0
answers
29
views
Cloud Composer meta-database keeps increasing even with cleanup script
I am running a Cloud Composer environment on GCP.
I see that the meta-database size keeps increasing.
I have followed the documentation's instructions here: https://cloud.google.com/composer/docs/...
0
votes
1
answer
13
views
SpannerQueryDatabaseInstanceOperator how to perform a parametrized query
From the documentation SpannerQueryDatabaseInstanceOperator accepts a query parameter. However there's no something smart like the PostgresOperator that accepts also a parameters parameter to use ...
0
votes
0
answers
14
views
Google Cloud Airflow Composer
I am trying to trigger a python script that i upload in GCS bucket, using Airflow composer. The task was sucess but actually it did non run and new data did not created. I tried to use BashOperator or ...
0
votes
0
answers
39
views
Most efficient way to parameterize variables in Cloud Composer?
CONTEXT
I'm currently using Airflow 2.4.3 in Google Cloud Composer with Python 3.8.12. I have about 100+ DAGs with approximately 4-8 tasks each and almost each task is accessing a value in Airflow ...
2
votes
0
answers
47
views
Pulling XCOM when creating a dataproc cluster with DataprocCreateClusterOperator in airflow
I'm trying to pull an XCOM variable when creating a dataproc cluster in airflow with DataprocCreateClusterOperator. I need it because I need a variable that I am pulling by querying an API and saving ...
1
vote
1
answer
83
views
Airflow task fails after sending notification to PubSub via Airflow Plugin - Error with psycopg2 and SQLAlchemy
General description
I'm encountering an issue with my custom Apache Airflow plugin which sends notifications to Google Cloud PubSub upon various task events. After the PubSub notification is sent ...
0
votes
0
answers
207
views
Airflow Worker giving error (Celery command failed on host) after successfully completing task in GCP Cloud Composer
My task completes successfully in the airflow worker. But when I check the logs I always see this error after successful completion. The actual task/dag does not fails, but still It gives following ...
0
votes
1
answer
55
views
Airflow 2.5.3 - invalid source format 'parquet' is not valid value
I'm migrating to Composer version 2.4.1 and AIrflow 2.5.3. Some DAGS that I use the BigQueryCreateExternalTableOperator are returning an error when creating the external table, I am passing the file ...
0
votes
1
answer
219
views
Google Cloud Composer failed to connect to Cloud SQL PostgreSQL database
I am creating a Google Cloud Composer to use Apache Airflow CloudSQLExecuteQueryOperator to sumbit a simple query to Cloud SQL PostgreSQL database. But it failed with the following error message:
...
1
vote
0
answers
210
views
Why Repository could not be reached in Dataform, with connection to Gitlab
I have implement data transformation pipeline on Google Cloud Composer, by using
"DataformCreateCompilationResultOperator" to create comilation on Dataform
"...
0
votes
0
answers
74
views
Airflow task status is failed but the task is running
One of my DAGs is shown as success but one task is failed. But this task is still running. The task which should run after it is already finished and a success. When the "failed" task is ...
0
votes
0
answers
134
views
Issue with Airflow Composer Upgrade to composer-2.3.5-airflow-2.5.3
I'm currently facing an issue after upgrading my GCP Airflow Composer from version 2.2.4 to 2.5.3. I'm reaching out to seek your assistance in resolving this issue.
After the upgrade, I encountered ...
0
votes
1
answer
105
views
Import bigquery_to_postgres on GCP Cloud composer
I would like to use BigQueryToPostgresOperator in my Composer Dag in GCP. I am getting:
ImportError: cannot import name 'bigquery_to_postgres' from
'airflow.providers.google.cloud.transfers'
(/opt/...
0
votes
0
answers
21
views
Can two tasks in Airflow 1.X have the same name?
I have a DAG and a backfill DAG. They have different dag ids but same task names. I found out that once I pushed the backfill DAG to production, Airflow immediately removed the other one from Airflow ...
0
votes
1
answer
44
views
Does Composer not support comments in requriements.txt?
This is my requirements.txt
# This is a comment
python-json-logger==2.0.4
# Here's a private package
my_private_pypi_package==1.2.3
I get an error:
> gcloud composer environments update myenv --...
0
votes
0
answers
38
views
How do I check if there are changes in pip dependencies before updating and installing them?
Installing a package like this (or pointing to a requirements file) causes AF to trigger an environment update:
gcloud composer environments update myenv \
--location us-central1 \
--project ...
0
votes
0
answers
23
views
Is there a way to make Public Composer Instances to Private Without Recreating Environments in GCP?
I am currently facing a challenge with our public Composer instances on Google Cloud Platform. We want to make these instances private and restrict public IPs without going through the process of ...
0
votes
1
answer
110
views
How do I programmatically check a composer image version is valid before creating a new environment?
The following image version is not valid: composer-2.3.3-airflow-2.5.1
If you try to create a new environment (gcloud composer environments create) with that image it will hang for 40+ minutes and ...
0
votes
0
answers
221
views
GCP (Google Cloud) Kubernetes Pod Opeartors (that run DBT models) Error when upgrading Cloud Composer Environment despite same IAM policies
I recently upgraded our GCP cloud composer environment that has DAGs that uses Kubernetes Pods to run DBT models that transform BQ from:
composer-2.0.28-airflow-2.2.5 (Image 1)
to:
composer-2.1.11-...
0
votes
1
answer
383
views
Cloud Composer GKE coupling broken since composer-2.3.5-airflow-2.5.3
Since upgrading to the latest composer (composer-2.3.5-airflow-2.5.3
), our all our long-running GKE pods seem to fail.
Example error:
[2023-08-01, 14:20:23 CEST] {before.py:40} INFO - Starting call ...
0
votes
1
answer
107
views
How can I update multiple configuration fields of a composer environment at once?
When I try to run gcloud composer environments update and specify --update-labels and --update-env-variables I get an error:
Composer deployment failed with error: ERROR: (gcloud.composer.environments....
0
votes
1
answer
574
views
Scaling GCP Cloud Composer environment size
I am using Cloud Composer 2.
I understand the purpose of scaling the number of workers, their compute and their memory. I also understand why I would want additional schedulers (mostly resiliency, I'...
2
votes
1
answer
142
views
Where is the actual documentation for the composer python SDK?
Am I crazy or is this documentation non-existent: https://cloud.google.com/python/docs/reference/composer/latest
"Code samples and snippets live in the samples/ folder." What samples folder?
...