Questions tagged [google-cloud-dataproc]
Google Cloud Dataproc is a managed Hadoop MapReduce, Spark, Pig and Hive service on Google Cloud Platform. The service provides GUI, CLI and HTTP API access modes for deploying/managing clusters and submitting jobs onto clusters.
1,470
questions
0
votes
0
answers
25
views
Connecting to MongoDB / Atlas from pyspark (DataProc) is getting Error
Getting the Below error when trying to connect to mongodb via PySpark(Google DATAPROC), its happening both locally and in Dataproc in GCP. We’re getting the below errors:
py4j.protocol.Py4JJavaError:...
1
vote
0
answers
41
views
GCP Dataproc version control
We work on jupyter notebooks interchangeably between Dataproc and on local computer. Usually, we write and test the code on a smaller sample locally and run it on all of the data in Dataproc. However, ...
0
votes
0
answers
14
views
MimeUtility decode code can't work on cloud
I have a hive udf for cleaning text, the text is encoded with "quoted-printable", this is the code I refer from online:
InputStream is = new ByteArrayInputStream(text.getBytes());
...
0
votes
0
answers
21
views
Gateway timeout error when opening JupyterLab
i have a cluster in Google Cloud with a static IP set up. This has been working fine more than a year. Recently I noticed that when I start the cluster and try to open JupyterLabs I get a gateway ...
-1
votes
0
answers
15
views
Pyspark jobs on dataproc using documents from Firestore
I need to run some simple Pyspark jobs on Big Data that are stored in Google's Firestore.
The dataset contains 42 million documents regarding Instagram posts. I want to do some simple aggregations ...
0
votes
1
answer
53
views
How to trigger an event to monitor dataproc job status in cloud function?
I am looking for some way by which I can track my dataproc spark job status from google cloud function in real time. I am not aware of any google cloud service which can fit in this situation similar ...
0
votes
0
answers
31
views
How to trigger GCP cloud function when data proc Job state change and read the the events
I am trying to figure out the equivalent of AWS lambda EventBridge (CloudWatch Events) where the lambda is triggered when a glue job state changed. But In GCP I am not found a way how to do it.
If ...
0
votes
1
answer
38
views
how to pass custom job id via google dataproc cluster job for spark using dataproc client
i am using the following code snippet but would not found any luck.
can anyone help me to pass custom job ID
job = {
"placement": {"cluster_name": cluster_name},
"...
1
vote
0
answers
18
views
Sqoop MDSYS.SDO_GEOMETRY columns in GCP DataProc
I am trying to sqoop MDSYS.SDO_GEOMETRY column into GCS Bucket using dataproc but sqoop is ignoring MDSYS.SDO_GEOMETRY column from selection. I am not sure what is the issue. Also Do I need to convert ...
1
vote
1
answer
37
views
How to get DB password from Azure app vault using python ? I am running this python file on google Dataproc cluster
My Sql server DB password is saved on Azure app vault which has DATAREF ID as a identifier. I need that password to create spark dataframe from table which is present in SQL server. I am running this ....
1
vote
0
answers
64
views
How to write in GCS using spark java in dataproc?
I am trying to use GCP to run spark application, i am using dataproc to trigger application and want to write output in Google Cloud Storage for this i have GCS connector added in my pom.xml
<!-- ...
3
votes
0
answers
129
views
Googld cloud dataproc serverless (batch) pyspark reads parquet file from google cloud storage (GCS) very slow
I have an inverse frequency parquet file of the wiki corpus on Google Cloud Storage (GCS). I want to load it from GCS to dataproc serverless (batch). However, the time to load the parquet with pyspark....
1
vote
2
answers
61
views
How to read csv file from GCS using spark-java?
I am trying to read csv file which is stored in GCS using spark,
I have a simple spark java project which does nothing but reading a csv.
the following code are used in it.
SparkConf conf = new ...
1
vote
0
answers
22
views
how to install python module on dataproc cluster executors
how can we import xmltodict module to all worker nodes of cluster, now it's giving me error as ' no name module ' while performing xmltodict on pyspark dataframe
xmltodict python module should be ...
2
votes
0
answers
65
views
Run Spark on GKE vs Dataproc on GKE
We are planning to move our Spark Structured Streaming jobs from Cloudera to GCP. My question is similar to the one asked here. But since it has been a while that the question is active, we are ...
3
votes
1
answer
80
views
Error in submitting a pig job to Google Dataproc with properties file
I'm new to Dataproc and trying to submit a pig job to google dataproc via gcloud
gcloud config set project PROJECT
gcloud dataproc jobs submit pig --cluster=cluster-workaround --region=us-...
1
vote
0
answers
35
views
Passing Database Password to Dataproc PySpark batch job
I want to create a pyspark job, and I want to have two separate environments: sandbox and production. Based on the environment it runs in, the script should connect either to the sandbox or production ...
2
votes
0
answers
72
views
GCP Dataproc - unable to access GCP Cloud storage
I have a Dataproc(Spark Structured Streaming) job which takes data from Kafka, and does some processing. The checkpoint is a GCP Cloud storage, and it is somehow unable to list the objects in GCP ...
1
vote
1
answer
49
views
How to specify pre-emptible SPOT VMs in Airflow cluster creation operator
Cluster creation through GCP console or GCP API provides an option to specify secondary workers[SPOT, pre-emptible or non-preemptible]. However, not able to find the corresponding CLUSTER_CONFIG to ...
1
vote
0
answers
37
views
Spark Pi Applications running on Ideal Dataproc Cluster
I witness a lot of Spark Pi application runs on my ideal Dataproc Cluster. These runs took place every 5 minutes on different workers and master node. My cluster is linked to a PHS, these Spark Pi ...
2
votes
1
answer
67
views
Pyspark monitoring metrics not making sense
I am trying to understand the spark ui and hdfs ui while using pyspark.
Following are my properties for the Session that I am running
pyspark --master yarn --num-executors 4 --executor-memory 6G --...
1
vote
0
answers
39
views
Dataproc: How to implement auto scale based on presto load
I am applying presto optionally to Dataproc.
I want to autoscale based on the load of the presto query, but when I set autoscaling-policies and increase the query load, it does not autoscale
I read ...
2
votes
1
answer
52
views
How to set --num-executors in dataproc cluster
We have a dataproc cluster with 10 Nodes and unable to understand how to set the parameter for --num-executor for spark jobs.
Below are the points which are confusing -
Can we have less executor than ...
1
vote
0
answers
101
views
Getting error while reading BIGNUMERIC data type from a BigQuery table using Apache Spark
I have a Dataproc Spark job which reads data from a Big Query table. The Big Query table is having a column of type BIGNUMERIC. Spark is able to read from the table successfully but the problem arises ...
1
vote
2
answers
160
views
Pass dynamic args to DataprocSubmitJobOperator from xcom
I am trying to receive an event from pub/sub and based on the message, it should pass some arguments to my dataproc spark job.
Now job_args in below code is dictionary. I have managed to push job_args ...
1
vote
1
answer
54
views
How to create SPOT VM's in my secondary_worker_config in airflow DAG for using google cloud dataproc operators?
In UI, you can select Preemptibility as Spot in the Secondary worker nodes to create something similar:
Following is my JSON:
"secondary_worker_config": {
"num_instances&...
3
votes
1
answer
189
views
Error Using DeltaTable.forPath on Pyspark Job in cloud Dataproc
I am executing some pyspark jobs on Dataproc cluster. All went fine until yesterday. However, today I started having this error while using command DeltaTable.forPath(sparkSession, path) to read delta ...
1
vote
0
answers
73
views
How to upload CSV files over the GCP (automated process)
I have GCP server and I want to upload CSV files in an automated way.
I want to upload all the old CSV file (existing files) over the GCP (BigQuery Table/View)
I want to upload all new arriving ...
1
vote
1
answer
90
views
Apache Hudi create and append Upsert table (Parquet-format) on Dataproc & Cloud Storage
is Dataproc-noob again.
My main goal is to ingest the tables from on-premise sources, store them as a Parquet-file in a Cloud Storage bucket and create/update tables in BigQuery from this file, ...
1
vote
1
answer
66
views
Reading CSV file with Spark runs sometimes forever
i'm using Spark 2.4.8 with the gcs-connector from com.google.cloud.bigdataoss in version hadoop2-2.1.8. For development i'm using a Compute Engine VM with my IDE. I try to consume some CSV files from ...
2
votes
0
answers
69
views
dataproc - unable to read archive files in pyspark
I run my pyspark code using the below command in dataproc.
gcloud dataproc batches submit pyspark src/features/spark_script.py \
--project=$PROJECT_ID \
--region=$REGION \
--deps-bucket=$...
2
votes
1
answer
130
views
Apache Hudi on Dataproc
Is there any guide to deploy Apache Hudi on a Dataproc Cluster? i´m trying to deploy via
Hudi Quick Start Guide but i can´t.
Spark 3.1.1
Python 3.8.13
Debian 5.10.127 x86_64
launch code:
pyspark --...
1
vote
1
answer
33
views
Reducing Dataproc Serverless CPU quota
Aim: I want to run spark jobs on Dataproc Serverless for Spark.
Problem: The minimum CPU cores requirement is 12 cores for a Spark app. That doesn't fit into the default regional CPU quota we have and ...
1
vote
1
answer
26
views
Why is spark process on my cluster even before I have started my sparksession?
I have set up a dataproc cluster to run my spark jobs on. I have just set up the cluster and have not started any spark session yet. Still, I am seeing spark process, mapreduce process, yarn etc in my ...
1
vote
1
answer
50
views
Where can we see the billing details or cost incurred details for each dataproc cluster in GCP console
I wanted to analyze the actual cost of each Dataproc cluster by having multiple machine types but I dont see any option to get the details of cost incurred of a single dataproc cluster in a GCP ...
1
vote
1
answer
49
views
Dataproc - SparkSession.builder.appName not reflected on dataproc properties
I am running a PySpark application using Dataproc Serverless for Spark, and my config file looks like this:
spark = (
pyspark.sql.SparkSession.builder.appName("app_name")
.config(...
2
votes
1
answer
159
views
Spark --archives file not found error, exception from executor
I am submitting a Spark job using Dataproc Serverless. My Spark code uses a few .yaml files as configuration and I pass them as --archives to the code.
Command to run the code:
gcloud dataproc batches ...
1
vote
0
answers
34
views
How to list cluster in Dataproc with Java API
I need to access dataproc by authenticating with a Service Account json key and get the list of clusters, through a command quite similar to "gcloud dataproc clusters list". I am using the ...
1
vote
1
answer
76
views
Dataproc on GKE via Terraform not working (example provided by Terraform doc)
I was doing some test in my GCP project to verify if i can migrate Dataproc on GKE and keep it up and running, while leveraging on auto scaling for workloads.
However, i'm blocked since teh beginning.
...
1
vote
1
answer
74
views
dataframe.show() not work in Pyspark inside a Debian VM (Dataproc)
currently using GCP and Dataproc, i´m new to apache spark, pyspark and debian vm. So, i´m trying to replicated inside dataproc cluster (Debian VM) a spark job that i run perfectly in my local machine (...
2
votes
2
answers
112
views
Local data cannot be read in a Dataproc cluster, when using SparkNLP
I am trying to build a Dataproc cluster, with Spark NLP installed in it, then quick test it by reading some CoNLL 2003 data. First, I used this codelab as inspiration, to build my own smaller cluster (...
1
vote
0
answers
74
views
Incorrect Dataproc cluster substate in google client library
I am trying to get the dataproc cluster's substate by using the google-cloud-dataproc client library. I have used the sample script from https://github.com/googleapis/python-dataproc/blob/...
1
vote
0
answers
78
views
GCS Connector Hadoop3 hadoop3-2.2.8 FileSystem.delete returns false but file/dir is actually deleted
In my Java application I have an implementation for a file-system, where my file class is a wrapper for Hadoop filesystem methods. I am upgrading the from hadoop3-1.9.17 to hadoop3-2.2.8 and I am ...
0
votes
1
answer
128
views
Spark-submit on GCP dataproc shows java.io. FileNotFoundException error
All the cases are performed in GCP data proc cluster with a master and 4 executors.
Case 1: When I execute a runnable JAR file with the spark-submit in Local mode with spark-config as
SparkConf config ...
1
vote
0
answers
43
views
How to estimate Bigquery data size for Dataproc job?
In GCP we will be using Dataproc to process data present in the BigQuery tables.
We wanted to estimate the Dataproc cluster size based on the data size present in BigQuery.
In BigQuery UI console, we ...
2
votes
0
answers
31
views
Exposing an external data connection publicly
I have a simple python script that's being called by a remote procedure in BigQuery through an Apache Spark external data connector that's provided by the public preview.
print("Hello World!"...
1
vote
1
answer
61
views
How to read data and run a linear regression in Jupyter lab R kernel on DataProc GCP?
I tried reading data from the GCS bucket on pyspark using the following code:
df_business = spark.read.json("gs://[bucket_name]/[filename].json")
I want to do the same using the R kernel on ...
1
vote
0
answers
47
views
How to find the cumulative sum of a column in pyspark with multiple files
I have a multiple CSV files, where each csv have 1 month of data.
I want to add an extra column like Cumulative sum of a column of last 3 days.
I have the partial solution here:
grouped_df = df....
1
vote
0
answers
49
views
Importing external python file in DataprocSubmitJobOperator on Airflow
I want to send a zipfile which has common python scripts to DataprocSubmitJobOperator. Here are my job parameters.
COMMON = "gs://my-bucket/common/common.zip"
INGEST_JOB_URI = f"pyspark-...
6
votes
0
answers
194
views
GCS Connector Hadoop3 hadoop3-2.2.8 - Slow read, exists, rename and create operations
In my Java application I have an implementation for a file-system layer, where my file class is a wrapper for Hadoop filesystem methods. I am upgrading the from hadoop3-1.9.17 to hadoop3-2.2.8 and I ...