0

I am using Spark in Qubole by having the clusters created in AWS. In Qubole Workbench, when I execute the below Command Line, it works fine and the command is successful

/usr/lib/spark/bin/spark-submit s3://bucket-name/SparkScripts/test.py

But, when I execute the same command along with --properties-file option

/usr/lib/spark/bin/spark-submit --properties-file s3://bucket-name/SparkScripts/properties.file s3://bucket-name/SparkScripts/test.py

it gives below error message

Qubole > Shell Command failed with exit code: 1

App > Error occurred when getting effective config required to initialize Qubole security provider

App > Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Properties file s3:/bucket-name/SparkScripts/properties.file does not exist

Can someone help me fix this? I need some application properties to be stored on a separate file on Amazon S3 and passed on to --properties-file to my spark program.

1 Answer 1

0

@saravanan - Qubole does not have the ability to specify --properties file from S3 path currently. It will be available in release 59.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.