Simple question, but I can't find a simple guide on how to set the environment variable in Databricks. Also, is it important to set the environment variable on both the driver and executors (and would you do this via spark.conf)? Thanks
2 Answers
Before creation:
You can set environment variable while creating the cluster.
Click on Advanced Options => Enter Environment Variables.
After creation:
Select your cluster => click on Edit => Advance Options => Edit or Enter new Environment Variables => Confirm and Restart.
OR
You can achieve the desired results by appending my environment variable declarations to the file /databricks/spark/conf/spark-env.sh. You may change the init file as follows:
%scala
dbutils.fs.put("dbfs:/databricks/init/set_spark_params.sh","""
|#!/bin/bash
|
|cat << 'EOF' > /databricks/driver/conf/00-custom-spark-driver-defaults.conf
|[driver] {
| "spark.sql.sources.partitionOverwriteMode" = "DYNAMIC"
|}
|EOF
""".stripMargin, true)
For more details, refer “Databricks – Spark Configuration”.
Hope this helps.
-
18As an aside: these variables can be accessed via
os.getenv("myenvname")
. Commenting here because this info was amazingly difficult to find. Commented Nov 5, 2021 at 1:27 -
Can we save file in .env file and load the variables in another file– ShubhCommented Apr 12, 2022 at 4:02
Use databricks cluster policy configuration. The configuration will auto-add the environment variables during policy selection.
spark_env_vars.MY_ENV_VAR: {
"value":"2.11.2",
"type": "fixed"
}