19

Simple question, but I can't find a simple guide on how to set the environment variable in Databricks. Also, is it important to set the environment variable on both the driver and executors (and would you do this via spark.conf)? Thanks

2 Answers 2

28

Before creation:

You can set environment variable while creating the cluster.

Click on Advanced Options => Enter Environment Variables.

enter image description here

After creation:

Select your cluster => click on Edit => Advance Options => Edit or Enter new Environment Variables => Confirm and Restart.

enter image description here

OR

You can achieve the desired results by appending my environment variable declarations to the file /databricks/spark/conf/spark-env.sh. You may change the init file as follows:

%scala
dbutils.fs.put("dbfs:/databricks/init/set_spark_params.sh","""
  |#!/bin/bash
  |
  |cat << 'EOF' > /databricks/driver/conf/00-custom-spark-driver-defaults.conf
  |[driver] {
  |  "spark.sql.sources.partitionOverwriteMode" = "DYNAMIC"
  |}
  |EOF
  """.stripMargin, true)

For more details, refer “Databricks – Spark Configuration”.

Hope this helps.

2
  • 18
    As an aside: these variables can be accessed via os.getenv("myenvname") . Commenting here because this info was amazingly difficult to find.
    – defraggled
    Commented Nov 5, 2021 at 1:27
  • Can we save file in .env file and load the variables in another file
    – Shubh
    Commented Apr 12, 2022 at 4:02
2

Use databricks cluster policy configuration. The configuration will auto-add the environment variables during policy selection.

spark_env_vars.MY_ENV_VAR: {
  "value":"2.11.2",
  "type": "fixed"
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.