5

I can do a pip install pyspark on my windows. When I try to run a sample script below it tells me my SPARK_HOME is not set.

Do i need to set my SPARK_HOME still and how do I go about doing it?

The blogs which I have referred online do a manual extraction of the spark files from the spark website and then later they have to put the SPARK_HOME and the PYTHONPATH.

However, I thought this was elimated with pip install pyspark.

import findspark
findspark.init()

import pyspark # only run after findspark.init()
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

df = spark.sql('''select 'spark' as hello ''')
df.show()

For intellij, do I still need to do additional configuration once i have installed in pyspark and set up as necessary?

2 Answers 2

1

In general, if you do not need a full Spark installation, it is recommended that you just install it in your environment with pip:

pip install pyspark

If you are using conda, you may get it with:

conda install -c conda-forge pyspark

Once the installation is ready, you should be able to invoke Spark shell with the pyspark keyword. There are a couple of issues that you may encounter while running the executable. Notably, you may face following error:

Python was not found but can be installed from the Microsoft Store

Obviously, if you have just installed PySpark, you have Python ready in your virtual environment, so you can turn off these default paths by disabling python.exe and python3.exe App Installers in Apps >> Apps & features >> App execution aliases in Settings.

You may still face a seemingly related error:

Missing Python executable python3, defaulting to [path_to_spark_home] for SPARK_HOME environment variable. Please install Python or specify the correct Python executable in PYSPARK_DRIVER_PYTHON or PYSPARK_PYTHON environment variable to detect SPARK_HOME safely.

This error message be coupled with a few of those:

The system cannot find the path specified.

In this case, the issue could either be in inappropriate SPARK_HOME variable or an unspecified Python path.

In the first case, try setting SPARK_HOME explicitly:

$env:SPARK_HOME=python -c "import pyspark, pathlib; print(pathlib.Path(pyspark.__file__).parent)"

Additionally, if you are still facing issues while running pyspark, such as:

Cannot run program "python3": [...] The system cannot find the file specified

set Python-related variables to Python path in your environment. Specifically:

$env:PYSPARK_DRIVER_PYTHON=python -c "import sys; print(sys.executable)"
$env:PYSPARK_PYTHON=python -c "import sys; print(sys.executable)"

If you do not have Java 11 installed, you should download, install it and set the environment variable JAVA_HOME to the path, where it was installed. If you are using conda, you may use OpenJDK instead:

conda install -c conda-forge openjdk

You may still face an error stating:

Did not find winutils.exe. java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.

As expected, you need to download binaries for Hadoop and then set the HADOOP_HOME environment variable to that path.

Once all these paths are specified, PySpark executable should be able to run as long as you have necessary permissions.

Try testing its capability after running pyspark with a simple script:

df = spark.createDataFrame([("Spark", "Impala"), ("3.2", "2.2")])
df.show()
1

In 2022, this worked for me. My setup was jupyter notebook in VS Code.

Prereq:

  • Ensure JDK 1.8
  • Download repo zip from: https://github.com/steveloughran/winutils
    • Extract out hadoop version files of your choice (my ver: 3.0.0)
    • Place files in a some folder e.g. c:\blah\hadoop\bin (bin folder is required)
#!java -version         # check your java version
!pip install pyspark

hadoopFilesPath = r"c:\blah\hadoop"
import os
os.environ["HADOOP_HOME"] = hadoopFilesPath
os.environ["hadoop.home.dir"] = hadoopFilesPath
os.environ["PATH"] = os.environ["PATH"] + f";{hadoopFilesPath}\\bin"

from pyspark.sql import SparkSession, SQLContext
spark = SparkSession.builder.master('local[*]').getOrCreate()
print(spark.version)

df = spark.read.csv("test.csv", header=True, inferSchema=True)
df.show()

df.write.parquet(r"C:\blah\data\test.parquet")

Note that the environment variables do the trick. Without the, First two are obvious (part of error message), the path one required googling.

Relevant error messages:

  • java.lang.RuntimeException: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
  • Py4JJavaError: An error occurred while calling o42.parquet.
  • java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
  • InsertIntoHadoopFsRelationCommand windows An error occurred while calling Py4JJavaError
1
  • Ideally, you use Java 11, or the higher version that Spark supports Commented Aug 24, 2022 at 2:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.