In general, if you do not need a full Spark installation, it is recommended that you just install it in your environment with pip:
pip install pyspark
If you are using conda, you may get it with:
conda install -c conda-forge pyspark
Once the installation is ready, you should be able to invoke Spark shell with the pyspark
keyword.
There are a couple of issues that you may encounter while running the executable. Notably, you may face following error:
Python was not found but can be installed from the Microsoft Store
Obviously, if you have just installed PySpark, you have Python ready in your virtual environment, so you can turn off these default paths by disabling python.exe
and python3.exe
App Installers in Apps >> Apps & features >> App execution aliases
in Settings.
You may still face a seemingly related error:
Missing Python executable python3
, defaulting to [path_to_spark_home]
for SPARK_HOME
environment variable. Please install Python or specify the correct Python executable in PYSPARK_DRIVER_PYTHON
or PYSPARK_PYTHON
environment variable to detect SPARK_HOME
safely.
This error message be coupled with a few of those:
The system cannot find the path specified.
In this case, the issue could either be in inappropriate SPARK_HOME
variable or an unspecified Python path.
In the first case, try setting SPARK_HOME
explicitly:
$env:SPARK_HOME=python -c "import pyspark, pathlib; print(pathlib.Path(pyspark.__file__).parent)"
Additionally, if you are still facing issues while running pyspark, such as:
Cannot run program "python3": [...] The system cannot find the file specified
set Python-related variables to Python path in your environment. Specifically:
$env:PYSPARK_DRIVER_PYTHON=python -c "import sys; print(sys.executable)"
$env:PYSPARK_PYTHON=python -c "import sys; print(sys.executable)"
If you do not have Java 11 installed, you should download, install it and set the environment variable JAVA_HOME
to the path, where it was installed. If you are using conda, you may use OpenJDK instead:
conda install -c conda-forge openjdk
You may still face an error stating:
Did not find winutils.exe. java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
As expected, you need to download binaries for Hadoop and then set the HADOOP_HOME
environment variable to that path.
Once all these paths are specified, PySpark executable should be able to run as long as you have necessary permissions.
Try testing its capability after running pyspark
with a simple script:
df = spark.createDataFrame([("Spark", "Impala"), ("3.2", "2.2")])
df.show()