11

I'm trying to install Spark on my Windows desktop. Everything should work fine, but I get an error "'cmd' is not recognized as an internal or external command... "

I installed Scala, Java JDK and unzipped Spark tgz in C:\, but for some reason can't get Spark to start in cmd. Any ideas?

14 Answers 14

16

My colleague solved the problem. Although Java seemed to work ok (ref. picture), the Java path Spark was trying to read was incorrect with an extra \bin at the end. When that was removed, Spark started working! @gonbe, thank you so much for your efforts to help!

2
  • 1
    Same here: JAVA_HOME had /bin on the end, Spark didn't like that
    – Spork
    Commented Mar 10, 2016 at 0:35
  • I got the issue when missing the final semicolon(;) in the PATH. Commented Jun 16, 2021 at 3:23
7

I had the similar error. I fixed it after following changes:

  1. There were multiple Java/bin path in the System Path. So I corrected them to reflect single Java/Bin, which is in sync with JAVA_HOME
  2. Added C:Windows\system32 to System Path Variable.
  3. My Java_Home and java.exe was pointing different places. I fixed them.

Now it works.

Thanks guys.

5

(I'm not Windows Spark user) The spark-shell.cmd for Windows source code expects "cmd" command is available in PATH.

https://github.com/apache/spark/blob/master/bin/spark-shell.cmd

Would you try adding the directory that contains "cmd.exe" in PATH environment variable? The directory location is shown title bar in your screenshot, and environment variable setting can be done via control panel.

10
  • 2
    I added the path C:\Windows\System32, to the beginning of the PATH and it looks like this now: C:\Windows\System32;%SCALA_HOME%\bin;C:\Anaconda;C:\dev\Java\jdk1.8.0_45\bin;C:\Program Files\R\R-3.1.1\library\bin\x64... etc. But it is still giving an error message, now its: "The system cannot find the path specified". I tried to change PATH, loaded Java again and so on, but no help.
    – ElinaJ
    Commented Jun 21, 2015 at 20:32
  • Can you paste entire error message? (especially the path the system cannot find)
    – suztomo
    Commented Jun 21, 2015 at 20:35
  • I'm afraid that's the only message there is, or is there a way to see more? I've been testing by taking objects away from the PATH, but nothing helps...
    – ElinaJ
    Commented Jun 21, 2015 at 20:40
  • I wanted to see what the error is complaining about. How about double-clicking spark-shell.cmd via Explorer? Or run the program specified in spark-shell.cmd.
    – suztomo
    Commented Jun 21, 2015 at 20:45
  • The cmd just flashes once open and closes immediately... I tried "Run as Administrator, but the same flash happens...
    – ElinaJ
    Commented Jun 21, 2015 at 21:05
4

Check values in JAVA_HOME and make sure it is pointing to correct value. Add %JAVA_HOME%/bin in path value. After modification close command prompt and restart it. Write spark-shell and it will run.

1
  • This worked for me. Added %JAVA_HOME%/bin in User Variable. Commented May 17, 2020 at 12:43
2

I was the getting the same error while executing Spark-shell in the command prompt.

I tried everything mentioned above but not able to resolve the issue.

So, at last I added "C:\Windows\System32" in 'PATH' variable of System Variable and it worked.

1

All my variables were OK, so I decided to debug the scripts and I found in "spark-class2.cmd" and put another pair of quotation marks in "%RUNNERS%". BEFORE "%RUNNER%" -Xmx128m -cp "%LAUNCH_CLASSPATH%" .... AFTER ""%RUNNER%"" -Xmx128m -cp "%LAUNCH_CLASSPATH%" ....

1

For my case I had similar issue. I had to fix a couple of things.

1- Check JAVA_HOME is correct on both places;

enter image description here

enter image description here

2- Then I had to change following lines in spark-2.1.1-bin-hadoop2.7\bin folder.

enter image description here

  • Add extra quotation marks around "%RUNNER%". So it will be like ""%RUNNER%""
  • Then execute .\spark-shell.cmd again.

enter image description here

0

Check the java jdk version and scala version according to the below version compatibility table:

enter image description here

0

I had the same issue (launch spark-shell and get the the system cannot find the path) After following the above process (changing the spark_home to exclude the /bin it worked fine thanks for sharing guys

0

These are detailed steps to resolve all these issues in Windows

Spark binaries

  1. It needs to be in a folder which doesn’t have space, if it must be in a folder which has space then you can use something like Unix’s symlink. I have kept the Spark binaries inside c:\ directly. Set SPARK_HOME to e.g. C:\SparkMedia\spark-2.4.2-bin-hadoop2.7. Please note “bin” is not part of the path.
  2. Set “path” for the user and system to %SPARK_HOME%\bin If you decide to keep it outside “program files” (as per point a), then please give permission to the user because Spark keeps temporary files here. If you want to keep Spark inside a directory where user doesn’t have access then you need to put a file %SPARK_HOME%\conf\spark-env.sh with an entry inside it as “SPARK_LOG_DIR=a directory where user has access”.

Winutils

  1. It can be inside a folder with space like C:\Program Files (x86)\Winutils Your winutils kept at C:\Program Files (x86)\Winutils works fine.
  2. Set HADOOP_HOME to C:\Program Files (x86)\Winutils. Please note “bin” is not part of the path.
  3. Set “path” for the user and system to %HADOOP_HOME%\bin

Java

  1. It can be inside a folder with space like C:\Program Files\AdoptOpenJDK\jdk-8.0.262.10-hotspot Set JAVA_HOME to C:\Program Files\AdoptOpenJDK\jdk-8.0.262.10-hotspot. Please note “bin” is not part of the path.

  2. Set “path” for the user and system to %JAVA_HOME%\bin

With the above setup a user should be able to just type “spark-shell” and spark shell should start.

0

The spark-shell command runs the spark-shell.cmd file in the spark 'bin' folder. This file contains the following command cmd /V /E /C ""%~dp0spark-shell2.cmd" %*".

The cmd command requires the path system variable to be set to %SystemRoot%\system32

0

It should be the issue with your System Variables path. As what have mention in the error message, your command prompt can't recognize 'cmd'. So, you should add the directory contain cmd.exe into PATH list under your System variables so that your command prompt know where to find 'cmd'.

Steps

  1. Open Control Panel > System.
  2. Click on "Advanced system settings" (Right side under Related settings).
  3. Click on "Environment Variables..." at the bottom.
  4. Under "System variables" section look for "Path" and click on it, then click "Edit" button at the bottom. System variable setting
  5. In the editing tab, add "C:\Windows\System32" or "%SystemRoot%\System32".

After you added the new Path into System variable,

  1. Open Command Prompt
  2. cd %SPARK_HOME%\bin or cd to the spark directory.
  3. spark-shell

If everythings done right, you should be able to see the screen below. spark-shell

If still not working, you should check whether you have added SPARK_HOME, HADOOP_HOME, JAVA_HOME correctly under your User variables.

0

If you think the SPARK, HADOOP and JAVA and PATH variables are correctly set, Double-check if your User Variables have... C:\Windows\System32

I dealt with this problem and adding System32 to the user variables worked for me.

0

I was facing the same issue. I added C:\spark\spark-3.3.2-bin-hadoop3\bin inside path under System Variable. It worked for me.

enter image description here

1
  • How is this different from the other answers? Commented Mar 10, 2023 at 5:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.