2

Description:

I installed spark on my MacBook following using Homebrew. I followed the instruction process from: https://www.tutorialkart.com/apache-spark/how-to-install-spark-on-mac-os/.

The step by step process included installing Java, followed by Scala and then Spark. Java and Scala got installed successful. Spark got installed successfully as well.

When I tried to verify spark installation using below input command, I ran into an error.

Input command: spark-shell

Expected Behavior: Expect Spark to start on terminal

Actual Behavior: I get this below error:

Unable to find any JVMs matching version "1.8".
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/2.4.5/libexec/jars/spark-unsafe_2.11-2.4.5.jar) to method java.nio.Bits.unaligned()
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" java.lang.ExceptionInInitializerError
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
    at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
    at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
    at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422)
    at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)
    at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:348)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$secMgr$1(SparkSubmit.scala:348)
    at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:356)
    at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:356)
    at scala.Option.map(Option.scala:146)
    at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:355)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
    at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3756)
    at java.base/java.lang.String.substring(String.java:1902)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52)

What I tried:

I tried to change JAVA_HOME using below commands:

export JAVA_HOME=/usr/local/opt/java

Previous JAVA_HOME path was /opt/anaconda3. I can see that JAVA_HOME got changed to usr/local/opt/java.

I am still getting the error. Appreciate your answers/feedback. Thanks!!!

7
  • Did you install a newer version of Java? What is the result of /usr/local/opt/java/bin/java -version? Apr 7, 2020 at 23:29
  • Yes, as part of spark installation I did install a newer version of Java. Strange... /usr/local/opt/java/bin/java -version gives me this: openjdk version "13.0.2" 2020-01-14 OpenJDK Runtime Environment (build 13.0.2+8) OpenJDK 64-Bit Server VM (build 13.0.2+8, mixed mode, sharing). and when I just input java -version from my default prompt, I get this: openjdk version "1.8.0_152-release" OpenJDK Runtime Environment (build 1.8.0_152-release-1056-b12) OpenJDK 64-Bit Server VM (build 25.152-b12, mixed mode)
    – Skandarks
    Apr 8, 2020 at 0:22
  • Okay. Now try which java Apr 8, 2020 at 0:33
  • When I try 'which which java I am getting this: /Users/skanda_work/opt/anaconda3/bin/java
    – Skandarks
    Apr 8, 2020 at 0:51
  • export JAVA_HOME=/Users/skanda_work/opt/anaconda3 Apr 8, 2020 at 0:53

2 Answers 2

4

Follow these steps for macOS Step 1. install java 8 as Spark 2.2 onwards requires java8. See Spark Documentation! for details.

brew install openjdk@8

Then set update java path

export JAVA_HOME=/usr/local/opt/openjdk@8/libexec/openjdk.jdk/Contents/Home

2

I was searching across the web and saw instructions to install pyspark.

I ran this command pip install pyspark on Terminal

After I installed pyspark, both spark and pyspark are running.

Not sure what happened, but I am able to run spark now.

Thanks @Elliott for interacting and giving some directions!

1
  • Agreed, pip install works over conda install or binaries on a MacBook Pro.
    – sAguinaga
    Sep 27, 2020 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.