8

I wrote the following:

val a = 1 to 10000
val b = sc.parallelize(a)

and it shows error saying:

<console>:12: error: not found: value sc

Any help?

4
  • sc ie. Spark Context has been made available when the shell started Aug 8, 2014 at 12:57
  • "made available when the shell started". Or rather, it hasn't. Provide more details about the steps you took to get to that error? Aug 8, 2014 at 13:09
  • are there any import statements or classpath changes before this in the repl? Aug 8, 2014 at 13:32
  • @GovindSinghNagarkoti wow so much observation talents WOW :) Sep 13, 2015 at 11:33

7 Answers 7

3

In my case I have spark installed on local windows system and I observed the same error but it was because of below issue

Issue:Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

This was because of permission issue.I resolved it by changing the permissions using below command.Though log says "on hdfs" this is on windows system

E:\winutils\bin\winutils.exe chmod 777 E:\tmp\hive

2

It happens when your classpath is not correct. This is an open issue in Spark at the moment.

> spark-shell 

...
...
14/08/08 18:41:50 INFO SparkILoop: Created spark context..
Spark context available as sc.

scala> sc
res0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@2c1c5c2e

scala> :cp /tmp
Added '/tmp'.  Your new classpath is:
...

scala> sc
<console>:8: error: not found: value sc

You may need to correct your classpath from outside the repl.

1

You get this error, because sc is not defined. I would try:

sc = SparkContext(appName = "foo")

Another thing that usually happens to me is not getting a Kerberos ticket in the cluster, because I forgot too.


As for the "open issue in Spark" mentioned by Solnanki, I am pretty sure this is not the case any more.

1
  • Comment by another user: Response: Code above yielded a warning about SparkContext not taking any parameters. So just try: val sc = SparkContext That fixed my issue
    – gsamaras
    Jun 10, 2017 at 1:56
1

First check the log file after spark-shell command run whether SparkContext is initinalized as sc. if SparkContext is not initialized properly

you have to set the IP address in spark environment.

Open the env file in conf/spark.env.sh and add the below line

export SPARK_LOCAL_IP="127.0.0.1"

0

I hit this error when trying out Spark on Cloudera Quickstart VM. Turned out to be an hdfs file permission issue on /user/spark.

I could not switch to user "spark", I got a user not available error. Changing file permissions with the below command solved it for me.

sudo -u hdfs hadoop fs -chmod -R 1777 /user/spark

scala> val data = 1 to 10000
data: scala.collection.immutable.Range.Inclusive = Range(1, 2, 3, 4, 5, 6, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170...
scala> val distData = sc.parallelize(data)
distData: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:14
0

i faced the same problem. In my case the JAVA_HOME was not set properly which cause this issue. surprisingly SPARK would start but the sc context had issues creating an instance. When i fixed the JAVA_HOME to point to the correct java directory, this issue was resolved. I had to close the session and re-open a new one in order to ensure the path is updated and fresh session is turned on.

I hope this helps.

0

As stated in this thread, one solution may be to switch off permissions checking.

In cloudera manager, go to hdfs configuration under advanced and put the following code in "HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" :

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

After that, it is necessary to restart the HDFS component.

It worked for me. It might not be appropriate for a production environment, however.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.