2

I am trying to write a Spark program with Scala on a remote machine, but that machine has no internet access. Since I am using the pre-built version for Hadoop, I am able to run the pre-compiled examples:

[user@host spark-0.7.2]$ ./run spark.examples.LocalPi

but I can't compile anything that references spark on the machine:

$ scalac PiEstimate.scala
PiEstimate.scala:1: error: not found: object spark
import spark.SparkContext
       ^

Normally, I would use SBT to take care of any dependencies, but the machine does not have internet access, and tunneling internet through SSH is not possible.

Is it possible to compile an SBT project on a remote machine that has no internet access? Or how could I manually link the Spark dependencies to the Scala compiler.

2 Answers 2

1

If you're compiling your Spark program through scalac, you'll have to add Spark's jars to scalac's classpath; I think this should work:

scalac -classpath "$SPARK_HOME/target/scala-*/*.jar" PiEstimate.scala
1
  • 1
    Note: it might be possible to use sbt in an "offline mode": this thread looks relevant. Another option is to compile your program on a local machine and scp the compiled classes to the remote machine.
    – Josh Rosen
    Jun 11, 2013 at 16:15
0

I know this is an old post but I had to deal with this issue recently. I solved it by removing the dependencies from my .sbt file and adding the spark jar (spark-home/assembly/target/scala.2-10/spark-[...].jar) under my-project-dir/lib directory. You can also point to it using unmanagedBase = file("/path/to/jars/") Then I could use sbt package as usually

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.