0

My job runs fine in my notebook, but when I copy and paste the script into the Spark Scala scheduled job, I run into errors like "script.scala:15: error: not found: value sqlContext".

What do I need to do to run my Scala code as scheduled job?

2 Answers 2

0

Are you using the Qubole scheduler to run the spark scala job? I see you mentioned that the jobs run fine in the notebook.

At this point, there are two ways you can look into this:

  1. You can schedule the notebook (Spark command -> Notebook) More details can be found here: https://docs.qubole.com/en/latest/user-guide/engines/spark/cron-notebook.html

  2. To run spark scala via the Analyze tab and schedule that command, you can import sparkcontext: import org.apache.spark.SparkContext._

Ideally having a look at the command would help, but for now, you can view the example codes on the Analyze page, under workspace. The saved commands have some helpful examples.

0

@paul-mineau , In notebooks few variables are pre-created and associated with Spark JVM objects. For example, variable name sc is associated to SparkContext of the JVM, spark is associated with SparkSession object of Spark application, Similarly sqlContext is associated with SQLContext of the application.
In few cases, you might not be able to port the code directly to Analyze. Since all variables might not be pre-created for in analyze(Also even if pre-created, there is no gurantee that variable names would be same. example: SparkContext may bound to variable name sc1 etc).
I would suggest explicitly creating those object(SQLContext in this case) in your application and accessing it.
For this specific case, if Spark version > 2.0.0, you can access SQLContext using SparkSession (org.apache.spark.sql.SparkSession#sqlContext)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.