7

I follow this link http://ramhiser.com/2015/02/01/configuring-ipython-notebook-support-for-pyspark/ in order to create PySpark Profile for IPython.

00-pyspark-setup.py
# Configure the necessary Spark environment
import os
import sys

spark_home = os.environ.get('SPARK_HOME', None)
sys.path.insert(0, spark_home + "\python")

# Add the py4j to the path.
# You may need to change the version number to match your install
sys.path.insert(0, os.path.join(spark_home, '\python\lib\py4j-0.8.2.1-src.zip'))

# Initialize PySpark to predefine the SparkContext variable 'sc'
execfile(os.path.join(spark_home, '\python\pyspark\shell.py'))

My problem when I type sc in ipython-notebook, I got '' I should see output similar to <pyspark.context.SparkContext at 0x1097e8e90>.

Any idea about how to resolve it ?

2
  • What are you trying to achieve. Where in your code does the problem occur?
    – Rias
    Apr 21, 2015 at 21:43
  • In Python 3 execfile was removed so last line have to be changed to exec(open(os.path.join(spark_home, '\python\pyspark\shell.py')).read()). Jul 26, 2015 at 20:47

4 Answers 4

7

I was trying to do the same, but had problems. Now, I use findspark (https://github.com/minrk/findspark) instead. You can install it with pip (see https://pypi.python.org/pypi/findspark/):

$ pip install findspark

And then, inside a notebook:

import findspark
findspark.init()

import pyspark
sc = pyspark.SparkContext(appName="myAppName")

If you want to avoid this boilerplate, you can put the above 4 lines in 00-pyspark-setup.py.

(Right now I have Spark 1.4.1. and findspark 0.0.5.)

0
0

Please try to set proper value to SPARK_LOCAL_IP variable, eg.:

export SPARK_LOCAL_IP="$(hostname -f)"

before you run ipython notebook --profile=pyspark.

If this doesn't help, try to debug your environment by executing setup script:

python 00-pyspark-setup.py

Maybe you can find some error lines in that way and debug them.

0

Are you on windows? I am dealing with the same things, and a couple of things helped. In the 00-pyspark-setup.py, change this line (match your path to your spark folder)

# Configure the environment
if 'SPARK_HOME' not in os.environ:
    print 'environment spark not set'
    os.environ['SPARK_HOME'] = 'C:/spark-1.4.1-bin-hadoop2.6'

I am sure you added a new environment variable, if not, this will manually set it.

The next thing I noticed is that if you use ipython 4 (the latest), the config files don't work the same way you see it in all the tutorials. You can try out if your config files get called by adding a print statement or just messing them up so an error gets thrown.

I am using a lower version of iPython (3) and I call it using

ipython notebook --profile=pyspark
0

Change the 00-pyspark-setup.py to:

# Configure the necessary Spark environment
import os

# Spark home
spark_home = os.environ.get("SPARK_HOME")

######## CODE ADDED ########
os.environ["PYSPARK_SUBMIT_ARGS"] = "--master local[2] pyspark-shell"
######## END OF ADDED CODE #########

sys.path.insert(0, spark_home + "/python")
sys.path.insert(0, os.path.join(spark_home, 'python/lib/py4j-0.8.2.1-src.zip'))

# Initialize PySpark to predefine the SparkContext variable 'sc'
execfile(os.path.join(spark_home, 'python/pyspark/shell.py'))

Basically, the added code sets the PYSPARK_SUBMIT_ARGS environment variable to

--master local[2] pyspark-shell, which works for Spark 1.6 standalone.

Now run ipython notebook again. Run os.environ["PYSPARK_SUBMIT_ARGS"] to check whether its value is correctly set. If so, then type sc should give you the expected output like <pyspark.context.SparkContext at 0x1097e8e90>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.