I follow this link http://ramhiser.com/2015/02/01/configuring-ipython-notebook-support-for-pyspark/ in order to create PySpark Profile for IPython.
00-pyspark-setup.py
# Configure the necessary Spark environment
import os
import sys
spark_home = os.environ.get('SPARK_HOME', None)
sys.path.insert(0, spark_home + "\python")
# Add the py4j to the path.
# You may need to change the version number to match your install
sys.path.insert(0, os.path.join(spark_home, '\python\lib\py4j-0.8.2.1-src.zip'))
# Initialize PySpark to predefine the SparkContext variable 'sc'
execfile(os.path.join(spark_home, '\python\pyspark\shell.py'))
My problem when I type sc
in ipython-notebook, I got ''
I should see output similar to <pyspark.context.SparkContext at 0x1097e8e90>.
Any idea about how to resolve it ?