11

I wrote a simple Flask app to pass some data to Spark. The script works in IPython Notebook, but not when I try to run it in it's own server. I don't think that the Spark context is running within the script. How do I get Spark working in the following example?

from flask import Flask, request
from pyspark import SparkConf, SparkContext

app = Flask(__name__)

conf = SparkConf()
conf.setMaster("local")
conf.setAppName("SparkContext1")
conf.set("spark.executor.memory", "1g")
sc = SparkContext(conf=conf)

@app.route('/accessFunction', methods=['POST'])
def toyFunction():
    posted_data = sc.parallelize([request.get_data()])
    return str(posted_data.collect()[0])

if __name__ == '__main_':
    app.run(port=8080)

In IPython Notebook I don't define the SparkContext because it is automatically configured. I don't remember how I did this, I followed some blogs.

On the Linux server I have set the .py to always be running and installed the latest Spark by following up to step 5 of this guide.

Edit:

Following the advice by davidism I have now instead resorted to simple programs with increasing complexity to localise the error.

Firstly I created .py with just the script from the answer below (after appropriately adjusting the links):

import sys
try:
    sys.path.append("your/spark/home/python")
    from pyspark import context
    print ("Successfully imported Spark Modules")
except ImportError as e:
    print ("Can not import Spark Modules", e)

This returns "Successfully imported Spark Modules". However, the next .py file I made returns an exception:

from pyspark import SparkContext
sc = SparkContext('local')
rdd = sc.parallelize([0])
print rdd.count()

This returns exception:

"Java gateway process exited before sending the driver its port number"

Searching around for similar problems I found this page but when I run this code nothing happens, no print on the console and no error messages. Similarly, this did not help either, I get the same Java gateway exception as above. I have also installed anaconda as I heard this may help unite python and java, again no success...

Any suggestions about what to try next? I am at a loss.

1
  • note that I POSTed on my machine with iPython Notebook by using the restclient extension on firefox and it worked as expected
    – Matt
    Commented Sep 22, 2015 at 14:44

3 Answers 3

16

Okay, so I'm going to answer my own question in the hope that someone out there won't suffer the same days of frustration! It turns out it was a combination of missing code and bad set up.

Editing the code: I did indeed need to initialise a Spark Context by appending the following in the preamble of my code:

from pyspark import SparkContext
sc = SparkContext('local')

So the full code will be:

from pyspark import SparkContext
sc = SparkContext('local')

from flask import Flask, request
app = Flask(__name__)

@app.route('/whateverYouWant', methods=['POST']) # can set first param to '/'
def toyFunction():
    posted_data = sc.parallelize([request.get_data()])
    return str(posted_data.collect()[0])

if __name__ == '__main_':
    app.run(port=8080)    #note set to 8080!

Editing the setup: It is essential that the file (yourfilename.py) is in the correct directory, namely it must be saved to the folder /home/ubuntu/spark-1.5.0-bin-hadoop2.6.

Then issue the following command within the directory:

./bin/spark-submit yourfilename.py

which initiates the service at 10.0.0.XX:8080/whateverYouWant/.

Note that the port must be set to 8080 or 8081: Spark only allows web UI for these ports by default for master and worker respectively

You can test out the service with a restful service or by opening up a new terminal and sending POST requests with cURL commands:

curl --data "DATA YOU WANT TO POST" http://10.0.0.XX/8080/whateverYouWant/
4
  • Hi @Matt, I am facing similar problems. I am a bit confused with this. why is it necessary to put the project under spark directory? and what about this XX in the host address? Commented Nov 23, 2016 at 12:07
  • Hey @Larissa its been a while since I wrote that so I'm a little rusty. You need to put the project in the spark directory to run the command. You can get around this if you add ./bin/spark-submit to your spark bash and you can run it from anywhere, but I didn't want to overcomplicate this answer. As for the XX that was the local IP of the box, maybe I should have just written 0.0.0.0 instead, that will work. You should be able to send POST requests via postman from another machine on the same network though. Hope this helps!
    – Matt
    Commented Nov 27, 2016 at 9:28
  • how to do this on windows? ./bin/spark-submit yourfilename.py
    – Reub
    Commented Mar 20, 2018 at 17:09
  • I tried doing this, I got errors instead of being able to access the flask app
    – Reub
    Commented Mar 20, 2018 at 18:18
1

I was able to fix this problem by adding the location of PySpark and py4j to the path in my flaskapp.wsgi file. Here's the full content:

import sys
sys.path.insert(0, '/var/www/html/flaskapp')
sys.path.insert(1, '/usr/local/spark-2.0.2-bin-hadoop2.7/python')
sys.path.insert(2, '/usr/local/spark-2.0.2-bin-hadoop2.7/python/lib/py4j-0.10.3-src.zip')

from flaskapp import app as application
0

Modify your .py file as it is shown in the linked guide 'Using IPython Notebook with Spark' part second point. Insted sys.path.insert use sys.path.append. Try insert this snippet:

import sys
try:
    sys.path.append("your/spark/home/python")
    from pyspark import context
    print ("Successfully imported Spark Modules")
except ImportError as e:
    print ("Can not import Spark Modules", e)
2
  • Thanks for getting back to me. I have tried creating as simple a program as possible, firstly just consisting of your code above. When I run it I get teh "successfully...", which is great. I then make a simple program that I will add as an edit to the question. The result is an exception: "Java gateway process exited before sending the driver its port number"
    – Matt
    Commented Sep 23, 2015 at 17:19
  • If you have already initialized the variable 'sc' then you must use the same instance instead of reinitializing the variable. Commented Sep 24, 2015 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.