6

I googled this problem, yet no direct answer related to spark-2.2.0-bin-hadoop2.7. I am trying to read a text file from local directory, but I always get TypeError that name argument is missing. This is the code in jupyter notebook with Python3:

from pyspark import SparkContext as sc
data = sc.textFile("/home/bigdata/test.txt")

When I run the cell, I get this error:

TypeError                                 Traceback (most recent call last)
  <ipython-input-7-2a326e5b8f8c> in <module>()
  1 from pyspark import SparkContext as sc
  ----> 2 data = sc.textFile("/home/bigdata/test.txt")
  TypeError: textFile() missing 1 required positional argument: 'name'

Your help is appreciated.

2 Answers 2

14

You are calling the textFile() instance method

def textFile(self, name, minPartitions=None, use_unicode=True):

like it was a static method which results into "/home/bigdata/test.txt" string being used for the self value leaving name argument not specified, hence the error.

Create an instance of the SparkContext class:

from pyspark import SparkConf
from pyspark.context import SparkContext

sc = SparkContext.getOrCreate(SparkConf().setMaster("local[*]"))
data = sc.textFile("/home/bigdata/test.txt")
2
  • Thank you @alecxe for your help. I tried the code you suggested, but I got the following error: ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, master=local[*]) created by <module> at /home/bigdata/anaconda3/lib/python3.6/site-packages/IPython/utils/py3compat.py:186
    – Mohammed
    Commented Dec 6, 2017 at 2:03
  • 1
    @Mohammed ah, these local and test are there for demonstration purposes only..the main point is to initialized the SparkContext class. Thanks.
    – alecxe
    Commented Dec 6, 2017 at 2:06
6
from pyspark import SparkConf
from pyspark.context import SparkContext
sc = SparkContext.getOrCreate(SparkConf())
data = sc.textFile("my_file.txt")

Display some content

['this is text file and sc is working fine']

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.