58

I've got this JSON file

{
    "a": 1, 
    "b": 2
}

which has been obtained with Python json.dump method. Now, I want to read this file into a DataFrame in Spark, using pyspark. Following documentation, I'm doing this

sc = SparkContext()

sqlc = SQLContext(sc)

df = sqlc.read.json('my_file.json')

print df.show()

The print statement spits out this though:

+---------------+
|_corrupt_record|
+---------------+
|              {|
|       "a": 1, |
|         "b": 2|
|              }|
+---------------+

Anyone knows what's going on and why it is not interpreting the file correctly?

6 Answers 6

82

If you want to leave your JSON file as it is (without stripping new lines characters \n), include multiLine=True keyword argument

sc = SparkContext() 
sqlc = SQLContext(sc)

df = sqlc.read.json('my_file.json', multiLine=True)

print df.show()
0
62

You need to have one json object per row in your input file, see https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrameReader.json.html

If your json file looks like this it will give you the expected dataframe:

{ "a": 1, "b": 2 }
{ "a": 3, "b": 4 }

....
df.show()
+---+---+
|  a|  b|
+---+---+
|  1|  2|
|  3|  4|
+---+---+
3
  • 8
    How can I fix it if my JSON file is huge (a couple of 100K rows) and it has a lot of new lines in between the records (columns or features)? thanks.
    – M.Rez
    Apr 13, 2017 at 13:17
  • 2
    Maybe use jq to reformat(compact) the file? @M.Rez
    – ttimasdf
    Nov 12, 2019 at 10:09
  • 1
    @ttimasdf Namely, using jq -c option. Jun 8, 2021 at 16:02
26

In Spark 2.2+ you can read json file of multiline using following command.

val dataframe = spark.read.option("multiline",true).json( " filePath ")

if there is json object per line then,

val dataframe = spark.read.json(filepath)
2
  • 5
    This is scala, not python. Mar 21, 2019 at 11:06
  • 1
    Still works in Python however spark.read.option("multiline",True).json('filePath') Apr 15, 2022 at 9:18
3

Adding to @Bernhard's great answer

# original file was written with pretty-print inside a list
with open("pretty-printed.json") as jsonfile:
    js = json.load(jsonfile)      

# write a new file with one object per line
with open("flattened.json", 'a') as outfile:
    for d in js:
        json.dump(d, outfile)
        outfile.write('\n')
0
2

I want to share my experience in which I have a JSON column String but with Python notation, which means I have None instead of null, False instead of false and True instead of true.

When parsing this column, spark returns me a column named _corrupt_record. So what I had to do before parsing the JSON String is replacing the Python notation with the standard JSON notation:

df.withColumn("json_notation",
    F.regexp_replace(F.regexp_replace(F.regexp_replace("_corrupt_record", "None", "null"), "False", "false") ,"True", "true")

After this transformation I was then able to use for example the function F.from_json() on the json_notation column and here Pyspark was able to correctly parse the JSON object.

0

One another reason of this happening could be file encoding. If the file you are reading is say for example in latin encoding you will get this issue. Try using .option("encoding", "cp1252") while reading file. This resolved the issue for me

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.