0

I have a column in my data frame which contains list of JSONs but the type is of String. I need to run explode on this column, so first I need to convert this into a list. I couldn't find much references to this use case.

Sample data:

columnName: "[{"name":"a","info":{"age":"1","grade":"b"},"other":7},{"random":"x"}, {...}]"

The above is how the data looks like, the fields are not fixed (index 0 might have JSON with some fields while index 1 will have fields with some other fields). In the list there can be more nested JSONs or some extra fields. I am currently using this -

"""explode(split(regexp_replace(regexp_replace(colName, '(\\\},)','}},'), '(\\\[|\\\])',''), "},")) as colName""" where I am just replacing "}," with "}}," then removing "[]" and then calling split on "}," but this approach doesn't work since there are nested JSONs.

How can I extract the array from the string?

2
  • update the question with proper input JSON
    – Shankar
    Feb 24, 2022 at 13:45
  • It's proper, there are around 20 to 30 fields which are all nullable, I have tried showing that through a sample. Is there something specific you want to check?
    – Trayambak
    Feb 24, 2022 at 13:53

2 Answers 2

1

You can try this way:

// Initial DataFrame

df.show(false)

+----------------------------------------------------------------------+
|columnName                                                            |
+----------------------------------------------------------------------+
|[{"name":"a","info":{"age":"1","grade":"b"},"other":7},{"random":"x"}]|
+----------------------------------------------------------------------+

df.printSchema()

root
 |-- columnName: string (nullable = true)
 
// toArray is a user defined function that parses an array of json objects which is present as a string
     
import org.json.JSONArray

val toArray = udf { (data: String) => {
    val jsonArray = new JSONArray(data)
    var arr: Array[String] = Array()
    val objects = (0 until jsonArray.length).map(x => jsonArray.getJSONObject(x))
    objects.foreach { elem =>
        arr :+= elem.toString
    }
    arr
}
}

// Using the udf and exploding the resultant array

val df1 = df.withColumn("columnName",explode(toArray(col("columnName"))))

df1.show(false)

+-----------------------------------------------------+
|columnName                                           |
+-----------------------------------------------------+
|{"other":7,"name":"a","info":{"grade":"b","age":"1"}}|
|{"random":"x"}                                       |
+-----------------------------------------------------+

df1.printSchema()

root
 |-- columnName: string (nullable = true)
 
// Parsing the json string by obtaining the schema dynamically

val schema = spark.read.json(df1.select("columnName").rdd.map(x => x(0).toString)).schema
val df2 = df1.withColumn("columnName",from_json(col("columnName"),schema))

df2.show(false)

+---------------+
|columnName     |
+---------------+
|[[1, b], a, 7,]|
|[,,, x]        |
+---------------+

df2.printSchema()

root
 |-- columnName: struct (nullable = true)
 |    |-- info: struct (nullable = true)
 |    |    |-- age: string (nullable = true)
 |    |    |-- grade: string (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- other: long (nullable = true)
 |    |-- random: string (nullable = true)
 
// Extracting all the fields from the json

df2.select(col("columnName.*")).show(false)

+------+----+-----+------+
|info  |name|other|random|
+------+----+-----+------+
|[1, b]|a   |7    |null  |
|null  |null|null |x     |
+------+----+-----+------+

Edit:

You can try this way if you can use get_json_object function

// Get the list of columns dynamically

val columns = spark.read.json(df1.select("columnName").rdd.map(x => x(0).toString)).columns

// define an empty array of Column type and get_json_object function to extract the columns

var extract_columns: Array[Column] = Array()
    columns.foreach { column =>
    extract_columns :+= get_json_object(col("columnName"), "$." + column).as(column)
}

df1.select(extract_columns: _*).show(false)

+-----------------------+----+-----+------+
|info                   |name|other|random|
+-----------------------+----+-----+------+
|{"grade":"b","age":"1"}|a   |7    |null  |
|null                   |null|null |x     |
+-----------------------+----+-----+------+

Please note that info column is not of struct type. You may have to follow similar way to extract the columns of the nested json

2
  • I like the second approach but I am using an older version of Spark so can't "from_json", is it possible to achieve this with "get_json_object" ?. The first approach seems good as well but was looking for Spark library that I can reuse.
    – Trayambak
    Feb 24, 2022 at 14:09
  • Second approach seems different to what I was looking for. The first one worked like a charm. Thanks.
    – Trayambak
    Feb 24, 2022 at 14:38
0

val testString = """[{"name":"a","info":{"age":"1","grade":"b"},"other":7},{"random":"x"}]"""

val ds = Seq(testString).toDS()

spark.read.json(ds)
.select("info.age", "info.grade","name","other","random")
.show(10,false)

1
  • As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.
    – Community Bot
    Oct 2, 2022 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.