1

Is there any method where we can delete the records from a dataframe where any of the column values is null or empty?

+---+-------+--------+-------------------+-----+----------+
|id |zipcode|type    |city               |state|population|
+---+-------+--------+-------------------+-----+----------+
|1  |704    |STANDARD|                   |PR   |30100     |
|2  |704    |        |PASEO COSTA DEL SUR|PR   |          |
|3  |76166  |UNIQUE  |CINGULAR WIRELESS  |TX   |84000     |
+---+-------+--------+-------------------+-----+----------+

I want output to be:

+---+-------+------+-----------------+-----+----------+
|id |zipcode|type  |city             |state|population|
+---+-------+------+-----------------+-----+----------+
|4  |76166  |UNIQUE|CINGULAR WIRELESS|TX   |84000     |
+---+-------+------+-----------------+-----+----------+
2

2 Answers 2

3

Try this:

df
  .na.replace(df.columns,Map("" -> null)) // convert empty strings with null
  .na.drop() // drop nulls and NaNs
  .show()
1
1

Try this:

df_name.na.drop()
  .show(false)

Hope it helps...

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.