Consider this df
+----+------+
|cond|chaine|
+----+------+
| 0| TF1|
| 1| TF1|
| 1| TNT|
+----+------+
I would like to apply this withColumn instruction but only on rows having cond == 1
:
df.withColumn("New", when($"chaine" === "TF1", "YES!"))
.withColumn("New2", when($"chaine" === "TF1", "YES2!"))
.withColumn("New3", when($"chaine" === "TF1", "YES3!"))
.withColumn("New4", when($"chaine" === "TF1", "YES4!"))
I can't use .filter
because I still want to have rows with cond =!= 1
in output.
I can do it by adding my condition inside every where in code:
df.withColumn("New", when($"chaine" === "TF1" AND $"cond" === 1, "YES!"))
.withColumn("New2", when($"chaine" === "TF1" AND $"cond" === 1, "YES2!"))
.withColumn("New3", when($"chaine" === "TF1" AND $"cond" === 1, "YES3!"))
.withColumn("New4", when($"chaine" === "TF1" AND $"cond" === 1, "YES4!"))
but the problem is that I have a lot of new columns and I want a better solution (like a global confition ?)
Thank you.