2

I am looking for a way to identify and replace row-wise duplicate values in a data.table with NA.

The following answer helps me identify row-wise duplicates...

Removing duplicate values row-wise in R

...but I am looking for a way to address those duplicates.

PROBLEM

(dt <- data.table(X = 1:10, Y = seq(1, 19, by = 2), Z = c(1, rep(3, 9))))
     X  Y Z
 1:  1  1 1
 2:  2  3 3
 3:  3  5 3
 4:  4  7 3
 5:  5  9 3
 6:  6 11 3
 7:  7 13 3
 8:  8 15 3
 9:  9 17 3
10: 10 19 3

EXPECTED RESULT

     X   Y  Z
 1:  1  NA NA
 2:  2   3 NA
 3:  3   5 NA
 4:  4   7  3
 5:  5   9  3
 6:  6  11  3
 7:  7  13  3
 8:  8  15  3
 9:  9  17  3
10: 10  19  3

Please help

1
  • 1
    Reshape to long format. Then this becomes easy (and efficient).
    – Roland
    Commented Aug 15, 2019 at 7:23

3 Answers 3

3

Here is a data.table solution:

dt[, row := .I
   ][, melt(.SD, id.cols = "row", measure.vars = c("X", "Y", "Z"))
     ][, value := replace(value, duplicated(value), NA), by = row
       ][, dcast(.SD, row ~ variable)
         ][, !"row"]
3
  • @Ronald's comments have made me consider this answer. @sindri_baldur: could you modify this to not have the row column in the final output?
    – Sanjid
    Commented Aug 19, 2019 at 2:22
  • thanks @sindhi_baldur One last thing. Anything I can do to retain an ID column? For instance, a data.table like the following: (dt <- data.table(ID = LETTERS[1:10], X = 1:10, Y = seq(1, 19, by = 2), Z = c(1, rep(3, 9))))
    – Sanjid
    Commented Aug 20, 2019 at 11:11
  • 1
    @SanjidRahman dt[, row := .I ][, melt(.SD, id.cols = c("ID", "row"), measure.vars = c("X", "Y", "Z")) ][, value := replace(value, duplicated(value), NA), by = row ][, dcast(.SD, ID + row ~ variable) ][, !"row"]
    – s_baldur
    Commented Aug 20, 2019 at 11:14
3

An even shorter version of what basically is @akrun 's answer:

dt[t(apply(dt, 1, duplicated))] <- NA
3
  • This is terribly inefficient.
    – Roland
    Commented Aug 16, 2019 at 7:07
  • @Roland please elaborate or provide a more efficient solution if you have any. Do take in mind that the OP asked for a working solution, and didn't specify the size of the dataset. Nor did they ask for a solution aimed at very large datasets.
    – P1storius
    Commented Aug 16, 2019 at 7:23
  • 1
    apply deep-copies the whole dataset. The correct data.table way is already shown in the answer by sindri_baldur.
    – Roland
    Commented Aug 16, 2019 at 7:26
2

An option with base R

setDF(dt)
dt[] <- t(apply(dt, 1, function(x) replace(x, duplicated(x), NA)))
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.