104

I have a script that reads in data from a CSV file into a data.table and then splits the text in one column into several new columns. I am currently using the lapply and strsplit functions to do this. Here's an example:

library("data.table")
df = data.table(PREFIX = c("A_B","A_C","A_D","B_A","B_C","B_D"),
                VALUE  = 1:6)
dt = as.data.table(df)

# split PREFIX into new columns
dt$PX = as.character(lapply(strsplit(as.character(dt$PREFIX), split="_"), "[", 1))
dt$PY = as.character(lapply(strsplit(as.character(dt$PREFIX), split="_"), "[", 2))

dt 
#    PREFIX VALUE PX PY
# 1:    A_B     1  A  B
# 2:    A_C     2  A  C
# 3:    A_D     3  A  D
# 4:    B_A     4  B  A
# 5:    B_C     5  B  C
# 6:    B_D     6  B  D 

In the example above the column PREFIX is split into two new columns PX and PY on the "_" character.

Even though this works just fine, I was wondering if there is a better (more efficient) way to do this using data.table. My real datasets have >=10M+ rows, so time/memory efficiency becomes really important.


UPDATE:

Following @Frank's suggestion I created a larger test case and used the suggested commands, but the stringr::str_split_fixed takes a lot longer than the original method.

library("data.table")
library("stringr")
system.time ({
    df = data.table(PREFIX = rep(c("A_B","A_C","A_D","B_A","B_C","B_D"), 1000000),
                    VALUE  = rep(1:6, 1000000))
    dt = data.table(df)
})
#   user  system elapsed 
#  0.682   0.075   0.758 

system.time({ dt[, c("PX","PY") := data.table(str_split_fixed(PREFIX,"_",2))] })
#    user  system elapsed 
# 738.283   3.103 741.674 

rm(dt)
system.time ( {
    df = data.table(PREFIX = rep(c("A_B","A_C","A_D","B_A","B_C","B_D"), 1000000),
                     VALUE = rep(1:6, 1000000) )
    dt = as.data.table(df)
})
#    user  system elapsed 
#   0.123   0.000   0.123 

# split PREFIX into new columns
system.time ({
    dt$PX = as.character(lapply(strsplit(as.character(dt$PREFIX), split="_"), "[", 1))
    dt$PY = as.character(lapply(strsplit(as.character(dt$PREFIX), split="_"), "[", 2))
})
#    user  system elapsed 
#  33.185   0.000  33.191 

So the str_split_fixed method takes about 20X times longer.

1
  • I think doing the operation outside of the data.table first might be better. If you use the stringr package, this is the command: str_split_fixed(PREFIX,"_",2). I'm not answering because I haven't tested the speedup...Or, in one step: dt[,c("PX","PY"):=data.table(str_split_fixed(PREFIX,"_",2))]
    – Frank
    Aug 9, 2013 at 21:17

5 Answers 5

158

Update: From version 1.9.6 (on CRAN as of Sep'15), we can use the function tstrsplit() to get the results directly (and in a much more efficient manner):

require(data.table) ## v1.9.6+
dt[, c("PX", "PY") := tstrsplit(PREFIX, "_", fixed=TRUE)]
#    PREFIX VALUE PX PY
# 1:    A_B     1  A  B
# 2:    A_C     2  A  C
# 3:    A_D     3  A  D
# 4:    B_A     4  B  A
# 5:    B_C     5  B  C
# 6:    B_D     6  B  D

tstrsplit() basically is a wrapper for transpose(strsplit()), where transpose() function, also recently implemented, transposes a list. Please see ?tstrsplit() and ?transpose() for examples.

See history for old answers.

5
  • Thanks Arun. I hadn't thought of the method of first creating the list, then the index and then the columns as described in "a_spl". I always thought that doing everything in a single line was the best way. Just out of curiosity why does the index way work so much faster ? Aug 12, 2013 at 3:34
  • @Arun, related to this question, what are some of the pitfalls you would see in a function like I've written here: gist.github.com/mrdwab/6873058 Basically, I've made use of fread, but to do so, I had to use a tempfile (which would seem like it would be a bottleneck) since it doesn't seem like fread has an equivalent to a text argument. Testing with this sample data, its performance is between your a_spl and a_sub approaches. Jun 2, 2014 at 18:11
  • 4
    I was wondering how one could guess the number of columns on the LHS of := and dynamically create the names of the new columns based on the grep tstrsplit occurences
    – amonk
    Jul 5, 2017 at 13:55
  • Is there an efficient way to drop the original PREFIX column all at once using this approach? I mean that might be faster or use less memory in process than chaining or doing it as a separate operation.
    – Mark E.
    Jan 8, 2021 at 22:05
  • Trying to revive this old answer. How do you split by position (5 characters in c1, the rest in c2) directly in tstrsplit()? I made a turnaround using beforehand gsub to insert an empty space after 5 characters in my string, and then apply tstrsplit(x, " ", fixed=TRUE) Oct 6, 2022 at 13:03
16

I add answer for someone who do not use data.table v1.9.5 and also want an one line solution.

dt[, c('PX','PY') := do.call(Map, c(f = c, strsplit(PREFIX, '-'))) ]
8

Using splitstackshape package:

library(splitstackshape)
cSplit(df, splitCols = "PREFIX", sep = "_", direction = "wide", drop = FALSE)
#    PREFIX VALUE PREFIX_1 PREFIX_2
# 1:    A_B     1        A        B
# 2:    A_C     2        A        C
# 3:    A_D     3        A        D
# 4:    B_A     4        B        A
# 5:    B_C     5        B        C
# 6:    B_D     6        B        D
0
6

We could try:

library(data.table)  
cbind(dt, fread(text = dt$PREFIX, sep = "_", header = FALSE))
    #    PREFIX VALUE V1 V2
    # 1:    A_B     1  A  B
    # 2:    A_C     2  A  C
    # 3:    A_D     3  A  D
    # 4:    B_A     4  B  A
    # 5:    B_C     5  B  C
    # 6:    B_D     6  B  D
4

With tidyr the solution is:

separate(df,col = "PREFIX",into = c("PX", "PY"), sep = "_")
4

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.