12

What is the fastest way to write a vector to a file? I have a character vector that is ~2 million rows and that has rather large values (200 characters). I am currently doing

write(myVector, "myFile.txt")

But this is extremely slow. I have searched around for solutions but the fast writing functions (such as fwrite) only take a data frame/matrix as input. Thanks!

1
  • You could try readr::write_lines? Or you could make the vector a column of a data frame
    – Calum You
    Mar 12, 2018 at 22:43

3 Answers 3

15

After trying several options I found the fastest to be data.table::fwrite. Like @Gregor says in his first comment, it is faster by an order of magnitude, which is worth the extra package loaded. It is also one of the ones that produces bigger files. (The other one is readr::write_lines. Thanks to the comment by Calum You, I had forgotten this one.)

library(data.table)
library(readr)

set.seed(1)    # make the results reproducible
n <- 1e6
x <- rnorm(n)

t1 <- system.time({
    sink(file = "test_sink.txt")
    cat(x, "\n")
    sink()
})
t2 <- system.time({
    cat(x, "\n", file = "test_cat.txt")
})
t3 <- system.time({
    write(x, file = "test_write.txt")
})
t4 <- system.time({
    fwrite(list(x), file = "test_fwrite.txt")
})
t5 <- system.time({
    write_lines(x, "test_write_lines.txt")
})

rbind(sink = t1[1:3], cat = t2[1:3], 
      write = t3[1:3], fwrite = t4[1:3],
      readr = t5[1:3])
#       user.self sys.self elapsed
#sink        4.18    11.64   15.96
#cat         3.70     4.80    8.57
#write       3.71     4.87    8.64
#fwrite      0.42     0.02    0.51
#readr       2.37     0.03    6.66

In his second comment, Gregor notes that as.list and list behave differently. The difference is important. The former writes the vector as one row and many columns, the latter writes one column and many rows.

The speed difference is also noticeable:

fw1 <- system.time({
    fwrite(as.list(x), file = "test_fwrite.txt")
})
fw2 <- system.time({
    fwrite(list(x), file = "test_fwrite2.txt")
})

rbind(as.list = fw1[1:3], list = fw2[1:3])
#        user.self sys.self elapsed
#as.list      0.67     0.00    0.75
#list         0.19     0.03    0.11

Final clean up.

unlink(c("test_sink.txt", "test_cat.txt", "test_write.txt",
         "test_fwrite.txt", "test_fwrite2.txt", "test_write_lines.txt"))
3
  • 3
    Maybe worth calling out in the first sentence that, not only is fwrite fastest, it's fastest by more than 10x. readr is faster than write and cat, but only marginally. Mar 12, 2018 at 23:08
  • 3
    Also, your fwrite code is incorrect, you want list(x) not as.list(x). as.list() will write a CSV with one row and n columns, (and be a little slower). list(x) will give n rows and 1 column. Mar 12, 2018 at 23:10
  • @GregorThomas thanks for the list() vs as.list() note!
    – RTD
    Jul 13, 2020 at 11:29
6

I found writeBin to be twice as fast as fwrite. Try this:

 zz <- file("myFile.txt", "wb")
 writeBin( paste(myVector, collapse="\n"), zz ) 
  close(zz)

Using the same timing approach offered by Rui I get (older box):

            user.self sys.self elapsed
sink            9.650    7.900  17.418
cat             6.507    7.870  14.254
write           6.436    7.849  14.171
fwrite          0.500    0.051   0.593
write_lines     4.337    0.150   4.451
writeBin        0.238    0.006   0.242 
2

You could use data.table's fwrite:

library(data.table) # install if not installed already
fwrite(list(myVector), file = "myFile.csv")
1
  • 4
    Since OP has a vector, not a list or data frame, you'll need fwrite(list(myVector), "myFile.csv"). Mar 12, 2018 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.