A couple strategies that can be used separately or combined:
- Using
readLines
with fread
, you can read the .csv files in chunks.
- Convert the .csv files (possibly in chunks) into .fst files.
The idea is similar to the disk.frame package, which has been soft-deprecated, but the basic functionality is easy enough to do manually.
An example function that does both:
library(data.table)
library(fst)
csv_to_fst <- function(path, maxrow = 1e6, compress = 100) {
fl <- file(path, "r")
i <- 0L
dt <- fread(text = readLines(fl, maxrow + 1))
cols <- colnames(dt)
while (nrow(dt)) {
i <- i + 1L
setnames(dt, cols)
write.fst(dt, gsub(".csv$", paste0(i, ".fst"), path), compress)
dt <- fread(text = readLines(fl, maxrow))
}
close(fl)
i
}
With 100% compression, I've seen 10x+ size reduction, so the .fst files should take up considerably less space on disk. Reading .fst files is fast, and read.fst
offers the additional ability to read in only specified columns and row ranges. [Side note: If you plan to do a lot of filtering on a particular column (like your "book_author" column), you could sort the data.table
s by that column as you are creating the .fst files. Then, to filter, you could read in just the filtering column, find the span of rows needed, then read in just those rows for rest of the columns that you need.]
Then, say we had the following .csv file, we could convert it into multiple .fst files. Here, I break it into 15 files of 10 rows each:
path <- "C:/temp/iris.csv"
data(iris)
fwrite(iris, path)
(i <- 1:csv_to_fst(path, 10))
#> [1] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
A simple function to operate on multiple .fst files and combine the results:
apply_fst <- function(paths, f) {
rbindlist(lapply(paths, \(x) f(read.fst(x, as.data.table = TRUE))))
}
Example usage:
f <- function(dt) {
dt[Sepal.Length < mean(Sepal.Length) & Sepal.Width > mean(Sepal.Width)]
}
apply_fst(sapply(i, \(i) gsub(".csv$", paste0(i, ".fst"), path)), f)
#> Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#> <num> <num> <num> <num> <char>
#> 1: 4.6 3.4 1.4 0.3 setosa
#> 2: 5.1 3.8 1.5 0.3 setosa
#> 3: 4.6 3.6 1.0 0.2 setosa
#> 4: 4.8 3.4 1.9 0.2 setosa
#> 5: 5.0 3.4 1.6 0.4 setosa
#> 6: 4.9 3.6 1.4 0.1 setosa
#> 7: 5.6 2.9 3.6 1.3 versicolor
#> 8: 5.6 3.0 4.5 1.5 versicolor
#> 9: 5.8 2.7 4.1 1.0 versicolor
#> 10: 5.9 3.2 4.8 1.8 versicolor
#> 11: 6.0 2.9 4.5 1.5 versicolor
#> 12: 5.4 3.0 4.5 1.5 versicolor
#> 13: 5.6 3.0 4.1 1.3 versicolor
#> 14: 6.3 3.3 6.0 2.5 virginica
#> 15: 6.5 3.0 5.8 2.2 virginica
#> 16: 6.5 3.2 5.1 2.0 virginica
#> 17: 6.4 3.2 5.3 2.3 virginica
#> 18: 6.5 3.0 5.5 1.8 virginica
#> 19: 6.1 3.0 4.9 1.8 virginica
#> 20: 6.3 3.4 5.6 2.4 virginica
#> 21: 6.4 3.1 5.5 1.8 virginica
#> 22: 6.2 3.4 5.4 2.3 virginica
You'll likely want to modify the above functions to meet your needs, but this should give you a good starting point.