5

I have a matrix with the dimension of 100 million records and 100 columns.

Now I want to multiply that matrix by rowwise.

My sample code for matrix multiplication is

df<-as.matrix(mtcars)
result<-apply(df,1,prod)

The above syntax is very slow in my case.

I tried rowprods function in Rfast package.

result<-rowprods(mtcars)

But the above function giving me space issues.

NOTE: I have 8 GB ram in my system.

8
  • 1
    Is really a matrix or a data.table ? (I'm asking because you've added data.table tag)
    – digEmAll
    Commented Feb 20, 2018 at 7:16
  • 2
    If this is a matrix try matrixStats::rowProds(df). Also, what are those mysterious "space issues"? Commented Feb 20, 2018 at 7:17
  • 1
    Have you tried run rowprods by chunks of 1 or 10 million ? Commented Feb 20, 2018 at 7:18
  • It is a matrix only. Why I add data.table is, it performs operation much faster.
    – RSK
    Commented Feb 20, 2018 at 7:19
  • 3
    100 million records and 100 columns is 76 GBs. Can you have your data in memory in the first place?
    – F. Privé
    Commented Feb 20, 2018 at 7:35

4 Answers 4

6

If you have a matrix that is too large to fit in memory, you can use package bigstatsr (disclaimer: I'm the author) to use data stored on your disk (instead of the RAM). Using function big_apply enables you to apply standard R functions on data blocks (and to combine them).

library(bigstatsr)
fbm <- FBM(10e6, 100)
# inialize with random numbers
system.time(
  big_apply(fbm, a.FUN = function(X, ind) {
    print(min(ind))
    X[, ind] <- rnorm(nrow(X) * length(ind))
    NULL
  }, a.combine = 'c')
) # 78 sec

# compute row prods, possibly in parallel
system.time(
  prods <- big_apply(fbm, a.FUN = function(X, ind) {
    print(min(ind))
    matrixStats::rowProds(X[ind, ])
  }, a.combine = 'c', ind = rows_along(fbm),
  block.size = 100e3, ncores = nb_cores())  
) # 22 sec with 1 core and 18 sec with 6 cores
0
3

Try package data.table with Reduce. That might avoid internal copies of a 1e10 length vector.

library(data.table)
df <- data.table(df, keep.rownames=TRUE)
df[, rowprods:= Reduce("*", .SD), .SDcols = -1]
df[, .(rn, rowprods)]
#                     rn   rowprods
# 1:           Mazda RX4          0
# 2:       Mazda RX4 Wag          0
# 3:          Datsun 710  609055152
# 4:      Hornet 4 Drive          0
# 5:   Hornet Sportabout          0
# 6:             Valiant          0
# 7:          Duster 360          0
# 8:           Merc 240D          0
# 9:            Merc 230          0
#10:            Merc 280          0
#11:           Merc 280C          0
#12:          Merc 450SE          0
#13:          Merc 450SL          0
#14:         Merc 450SLC          0
#15:  Cadillac Fleetwood          0
#16: Lincoln Continental          0
#17:   Chrysler Imperial          0
#18:            Fiat 128  470578906
#19:         Honda Civic  564655046
#20:      Toyota Corolla  386281789
#21:       Toyota Corona          0
#22:    Dodge Challenger          0
#23:         AMC Javelin          0
#24:          Camaro Z28          0
#25:    Pontiac Firebird          0
#26:           Fiat X1-9  339825992
#27:       Porsche 914-2          0
#28:        Lotus Europa 1259677924
#29:      Ford Pantera L          0
#30:        Ferrari Dino          0
#31:       Maserati Bora          0
#32:          Volvo 142E 1919442833
#                     rn    rowsums

However, 8 GB RAM (minus what your OS and other software needs) is not much if you want to work with data of this size. R sometimes needs to make internal copies to use your data.

3
  • Do you disagree with David that matrix operations are faster than data.table operations ? Also you may want to name your rowsums column rowprods. Commented Feb 20, 2018 at 7:34
  • 2
    I disagree with David for this specific example. Matrix algebra is probably always faster than alternatives (if no additional data copies are needed to apply it), but OP's example is not matrix algebra and I think the data is copied. (Don't know about the rowprods function though.) Using * 99 times in a loop should be pretty fast.
    – Roland
    Commented Feb 20, 2018 at 7:36
  • 1
    I see that matrixStats::rowProds is working without any issues but it is also taking significant amount of time to perform the operation.
    – RSK
    Commented Feb 20, 2018 at 8:10
3

Some timings for reference

library(matrixStats)
library(inline)
library(data.table)
#devtools::install_github("privefl/bigstatsr")
library(bigstatsr)
library(RcppArmadillo)
library(microbenchmark)
set.seed(20L)
N <- 1e6
dat <- matrix(rnorm(N*100),ncol=100)

fbm <- FBM(N, 100)
big_apply(fbm, a.FUN = function(X, ind) {
    print(min(ind))
    X[, ind] <- rnorm(nrow(X) * length(ind))
    NULL
}, a.combine = 'c')   

bigstatsrMtd <- function() {
    prods <- big_apply(fbm, a.FUN = function(X, ind) {
        print(min(ind))
        matrixStats::rowProds(X[ind, ])
    }, a.combine = 'c', ind = rows_along(fbm),
        block.size = 100e3, ncores = nb_cores())  
}

df <- data.table(as.data.frame(dat), keep.rownames=TRUE)
data.tableMtd <- function() {
    df[, rowprods:= Reduce("*", .SD), .SDcols = -1]
    df[, .(rn, rowprods)]    
}

code <- '
  arma::mat prodDat = Rcpp::as<arma::mat>(dat);
  int m = prodDat.n_rows;
  int n = prodDat.n_cols;
  arma::vec res(m);
  for (int row=0; row < m; row++) {
    res(row) = 1.0;
    for (int col=0; col < n; col++) {
      res(row) *= prodDat(row, col);
    }
  }
  return Rcpp::wrap(res);
'
rcppProd <- cxxfunction(signature(dat="numeric"), code, plugin="RcppArmadillo")

rcppMtd <- function() {
    rcppData <- rcppProd(dat)                # generated by C++ code
}

baseMtd <- function() {
    apply(dat, 1, prod)   
}

microbenchmark(bigstatsrMtd(),
    data.tableMtd(),
    rcppMtd(),
    baseMtd(),
    times=5L
)

Note: Compiling the function in cxxfunction seems to take some time

Here are the timing results:

# Unit: milliseconds
#            expr       min        lq      mean    median        uq       max
#  bigstatsrMtd() 4519.1861 4993.0879 5296.7000 5126.2282 5504.3981 6340.5995
# data.tableMtd()  443.1946  444.9686  690.3703  493.2399  513.4787 1556.9695
#       rcppMtd()  787.9488  799.1575  828.3647  809.0645  871.0347  874.6178
#       baseMtd() 5658.1424 6208.5123 6232.0040 6331.7431 6458.6806 6502.9417
12
  • If you have a standard R matrix dat you can do fbm <- big_copy(dat).
    – F. Privé
    Commented Feb 20, 2018 at 8:59
  • thanks, @F.Privé i have left out creation of fbm in the timings.
    – chinsoon12
    Commented Feb 20, 2018 at 9:01
  • And note that one can write some Rcpp code to be used on the FBM too.
    – F. Privé
    Commented Feb 20, 2018 at 9:03
  • given github.com/privefl/bigstatsr/blob/master/DESCRIPTION, prob not surprising :)
    – chinsoon12
    Commented Feb 20, 2018 at 9:04
  • I tried rcppMtd(), but I am getting an error. My error message is: error: Mat::operator(): index out of bounds Error in rcppProd(dat) : Mat::operator(): index out of bounds my code is 'dat<-matrix(1:100,10,10)'
    – RSK
    Commented Feb 21, 2018 at 9:19
1

The Rfast command "rowprods" accepts a matrix, not a data.frame. Secondly, any row or colprods command will have numerical overflow errors. So ti best to use Rfast::colprods(x, method = "expsumlog").

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.