7

I'm seeing odd memory usage when using assignment by reference by group in a data.table. Here's a simple example to demonstrate (please excuse the triviality of the example):

N <- 1e6
dt <- data.table(id=round(rnorm(N)), value=rnorm(N))

gc()
for (i in seq(100)) {
  dt[, value := value+1, by="id"]
}
gc()
tables()

which produces the following output:

> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells  303909 16.3     597831 32.0   407500 21.8
Vcells 2442853 18.7    3260814 24.9  2689450 20.6
> for (i in seq(100)) {
  +   dt[, value := value+1, by="id"]
  + }
> gc()
used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells   315907  16.9     597831  32.0   407500  21.8
Vcells 59966825 457.6   73320781 559.4 69633650 531.3
> tables()
NAME      NROW MB COLS     KEY
[1,] dt   1,000,000 16 id,value    
Total: 16MB

So about 440MB of used Vcells memory were added after the loop. This memory is not accounted for after removing the data.table from memory:

> rm(dt)
> gc()
used  (Mb) gc trigger (Mb) max used  (Mb)
Ncells   320888  17.2     597831   32   407500  21.8
Vcells 57977069 442.4   77066820  588 69633650 531.3
> tables()
No objects of class data.table exist in .GlobalEnv

The memory leak seems to disappear when removing the by=... from the assignment:

>     gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells  312955 16.8     597831 32.0   467875 25.0
Vcells 2458890 18.8    3279586 25.1  2704448 20.7
>     for (i in seq(100)) {
  +       dt[, value := value+1]
  +     }
>     gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells  322698 17.3     597831 32.0   467875 25.0
Vcells 2478772 19.0    5826337 44.5  5139567 39.3
>     tables()
NAME      NROW MB COLS     KEY
[1,] dt   1,000,000 16 id,value    
Total: 16MB

To summarize, two questions:

  1. Am I missing something or is there a memory leak?
  2. If there is indeed a memory leak, can anyone suggest a workaround that lets me use assignment by reference by group without the memory leak?

For reference, here's the output of sessionInfo():

R version 3.0.2 (2013-09-25)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8   
[6] LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
  [1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
  [1] data.table_1.8.10

loaded via a namespace (and not attached):
  [1] tools_3.0.2
3
  • 1
    If there is the possibility of a bug, please specify what versions of R and data.table you are using.
    – flodel
    Commented Dec 3, 2013 at 12:12
  • Seems to happen only when the number of elements/group is uneven. If you do: dt <- data.table(id=1:N, value=rnorm(N)) or any other way so that the number of elements per value of id is constant, then this doesn't happen.
    – Arun
    Commented Dec 3, 2013 at 12:46
  • @flodel - thanks, I'll edit the post to include these details.
    – ytsaig
    Commented Dec 4, 2013 at 7:17

1 Answer 1

6

UPDATE from Matt - Now fixed in v1.8.11. From NEWS :

Long outstanding (usually small) memory leak in grouping fixed. When the last group is smaller than the largest group, the difference in those sizes was not being released. Most users run a grouping query once and will never have noticed, but anyone looping calls to grouping (such as when running in parallel, or benchmarking) may have suffered, #2648. Test added.

Many thanks to vc273, Y T and others.



From Arun ...

Why was this happening?

I wish I had come across this post before sitting on this issue. Nevertheless, a nice learning experience. Simon Urbanek summarises the issue pretty succinctly, that it's not a memory leak but bad-reporting of memory used/freed. I had the feeling this is what was happening.


What's the reason for this to happen in data.table? This part is on identifying the portion of code from dogroups.c responsible for the apparent memory increase.

Okay, so after some tedious testing, I think I've managed to at least find as to what the reason is for this to happen. Hopefully someone can help me get there from this post. My conclusion is that this is not a memory leak.

The short explanation is that this seems to be an effect of the usage of SETLENGTH function (from R's C-interface) in data.table's dogroups.c .

In data.table, when you use by=..., for example,

set.seed(45)
DT <- data.table(x=sample(3, 12, TRUE), id=rep(3:1, c(2,4,6)))
DT[, list(y=mean(x)), by=id]

Corresponding to id=1, the values of "x" (=c(1,2,1,1,2,3)) has to be picked. This means, having to allocate memory for .SD (all columns not in by) per by value.

To overcome this allocation for each group in by, data.table accomplishes this cleverly by first allocating .SD with the length of the largest group in by (which here is corresponding to id=1, length 6). Then, we could, for each value of id, re-use the (overly) allocated data.table and by using the function SETLENGTH we can just adjust the length to the length of current group. Note that, by doing this, no memory is being actually allocated here, except just the once that's allocated for the biggest group.

But what seems strange is that when the number of elements for each group in by all have the same number of items, nothing special seems to be happening with regard to gc() output. However, when they aren't the same, gc() seems to report increasing usage in Vcells. This is in spite of the fact that no extra memory is being allocated in both cases.

To illustrate this point, I've written a C-code that mimics the SETLENGTH function usage in dogroups.c in `data.table.

// test.c
#include <R.h>
#define USE_RINTERNALS
#include <Rinternals.h>
#include <Rdefines.h>

int sizes[100];
#define SIZEOF(x) sizes[TYPEOF(x)]

// test function - no checks!
SEXP test(SEXP vec, SEXP SD, SEXP lengths)
{
    R_len_t i, j;
    char before_address[32], after_address[32];
    SEXP tmp, ans;
    PROTECT(tmp = allocVector(INTSXP, 1));
    PROTECT(ans = allocVector(STRSXP, 2));
    snprintf(before_address, 32, "%p", (void *)SD);
    for (i=0; i<LENGTH(lengths); i++) {
        memcpy((char *)DATAPTR(SD), (char *)DATAPTR(vec), INTEGER(lengths)[i] * SIZEOF(tmp));
        SETLENGTH(SD, INTEGER(lengths)[i]);
        // do some computation here.. ex: mean(SD)
    }
    snprintf(after_address, 32, "%p", (void *)SD);
    SET_STRING_ELT(ans, 0, mkChar(before_address));
    SET_STRING_ELT(ans, 1, mkChar(after_address));
    UNPROTECT(2);
    return(ans);
}

Here vec is equivalent to any data.table dt and SD is equivalent to .SD and lengths is the length of each group. This is just a dummy program. Basically for each value of lengths, say n, the first n elements are copied from vec on to SD. Then one can compute whatever one wants on this SD (which is not done here). For our purposes, the address of SD before and after operation using SETLENGTH are being returned, to illustrate that there's no copy being made by SETLENGTH.

Save this file as test.c and then compile it as follows from terminal:

R CMD SHLIB -o test.so test.c

Now, open a new R-session, go to the path where test.so exists and then type:

dyn.load("test.so")
require(data.table)
set.seed(45)
max_len <- as.integer(1e6)
lengths <- as.integer(sample(4:(max_len)/10, max_len/10))
gc()
vec <- 1:max_len
for (i in 1:100) {
    SD <- vec[1:max(lengths)]
    bla <- .Call("test", vec, SD, lengths)
    print(gc())
}

Note that for each i here, .SD will be allocated a different memory location and that's being replicated here by assigning SD for each i.

By running this code, you'll find that 1) the two values returned are identical for each i to that of address(SD) and 2) Vcells used Mb keeps increasing. Now, remove all variables from the workspace with rm(list=ls()) and then do gc(), you'll find that not all memory is being restored/freed.

Initial:

          used (Mb) gc trigger (Mb) max used (Mb)
Ncells  332708 17.8     597831 32.0   467875 25.0
Vcells 1033531  7.9    2327578 17.8  2313676 17.7

After 100 runs:

          used (Mb) gc trigger (Mb) max used (Mb)
Ncells  332912 17.8     597831 32.0   467875 25.0
Vcells 2631370 20.1    4202816 32.1  2765872 21.2

After rm(list=ls()) and gc():

          used (Mb) gc trigger (Mb) max used (Mb)
Ncells  341275 18.3     597831 32.0   467875 25.0
Vcells 2061531 15.8    4202816 32.1  3121469 23.9

If you remove the line SETLENGTH(SD, ...) from the C-code, and run it again, you'll find that there's no change in the Vcells.

Now as to why SETLENGTH on grouping with non-identical group lengths has this effect, I'm still trying to understand - check out the link in the edit above.

5
  • Thanks for looking under the hood and digging up the reason. In addition to your proposed test above, I ran a more brute force test by cranking up the the number of iterations to 1000 and monitoring memory usage using the OS facilities rather than gc(). Indeed at the end gc reported over 5GB of used memory while the OS reported that R never used more than a few hunderd MBs. Thanks also for the useful pointer to the R devel post, though it's a bit disheartening to learn that we can't trust the numbers coming out of gc(), particularly given how memory hungry R is.
    – ytsaig
    Commented Dec 4, 2013 at 7:52
  • 2
    It could be a slow real leak actually. A similar leak was fixed by the finalizer at the top of assign.c, but a few reports since then suggest a problem remains but I never got to the bottom of it (e.g. here). Looking again now with fresh eyes, dogroups.c seems to be missing a SETLENGTH at the end on .SD and I back to their original lengths (the size of the largest group) before returning. If that is all it is I'll kick myself for not spotting that before!
    – Matt Dowle
    Commented Dec 4, 2013 at 10:52
  • @MattDowle, hm, there seems to be absolutely no increase in the memory used by r-session process on my mac (or any free/active/inactive memory). This leads me to think that it's not actually a leak. That being said, I did not realise that simply setting the length back could ensure the "correct" reporting of memory. That's brilliant! I'll check this out.
    – Arun
    Commented Dec 4, 2013 at 11:00
  • 1
    If you repeatedly do DT[,j,by] on a larger dataset (and the largest group needs to be bigger) after possibly 30 mins you should see R's RAM usage grow i.e. leak. Best with a fresh R session each time to see it. If this theory is correct, a large table with two groups one of which is just 1 row, the other n-1 rows, and the 1 row group is computed by dogroups second - that should leak the worst. The size of the last group relative to the largest should make a difference, if correct. If so, adding SETLENGTH to the end of dogroups should fix the leak.
    – Matt Dowle
    Commented Dec 4, 2013 at 11:25
  • Is there a guarantee that when you decrease the length, the rest of the data won't get overridden?
    – eddi
    Commented Dec 4, 2013 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.