5

I have a cell, Data, it contains three double arrays,

 Data = 

[74003x253 double]    [8061x253 double]    [7241x253 double]

I'm using a loop to read these arrays and perform some functions,

for ii = 1 : 3
    D = Data {ii} ;
    m = mean (D') ;
    // rest of the code
end

Which gets a warning for mean and says:

consider using different DIMENSION input argument for MEAN

However when I change it to,

for ii = 1 : 3
    D = Data {ii}' ;
    m = mean (D) ;
    // rest of the code
end

I get Out of memory error.

Comparing two codes, can someone explain what happens?

It seems that I get the error only with a Complex conjugate transpose (my data is real valued).

6
  • On which iteration does Out of memory error occur? Can you tell the value of ii before failure?
    – fdermishin
    Oct 19, 2014 at 18:56
  • @user502144, Actually, out of memory error doesn't mention the line (I had made a mistake to mention it for the error, which I corrected). However, with the only a transpose change in these two codes I can make the error go away.
    – Rashid
    Oct 19, 2014 at 19:02
  • For one ' has nothing to do with complex numbers - it transposes matrices which also means it needs to make a copy (needs extra ram). My guess is, since Matlab uses a very advanced just in time compiler, it recognizes the first cases and correctly replaces it with mean(D,2).
    – bdecaf
    Oct 22, 2014 at 8:42
  • @bdecaf ' is not the same as transpose .'. The operator ' is actually the hermitian (complex) transpose.
    – ederag
    Oct 24, 2014 at 22:30
  • @huntj, as I've mentioned my data is real valued.
    – Rashid
    Oct 24, 2014 at 22:33

3 Answers 3

8
+50

To take the mean for the n:th dimension it is possible use mean(D,n) as already stated. Regarding the memory consumption, I did some tests monitoring with the windows resource manager. The output was kind of expected.

When doing the operation D=Data{ii} only minimum memory is consumed since here matlab does no more than copying a pointer. However, when doing a transpose, matlab needs to allocate more memory to store the matrix D, which means that the memory consumption increases.

However, this solely does not cause a memory overflow, since the transpose is done in both cases.

Case 1

Separately inD = Data{ii}';

Case 2

in D = Data {ii}; m = mean(D');

The difference is that in case 2 matlab only creates a temporary copy of Data{ii}' which is not stored in the workspace. The memory allocated is the same in both cases, but in case 1 Data{ii}' is stored in D. When the memory later increases this can cause a memory overflow.

The memory consumption of D is not that bad (< 200 Mb), but the guess is that the memory got high already and that this was enough to give memory overflow.

6

The warning message means that instead of,

m = mean (D') ;

you should do:

m = mean (D,2);

This will take the mean along the second dimension, leaving you with a column vector the length of size(D,1).

I don't know why you only get the out of memory error when you do D = Data {ii}'. Perhaps it's becauase when you have it in side of mean (m = mean (D') ; the JIT manages to optimize somehow and save you wasted memory.

8
  • I didn't know about mean(D,2), Thanks. And for the memory error, it's really strange.
    – Rashid
    Oct 19, 2014 at 21:52
  • 2
    @Kamtal: Whenever reading MATLAB documentation, the phrase "... operates on the first non-singleton dimension" means that the operation is a "reduction, or fold, or multi-input single-output operator", and that the operator will group together elements taken from the "first non-singleton dimension".
    – rwong
    Oct 19, 2014 at 22:32
  • 2
    If I had to guess, D=Data{i}' actually creates a full transposed copy of the data before passing it to mean(D). On the other hand, D=Data{i} first creates a shared-data copy (think lazy copy-on-write), then the call mean(D') is passed a "temporary" expression, which is a possible target for JIT optimization (by recognizing that mean can compute the result without actually transposing the matrix in a new copy, it suffices to traverse the existing matrix in strides of appropriate length)
    – Amro
    Oct 21, 2014 at 20:12
  • @Amro Well stated, that's what I was thinking regarding JIT compilation -- the ability to compute it along the other dimension based on the clue it gets from ' without actually transposing any data. But can this be proven/demonstrated? I don't think there's a good way that wouldn't possibly influence JIT optimization (like debugging would certainly do). Anyway, I'm skeptical because mean is not a built-in.
    – chappjc
    Oct 21, 2014 at 20:15
  • 1
    @Amro My Composer XE trial is expired and I'm not sure if VS gives much memory info. I'd suggest the OP try since I can't reproduce the issue right now and I'm not eager to start my system memory paging out. :)
    – chappjc
    Oct 21, 2014 at 20:25
1

Here are some ways of doing this:

for i = 1 : length(Data)
   % as chappjc recommends this is an excellent solution
   m = mean(Data{i}, 2);
end

Or if you want the transpose and you know the data is real (not complex)

for i = 1 : length(Data)
   m = mean(Data{i}.');
end

Note, the dot before the transpose.

Or, skip the loop all together

m = cellfun(@(d) mean(d, 2), Data, 'uniformoutput', false);

When you do:

D = Data{i}'

Matlab will create a new copy of your data. This will allocate 74003x253 doubles, which is about 150MB. As patrick pointed out, given that you might have other data you can easily exceed the allowed memory allocation usage (especially on a 32-bit machine).

If you are running with memory problems, the computations are not sensitive, you may consider using single precision instead of double, i.e.:

data{i} = single(data{i});

Ideally, you want to do the single precision at point of allocation to avoid unnecessary new allocation and copies.

Good luck.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.