1

I'm in a situation where I have a large sized matrix a. The size of a is 2708*2708 and type is <class 'scipy.sparse.csr.csr_matrix'>. I want to calculate some updating equation using a.

And the calculation process is like this:

adj = (a.dot(a.T) + a.T.dot(a)) / LA.norm(a.dot(a.T) + a.T.dot(a))
for k in range(1, 10):
    adj = (a.dot(adj).dot(a.T) +a.T.dot(adj).dot(a)) / (LA.norm(a.dot(adj).dot(a.T) + a.T.dot(eadj).dot(a)))

But this gives me

zsh: killed

I guess the size of matrix causing this but how can I prevent this?

Is there a way to finish this calculation without killed error?

1
  • It's not clear you need to keep the entire list anyway; you never need any value except the previously computed one.
    – chepner
    Jul 3, 2020 at 0:44

2 Answers 2

1

You are attempting to use about 1000x more memory than you appear to need. You don't need to keep the entire list zk, because you never use any value except zk[k-1]. Just keep a reference to the last value computed, and replace that with the next value at each iteration.

Don't put a conditional in the loop to handle the first iteration specially; just pull that out as a pre-loop computation.

b = (a.dot(a.T) + a.T.dot(a)) / LA.norm((a.dot(a.T) + a.T.dot(a)))

for _ in range(1, 1000):  # One fewer iteration
   
    b = ((a.dot(b).dot(a.T) + a.T.dot(b).dot(a))
         / LA.norm((a.dot(b).dot(a.T) + a.T.dot(b).dot(a))))

print(b)

You also appear to be doing more work than necessary; the same value is used as the numerator and the denominator's argument at each step, but you compute it twice.

tmp1 = a.dot(a.T) + a.T.dot(a)
b = tmp1 / LA.norm(tmp1)

for _ in range(1, 1000):
    tmp1 = a.dot(b).dot(a.T) + a.T.dot(b).dot(a)
    b = tmp1 / LA.norm(tmp1)

print(b)
1
  • I tried coding not to use an array, like yours, but still got same result killed
    – port trum
    Jul 3, 2020 at 1:03
0

For the code posted, I get an output on my system. I am using a zsh shell and python 3.6. I am assuming you are using numpy as there is no mention of the same in the question.

There might be an issue with something else, not your code. Generally, a process is killed if it runs out of memory. You can track the memory of your code using top, in another terminal. Or you can try using a smaller sized matrix first to debug.

2
  • I first tried for a 3*3 matrix to see if that code would work, and yes, it worked. But for 2708*2708, killed appered. So the problem should be the size of matrix and memory. And I checked the memory usage, PID COMMAND %CPU TIME #TH #WQ #PORT MEM PURG CMPRS PGRP 66291 Python 99.7 01:13.28 2/1 0 15 35G+ 0B 33G+ 6629. and the type of a is <class 'scipy.sparse.csr.csr_matrix'>.
    – port trum
    Jul 2, 2020 at 14:12
  • You can try calculating the same in batches, then. Split your problem into subproblems which use smaller matrices.
    – Parth Shah
    Jul 2, 2020 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.