1

Lets say I have a list as following:

points = np.array( [ . . . ] , [ . . .], [ . . .] )  # shape is (500000, 1000)

and another list as

target = np.array([ . . .])  #shape is (1000,)

Now I can compute the L2 norm as following:

norm = np.linalg.norm(points - target, axis=1)

This works perfectly but its super slow when I want execute this for 100K target values. For the moment I get a target value from the targets list and calculate norm for each target.

Is there a quick way to do this?

3
  • 3
    500000 * 1000 is a lot of elements. 500k * 100k is even more.
    – Reti43
    Jan 6, 2016 at 9:24
  • For 64-bit numbers, 500k * 1k elements is already almost 4GB. With 500k * 100k elements, you'll probably run out of memory. Maybe you could try using something like pytables? Here's another link that could help.
    – Praveen
    Jan 6, 2016 at 11:34
  • Does the last sentence mean you are doing an iteration on top of the norm calculation? If so show that code. Otherwise answers will focus on memory size.
    – hpaulj
    Jan 6, 2016 at 12:21

1 Answer 1

3

Yep, broadcasting.

deltas = points[:, None, :] - targets[None, :, :]  # shape (n_points, n_targets, n_dims)
norm = np.linalg.norm(deltas, axis=2)  # shape (n_points, n_targets)

But you might use up all your memory if you have 500k points and 100k targets and 1000-D (that's 500000*100000*1000*8 B = 400 TB of RAM you need!)

Chances are your current slowdown is due to the actual cost of all that computation and not the fact that you're looping in Python. Consider using GPU computation (maybe Theano) to speed things up if you need.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.