8

Until recently when I used numpy methods like np.dot(A,B), only a single core was used. However, since today suddently all 8 cores of my linux machine are being used, which is a problem.

A minimal working example:

import numpy as np
N = 100

a = np.random.rand(N,N)
b = np.random.rand(N,N)

for i in range(100000):
    a = np.dot(a,b)

On my other laptop it works all fine on a single core. Could this be due to some new libraries?

This morning I updated matplotlib and cairocffi via pip, but that's all.

Any ideas how to go back to a single core?

Edit:

When I run

np.__config__.show()

I get the following output

openblas_info:
    libraries = ['openblas', 'openblas']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    library_dirs = ['/usr/local/lib']
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    library_dirs = ['/usr/local/lib']
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    library_dirs = ['/usr/local/lib']
blas_mkl_info:
  NOT AVAILABLE
blas_opt_info:
    libraries = ['openblas', 'openblas']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    library_dirs = ['/usr/local/lib']

2 Answers 2

6

This could be because numpy is linking against multithreaded openBLAS libraries. Try setting the global environment variable to set threading affinity as:

export OPENBLAS_MAIN_FREE=1

# Now run your python script.

Another workaround could to use ATLAS instead of OpenBLAS. Please see this post for more information (https://shahhj.wordpress.com/2013/10/27/numpy-and-blas-no-problemo/). This post proposes some other workarounds as well which might be worth trying.

8
  • Thanks. I ran the export command in my linux shell and the used python test.py to call the code. However, it did not change anything. I updated my answer with np.__config__.show(), if this is of any help.
    – physicsGuy
    Jul 29, 2016 at 13:35
  • Try setting the affinity using the OS module. Something like os.system(“taskset -p 0x1 %s” %os.getpid()) and then run your code (for reference this is in the post I linked to). Jul 29, 2016 at 13:37
  • If multiprocessing is indeed the problem, than you might want to also try using ATLAS rather than openBLAS. This should be easy to do. Please refer to the post for details on how to make numpy use ATLAS rather than openBLAS. Jul 29, 2016 at 13:39
  • I also followed the wordpress post. Using os.system("taskset -p 0x1 %s" %os.getpid()) gives the same output as in the post, however it does not change the behaviour.
    – physicsGuy
    Jul 29, 2016 at 15:09
  • Installing Atlas is not so easy for me, as I have no root permissions. Is there an good guide on how to install Atlas locally and link with numpy?
    – physicsGuy
    Jul 29, 2016 at 15:10
4

if you're using OpenBLAS try setting the OPENBLAS_NUM_THREADS environment variable = 1. OpenBLAS uses all CPU cores and is inefficient on small matrices.

export OPENBLAS_NUM_THREADS = 1

You can read more here: https://github.com/numpy/numpy/issues/8120

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.