7

I am trying to run two tensorflow models in parallel in the same process.

In Tensorflow 1.x we could do e.g. Keras Tensorflow - Exception while predicting from multiple threads

graph = tf.Graph()
with graph.as_default():
    session = tf.Session()
    with session.as_default():
        # Create and use the model ...

In Tensorflow 2.0, the session has been removed.

When I try to run two models in parallel within the same process, Tensorflow crashes. Is there a way around this problem is TF 2.0?

Error message:

2019-09-18 16:26:56.978352: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.981418: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.983202: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.985757: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.987541: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.990140: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.992499: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.994886: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.997778: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.997802: W tensorflow/stream_executor/stream.cc:1919] attempting to perform BLAS operation using StreamExecutor without BLAS support
2019-09-18 16:26:56.997829: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Blas GEMM launch failed : a.shape=(1, 2520), b.shape=(2520, 2520), m=1, n=2520, k=2520
     [[{{node dense/MatMul}}]]
2019-09-18 16:26:57.000709: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:57.000744: W tensorflow/stream_executor/stream.cc:1919] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
  File 
    result = model.predict([numpy.array([inV,]), numpy.array([inS,])])
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 915, in predict
    use_multiprocessing=use_multiprocessing)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 722, in predict
    callbacks=callbacks)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 393, in model_iteration
    batch_outs = f(ins_batch)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/backend.py", line 3625, in __call__
    outputs = self._graph_fn(*converted_inputs)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1081, in __call__
    return self._call_impl(args, kwargs)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1121, in _call_impl
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError:  Blas GEMM launch failed : a.shape=(1, 2520), b.shape=(2520, 2520), m=1, n=2520, k=2520
     [[node dense/MatMul (defined at usr/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_keras_scratch_graph_378]

Function call stack:
keras_scratch_graph
2
  • did you find some kind of solution to this?
    – epa095
    Jan 23, 2020 at 11:47
  • I would think about raising this on the tensorflow page. Have you tried using tf.compat.v1.Session? The guys at tensorflow preserved some of the TF1.0 stuff... Worse case, is there a need for you to upgrade? If upgrading to 2.0 breaks your program. Keeping tensorflow to 1.0 might still be viable.
    – Jason Chia
    May 4, 2020 at 7:23

1 Answer 1

0

Try to allow fraction of GPU for each model.

import tensorflow as tf
config = tf.compat.v1.ConfigProto(gpu_options = 
                     tf.compat.v1.GPUOptions(per_process_gpu_memory_fraction=0.8)
# device_count = {'GPU': 1}
)
config.gpu_options.allow_growth = True
session = tf.compat.v1.Session(config=config)
tf.compat.v1.keras.backend.set_session(session)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.