9

I'm currently working on a system with 2 GPUs each of 12GB. I want to implement model parallelism across the two GPUs to train large models. I have been looking through all over the internet, SO, tensorflow documentation, etc, i was able to find the explanations of model parallelism and its results but nowhere did i find a small tutorial or small code snippets on how to implement it using tensorflow. I mean we have to exchange activations after every layer right? So how do we do that? Is there a specific or cleaner ways of implementing model parallelism in tensorflow? It would be very helpful if you could suggest me a place where i can learn to implement it or a simple code like mnist training on multiple GPU using 'MODEL PARALLELISM'.

Note: I have done data parallelism like in CIFAR10 - multi gpu tutorial but i haven't found any implementation of model parallelism.

1 Answer 1

14

Here's an example. The model has some parts on GPU0, some parts on GPU1 and some parts on CPU, so this is 3 way model parallelism.

with tf.device("/gpu:0"):
    a = tf.Variable(tf.ones(()))
    a = tf.square(a)
with tf.device("/gpu:1"):
    b = tf.Variable(tf.ones(()))
    b = tf.square(b)
with tf.device("/cpu:0"):
    loss = a+b
opt = tf.train.GradientDescentOptimizer(learning_rate=0.1)
train_op = opt.minimize(loss)

sess = tf.Session()
sess.run(tf.global_variables_initializer())
for i in range(10):
    loss0, _ = sess.run([loss, train_op])
    print("loss", loss0)
12
  • @Buatov - Thanks for the above implementation, but what i was looking for is that when implementing model parallelism for a neural network we will have to share the activations after forward pass through every layer, so my main doubt was how do i pass the weight matrices from one device to the other (i.e, between GPU's)..
    – krish567
    Feb 9, 2017 at 7:09
  • That's all done automatically for you. IE, just replace tf.square part in the code above with a=create_left_part_of_network and b=create_right_part_of_network, and you'll end up with a network partitioned between gpu0 and gpu1 Feb 9, 2017 at 15:10
  • It is working as i expected it to work but it is slower than the time it takes if i run everything in one gpu. Do you know why is it happening. here is the link to the codes: multi_gpu one gpu
    – krish567
    Feb 10, 2017 at 12:42
  • you have to look at timelines and check what's the bottleneck Feb 10, 2017 at 17:18
  • Thank you very much for the help, it is faster but the problem i was facing was the multi-gpu program is taking double the memory footprint than running the entire program in one GPU. Can you think of any reason why this might be happening? The links to the codes are above. I will update them with the latest codes
    – krish567
    Feb 11, 2017 at 4:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.