2

If I have three gpus and I need to transfer a huge buffer to all three of them, will it make any difference if I use a CUDA stream for each one of them so that their copy engines can perform the transfers simultaneously? I mean: the PCI-E bus to reach all three of them is the same, isn't it?

6
  • This should at least answer one of your questions: on-demand.gputechconf.com/gtc-express/2011/presentations/…
    – sj755
    Jul 25, 2013 at 20:33
  • 2
    The last part of the question completely depends on the PCI-e topology of the host machine. There is no general answer.
    – talonmies
    Jul 25, 2013 at 20:34
  • 1
    @sj755 Perhaps I'm missing something. The presentation you have linked explains the instructions to be used to exploit streams and P2P memcpy. I think the main question of the user is: if all those GPUs are exchanging data asynchronously (and perhaps the user already knows how to do it), by which mechanisms will they compete for bandwidth across the PCI-E bus, possibly depending on the PCI-E topology of the host machine, as pointed out by talonmies?
    – Vitality
    Jul 25, 2013 at 20:58
  • sj's presentation kind of helps me because I see a single bus from CPU to the "IOH" chips, even if there are multiple GPUs under them.. I dare suppose that the same bus is shared from the host to reach multiple gpus if they belong to the same IOH chip.. and probably isn't if they belong to different IOH chips... is that correct?
    – Marco A.
    Jul 25, 2013 at 21:03
  • Actually the PCIE lanes are not shared amongst any devices. PCIE is a point-to-point "bus". It's theoretically possible for a PCIE root complex (the logical entity immediately upstream of all the PCIE lanes, within the IOH/CPU) to sustain multiple transfers simultaneously. But other system bottlenecks will likely present themselves. As talonmies said, you really need to know your specific topology and capabilities, there is no general answer. Jul 25, 2013 at 21:21

1 Answer 1

1

PCIe scaling one of the areas covered in this text book on CUDA, using a number of different processor architectures.

Yes, you need to use a separate stream for each transfer, as this moves you away from the default, serialized, stream zero. You will also hit various bandwidth limits, but yes, they will run concurrently and you do get a speed up over doing the transfer sequentially.

However, you will be limited by the ability of the processor/memory/pcie controller to provide concurrent (PCIe 2) 5GB/s streams. Where adding more cards does not reduce the number of PCIe lanes available, you usually see a significant benefit. Generally for 2 cards this works well, but rapidly drops away at more than 3 cards as bandwidth issues get in the way when adding more cards. Especially with more than 2 cards, you're unlikely to have the full 16 PCIe lanes available on many systems.

The Nsight tool is very good at displaying timelines showing what is going on with the transfers, as well as showing the actual transfer rates achieved, so I suggest you give this a try to let you see what is really happening.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.