7

In order to leverage the GPUs on a system, I'd like to be able to draw a block diagram and understand the connections represented by "nvidia-smi topo -m" output.

Here is an example output: enter image description here

enter image description here

Can someone provide a system level block diagram of this? Descriptions of the connections would be great too. I'm sure this would help many people exploit their multi gpu systems.

5
  • what is this node? does not look like DGX1.
    – den.run.ai
    Commented Jul 26, 2019 at 16:01
  • 1
    have you tried the lstopo command from the hwloc package ? Commented Jul 28, 2019 at 11:33
  • @GillesGouaillardet good idea, I have not even thought about using tools from MPI side.
    – den.run.ai
    Commented Jul 29, 2019 at 15:11
  • note that even if the hwloc project is hosted by the Open MPI project, this is a standalone package that is independent from the MPI library. note you might have to build hwloc by yourself and configure it with --with-cuda Commented Jul 29, 2019 at 23:32
  • @GillesGouaillardet how do I check if hwloc is compiled with CUDA support? So I cannot depend on the hwloc installed by the package manager?
    – den.run.ai
    Commented Jul 30, 2019 at 14:34

1 Answer 1

20
+100

You have two CPU sockets. Each CPU has a PCIe host bridge (PHB) leading to two PCIe switches. Each of the (two times two) PCIe switches has two GPUs connected to it, for a total of eight. enter image description here

3
  • Actually I liked a lot the pictures in this arxiv paper: arxiv.org/pdf/1903.04611.pdf
    – den.run.ai
    Commented Jul 29, 2019 at 15:21
  • Awesome! I think this will help a bunch of people. It would be amazing if nvidia built this into nvidia-smi, Commented Aug 3, 2019 at 0:07
  • 1
    Are there any good resources to help me understand why PHB may be slower than PIX? Is the CPU itself involved in the critical path of copying data? Are system RAM pages used as bounce buffers?
    – cade
    Commented Dec 28, 2022 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.