0

I have a Python script called manager.py, looking like this:

#!/usr/bin/python3
from mpi4py import MPI
import os
import sys

print("Running C++ workers")
cpp_mpi_info = MPI.Info.Create()
cpp_mpi_info.Set("path", os.getcwd())
cpp_comm = MPI.COMM_WORLD.Spawn("./worker", maxprocs=4, info=cpp_mpi_info)
print("Done")

And I have a C++ programm worker.cpp, which looks like this:

#include <stdio.h>
#include <iostream>
#include <mpi.h>

int main(int argc, char **argv)
{
    MPI_Comm comm;
    int rank;
    int size;
    MPI_Init(&argc, &argv);
    MPI_Comm_get_parent(&comm);
    if (comm == MPI_COMM_NULL)
    {
        std::cout << "Running worker without manager" << std::endl;
    }
    else
    {
        MPI_Comm_rank(comm, &rank);
        MPI_Comm_size(comm, &size);
        std::cout << "C++ worker " << rank << " of " << size << std::endl;
    }

    MPI_Finalize();
    return 0;
}

But when I run manager.py (either with ./manager.py or with mpiexec -n 1 ./manager.py), I get the following output:

Running C++ workers
[melchior:228633] [[22763,2],1] selected pml ucx, but peer [[22763,1],0] on melchior selected pml ob1
[melchior:228633] [[22763,2],1] ORTE_ERROR_LOG: Unreachable in file dpm/dpm.c at line 493
[melchior:228635] [[22763,2],3] selected pml ucx, but peer [[22763,1],0] on melchior selected pml ob1
[melchior:228635] [[22763,2],3] ORTE_ERROR_LOG: Unreachable in file dpm/dpm.c at line 493
[melchior:228634] [[22763,2],2] selected pml ucx, but peer [[22763,1],0] on melchior selected pml ob1
[melchior:228634] [[22763,2],2] ORTE_ERROR_LOG: Unreachable in file dpm/dpm.c at line 493
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_dpm_dyn_init() failed
  --> Returned "Unreachable" (-12) instead of "Success" (0)
--------------------------------------------------------------------------
[melchior:228635] *** An error occurred in MPI_Init
[melchior:228635] *** reported by process [1491795970,3]
[melchior:228635] *** on a NULL communicator
[melchior:228635] *** Unknown error
[melchior:228635] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[melchior:228635] ***    and potentially your MPI job)
[melchior:228625] 2 more processes have sent help message help-mpi-runtime.txt / mpi_init:startup:internal-failure
[melchior:228625] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[melchior:228625] 2 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal unknown handle
--------------------------------------------------------------------------
A request has timed out and will therefore fail:

  Operation:  LOOKUP: orted/pmix/pmix_server_pub.c:345

Your job may terminate as a result of this problem. You may want to
adjust the MCA parameter pmix_server_max_wait and try again. If this
occurred during a connect/accept operation, you can adjust that time
using the pmix_base_exchange_timeout parameter.
--------------------------------------------------------------------------

After that, execution hangs an undetermined amount of time and I have to kill it manually with a SIGKILL.

However, when I write both manager and worker in Python or C++ respectively, it works just fine.

So, what is the problem here and how do I fix it?

6
  • as a workaround, try mpiexec --mca pml ^ucx -n 1 ./manager.py Jan 20, 2022 at 23:52
  • @GillesGouaillardet Thanks, that works. Not quit end user friendly, but I can deal with that
    – mlinke-ai
    Jan 21, 2022 at 8:09
  • @GillesGouaillardet Can you further explain what the --mca flag is and what the values pml and ^ucx do?
    – mlinke-ai
    Jan 21, 2022 at 9:08
  • Open MPI uses different mechanism to enable inter-process communications. The trace points to an error in the pml component called ucx, so --mca pml ^ucx is used to discard this component. Other (friendlier?) options are export OMPI_MCA_pml=^ucx or add pml = ^ucx in your /.../etc/openmpi-mca-params.conf` file Jan 21, 2022 at 9:36
  • Okay, thank you. I also read about the OMPI_MCA_pml=^ucx option, but that resulted in an error about an invalid communicator. Furthermore I tested the code on another system and it worked there like a charm, didn't even need the mpiexec, just called the Python script. Maybe it is because I use Fedora and the other system uses Mint.
    – mlinke-ai
    Jan 21, 2022 at 9:49

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.