2

I have a setup/installation of SLURM on my desktop computer to do some testing and understand how it works before deploying it to a cluster. The desktop computer is running Ubuntu 18.10 (Cosmic), as the nodes in the cluster are all running on. The used version of SLURM is 17.11.9. I have tested some of the features of SLURM, e.g. job-arrays and its deployment of tasks. However I would like to communicate with the different tasks sent out to each node or CPU in the cluster, in order to collect its results (without disk I/O). For that reason, I have looked in how manage that with e.g. message queuing, and MPI, or OpenMPI. (Any other implementation strategy, as an advice or recommendation is much appreciated.)

I have tested MPI with a simple Python snippet, starting a communication between two processes. I am using MPI4PY to handle this communication. This code snippet runs fine with mpiexec-command, but running it via SLURM and sbatch-command I cannot get it to work. SLURM is configured with OpenMPI and opmi_info states that SLURM is supported.

OpenMPI version 3.1.2-6 (from dpkg -l | grep mpi) SLURM_VERSION 17.11.9 Ubuntu 18.10 (Cosmic) MPI4PY version 3.0.1. (from pip list)

This is the Python3.6 code snippet:

    $cat mpi_test.py
    from mpi4py import MPI

    if __name__=='__main__':

      comm = MPI.COMM_WORLD
      rank = comm.Get_rank()

      if rank==0:
          data={'param1':1, 'param2':2, 'param3':3}        
          destinationNode = 1
          print('Im', rank, 'sending to ', destinationNode)
          comm.send(data, dest=destinationNode, tag=11)
      elif rank!=0:
          sourceNode = 0
          dataRx=comm.recv(source=sourceNode, tag=11)
          print('Im', rank, 'recieving from ', sourceNode)
          for keys in dataRx.keys():
              print('Data recieved: ',str(dataRx[keys]))

The python.mpi.sbatch used at the call with sbatch is:

    $cat python.mpi.sbatch
    #!/bin/bash -l
    #SBATCH --job-name=mpiSimpleExample
    #SBATCH --nodes=1
    #SBATCH --error=slurm-err-%j.err
    #SBATCH --export=all
    #SBATCH --time=0-00:05:00
    #SBATCH --partition=debug

    srun -N 1 mpiexec -n 2 python3 mpi_test.py
    #mpiexec -n 2 python3 mpi_test.py

    exit 0

Running "sbatch python.mpi.sbatch" with this setup yields the following output:

    $sbatch python.mpi.sbatch
    $cat slurm-err-104.err 
    ----------------------------------------------------------------------
    There are not enough slots available in the system to satisfy the 2 
    slots
    that were requested by the application:
    python3

    Either request fewer slots for your application, or make more slots
    available for use.
    --------------------------------------------------------------------

Modifying python.mpi.sbatch to instead use:

"srun -n 1 mpiexec -n 1 python3 mpi_test.py" yields the error:

    $cat slurm-err-105.error
    Traceback (most recent call last):
      File "mpi_test.py", line 18, in <module>
        comm.send(data, dest=destinationNode, tag=11)
      File "mpi4py/MPI/Comm.pyx", line 1156, in mpi4py.MPI.Comm.send
      File "mpi4py/MPI/msgpickle.pxi", line 174, in mpi4py.MPI.PyMPI_send
        mpi4py.MPI.Exception: MPI_ERR_RANK: invalid rank
    ---------------------------------------------------------------------
    mpiexec detected that one or more processes exited with non-zero 
    status, thus causing the job to be terminated. The first process to do 
    so was:

    Process name: [[44366,1],0]
    Exit code:    1
    ---------------------------------------------------------------------

Which is expected since it is only started with 1 node.

Running mpirun hostname, yields four instances of the machine, thus there should be four slots available for this machine. I may run the Python3.6 with up to four (after modification of mpi_test.py) processess with the command "mpiexec -n 4 python3 mpi_test.py", with success.

Any help is much appreciated.

slurm.conf-file:

# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ControlMachine=desktop-comp
#ControlAddr=
#BackupController=
#BackupAddr=
#
AuthType=auth/munge
#CheckpointType=checkpoint/none
CryptoType=crypto/munge
#DisableRootJobs=NO
#EnforcePartLimits=NO
#Epilog=
#EpilogSlurmctld=
#FirstJobId=1
#MaxJobId=999999
#GresTypes=
#GroupUpdateForce=0
#GroupUpdateTime=600
#JobCheckpointDir=/var/slurm/checkpoint
#JobCredentialPrivateKey=
#JobCredentialPublicCertificate=
#JobFileAppend=0
#JobRequeue=1
#JobSubmitPlugins=1
#KillOnBadExit=0
#LaunchType=launch/slurm
#Licenses=foo*4,bar
#MailProg=/bin/mail
#MaxJobCount=5000
#MaxStepCount=40000
#MaxTasksPerNode=128
MpiDefault=openmpi
#MpiParams=ports=#-#
#PluginDir=
#PlugStackConfig=
#PrivateData=jobs
#ProctrackType=proctrack/cgroup
#Prolog=
#PrologFlags=
#PrologSlurmctld=
#PropagatePrioProcess=0
#PropagateResourceLimits=
#PropagateResourceLimitsExcept=
#RebootProgram=
ReturnToService=1
#SallocDefaultCommand=
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
#SlurmdUser=root
#SrunEpilog=
#SrunProlog=
StateSaveLocation=/var/lib/slurm-llnl/slurmd
SwitchType=switch/none
#TaskEpilog=
#TaskPlugin=task/affinity
#TaskPluginParam=Sched
#TaskProlog=
#TopologyPlugin=topology/tree
#TmpFS=/tmp
#TrackWCKey=no
#TreeWidth=
#UnkillableStepProgram=
#UsePAM=0
#
#
# TIMERS
#BatchStartTimeout=10
#CompleteWait=0
#EpilogMsgTime=2000
#GetEnvTimeout=2
#HealthCheckInterval=0
#HealthCheckProgram=
InactiveLimit=0
KillWait=30
#MessageTimeout=10
#ResvOverRun=0
MinJobAge=300
#OverTimeLimit=0
SlurmctldTimeout=120
SlurmdTimeout=300
#UnkillableStepTimeout=60
#VSizeFactor=0
Waittime=0
#
#
# SCHEDULING
#DefMemPerCPU=0
FastSchedule=1
#MaxMemPerCPU=0
#SchedulerTimeSlice=30
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core
#
#
# JOB PRIORITY
#PriorityFlags=
#PriorityType=priority/basic
#PriorityDecayHalfLife=
#PriorityCalcPeriod=
#PriorityFavorSmall=
#PriorityMaxAge=
#PriorityUsageResetPeriod=
#PriorityWeightAge=
#PriorityWeightFairshare=
#PriorityWeightJobSize=
#PriorityWeightPartition=
#PriorityWeightQOS=
#
#
# LOGGING AND ACCOUNTING
#AccountingStorageEnforce=0
#AccountingStorageHost=
#AccountingStorageLoc=
#AccountingStoragePass=
#AccountingStoragePort=
AccountingStorageType=accounting_storage/none
#AccountingStorageUser=
AccountingStoreJobComment=YES
ClusterName=cluster
#DebugFlags=
#JobCompHost=
#JobCompLoc=
#JobCompPass=
#JobCompPort=
JobCompType=jobcomp/none
#JobCompUser=
#JobContainerType=job_container/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=3
#SlurmctldLogFile=
SlurmdDebug=3
#SlurmdLogFile=
#SlurmSchedLogFile=
#SlurmSchedLogLevel=
#
#
# POWER SAVE SUPPORT FOR IDLE NODES (optional)
#SuspendProgram=
#ResumeProgram=
#SuspendTimeout=
#ResumeTimeout=
#ResumeRate=
#SuspendExcNodes=
#SuspendExcParts=
#SuspendRate=
#SuspendTime=
#
#
# COMPUTE NODES
NodeName=desktop-comp CPUs=1 State=UNKNOWN
PartitionName=debug Nodes=desktop-compDefault=YES MaxTime=INFINITE State=UP

5
  • 1
    The sbatch manpage says that the default number of tasks (--ntasks) is one per node. You are asking for one node and do not specify --ntasks. Try adding #SBATCH --ntasks=2.
    – rtoijala
    Commented Jun 10, 2019 at 8:34
  • Thank you for your inerest rtoljala, however the proposed change did not work. Instead the work is put in queue and awaits a second node to be available (which I do not have).
    – Pälle
    Commented Jun 11, 2019 at 8:10
  • You should configure slurm with cores as resources (CR_CORE_MEMORY or such). Then you can run multiple tasks on one node. Also, if the mpi library has been setup with slurm integration, you should not use mpiexec, just srun.
    – janneb
    Commented Jun 11, 2019 at 11:44
  • Could you show the slurm.conf file? Commented Jun 11, 2019 at 13:44
  • Yes, please see the updated field above.
    – Pälle
    Commented Jun 11, 2019 at 14:25

1 Answer 1

2

In your update question you have in your slurm.conf the line

NodeName=desktop-comp CPUs=1 State=UNKNOWN

This tells slurm that you have only one CPU available on your node. You can try running slurmd -C to see what slurm discovers about your computer and copypaste the CPUs, CoresPerSocket etc. values to your slurm.conf.

3
  • Thank you so much janneb, that I had missed out on. Now trying to run my mpi-task I get that the work is put in queue and marked as pending with the reason, stated at the NODELIST-colum of "squeue", as "Resources". Collecting data from "sinfo" states that my partition, or node is in the state "drain". But I am only asking for two resources, which I should have. the output of "slurmd -C" yielded: CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=15945.
    – Pälle
    Commented Jun 12, 2019 at 6:58
  • If your node is drained it won't run anything. Bring your node up.
    – Poshi
    Commented Jun 12, 2019 at 7:54
  • Thank you Poshi. Me being a complete newbie on slurm, I can not get the node to be change its state. Not even after a cold reboot. How can I manage to reset its state using slurm-commands?
    – Pälle
    Commented Jun 12, 2019 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.