3

Though SLURM works fine for job submitting, running, and queueing, I got a minor error below.

sudo systemctl status slurmd

Jun 12 10:20:40 noki-System-Product-Name systemd[1]: slurmd.service: Can't open PID file /var/run/slurm-llnl/slurmd.pid (yet?) after start: No such file or directory

sudo systemctl status slurmctld

Jun 12 10:20:40 noki-System-Product-Name systemd[1]: slurmd.service: Can't open PID file /var/run/slurm-llnl/slurmd.pid (yet?) after start: No such file or directory

I followed the installation of a guide from

file:///home/noki/Downloads/Webinar_2_Slurm_II--Ubuntu16.04_and_18.04.pdf

This problem may come from the ownership of slurm.conf file?

Here are my slurm.conf and ownership for slur*.pid

# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ControlMachine=noki-System-Product-Name
#ControlAddr=
# 
#MailProg=/bin/mail 
MpiDefault=none
#MpiParams=ports=#-# 
ProctrackType=proctrack/pgid
ReturnToService=1
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
#SlurmctldPort=6817 
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
#SlurmdPort=6818 
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=noki
#SlurmdUser=root
StateSaveLocation=/var/spool/slurm-llnl
SwitchType=switch/none
TaskPlugin=task/none
# 
# 
# TIMERS 
#KillWait=30 
#MinJobAge=300 
#SlurmctldTimeout=120 
#SlurmdTimeout=300 
# 
# 
# SCHEDULING 
FastSchedule=1
SchedulerType=sched/backfill
SelectType=select/linear
#SelectTypeParameters=
# 
# 
# LOGGING AND ACCOUNTING 
AccountingStorageType=accounting_storage/none
ClusterName=linux
#JobAcctGatherFrequency=30 
JobAcctGatherType=jobacct_gather/none
#SlurmctldDebug=3 
SlurmctldLogFile=/var/log/slurm-llnl/SlurmctldLogFile
#SlurmdDebug=3 
SlurmdLogFile=/var/log/slurm-llnl/SlurmdLogFile
# 
# 
# COMPUTE NODES 
NodeName=noki-System-Product-Name CPUs=4 RealMemory=6963 Sockets=1 CoresPerSocket=4 ThreadsPerCore=1 State=UNKNOWN 
PartitionName=debug Nodes=noki-System-Product-Name Default=YES MaxTime=INFINITE State=UP
total 8
-rw-r--r-- 1 noki root 6 Jun 12 10:20 slurmctld.pid
-rw-r--r-- 1 root root 6 Jun 12 10:20 slurmd.pid

2 Answers 2

5

This can actually a bigger problem than it may appear at first. In your case, I see that you already implemented at least part of my recommendation below, but I will answer this question anyway for the benefit of others who, like me, find this question through Google.

The first step is to check if the PID file actually exists in the location configured in slurm.conf. If it does: verify that the service definition unit file for systemd also references the same PID file. If it does, and your service starts up normally, you can ignore the message - it is simply a timing issue; systemd may check for the PID file before it is created.

If the PID file does not exist, or the unit file points to a different location, systemd will not correctly recognize that the daemon has started, and may time out and report that the daemon has failed, even though it is actually running.

The cause is usually a permission problem. The location for the PID files in the example configurations is usually in /var/run. This directory is only root-writeable, but the daemon runs as user slurm.

To solve this, you need to create a subdirectory under /var/run (or preferably under /run, since /var/run is deprecated) with the correct ownership. At this point, you'll run into the next issue: /run is a tmpfs directory, so it gets deleted on each reboot. That means that you can't simply use mkdir, chown and chmod to create your directory.

The solution (hat tip to https://askubuntu.com/questions/303120/how-folders-created-in-var-run-on-each-reboot for the answer):

Create a file /etc/tmpfiles.d/slurm.conf with the following content:

d /run/slurm 0770 root slurm -

Finally, you need to tell slurm and systemd about the new location.

In slurm.conf, simply update the PID file entries.

To tell systemd about the new location, you need to modify the service definition to change where systemd will be looking for the PID file. You could directly modify the file /usr/lib/systemd/system/slurm*.service, but a better approach is to override the setting in /etc/systemd/system/slurm*.service.d

3
  • This is a great answer that gets to the root of the problem. @user8709546, please put a checkmark next to it. Dec 29, 2020 at 4:24
  • 1
    Thanks a lot. Apologies if I am ignorant! Can you please elaborate a bit more on your last comment: "You could directly modify the file /usr/lib/systemd/system/slurm*.service, but a better approach is to override the setting in /etc/systemd/system/slurm*.service.d" I assure you mean the "slurmd.service.d" slurmctld.service.d" files? I don't have such files after installation? Mar 5, 2021 at 8:42
  • 1
    @GertKruger No need to apologize; that's what Stackexchange is for. You are correct, of course, you have to substitute either d or ctld for the * depending on which daemon you want to change. These are directories, not files (the .d extension usually means that), and they are not created by default; you have to manually create them if you need them. Details are a bit off topic here; I would point you to the documentation for systemd. I believe the term to search for is drop-in files. Mar 7, 2021 at 7:05
3
  1. as I recall those files must be owned by slurm, not any other user
  2. If (1) wouldn't help: Are you running same Ubuntu as in example you are referencing? If not: double check if slurmd.pid locations in your slurm.conf file are the same as in slurmd.service. Those requirements are dependent on distro. Try to run:
# looking for slurmd.service file
find / -name "slurmd.service"
# assuming you found it somewhere. Check the content:
cat /full_path_to/slurmd.service 

Found the exact location where it looks for the PID file.

If needed: repeat the same for slurmctld.service file and SlurmctldPidFile value.

After checking and editing slurm.conf copy it to each node ans restart services.

1
  • 1
    Thank you so much. I'm doing another job. After testing this, I will comment. Maybe the next month. Sorry for this.
    – user8709546
    Jul 18, 2019 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.