I have a build job on jenkins that is building my project and after it is done, it opens an ssh shell script on a remote server and transfers files and then stop and starts a daemon.

When I stop and start the daemon from the command line on a RHEL server, it executes just fine. When the job executes in jenkins, there are no errors.

The daemon stops fine and it starts fine. But shortly after starting, the daemon dies suddenly.

sudo service daemonName stop
# transfer files.
sudo service daemonName start

I'm sure that the problem isn't pathing

Does anyone know what could be special about the way Jenkins is executing the ssh shell script that would cause the daemon start to not fully complete?

share|improve this question
    
Have you checked the logs? What errors do you read in there? – fedorqui May 21 '13 at 10:36
    
the jenkins logs? or the console logs? or the logs of the daemon? the console logs show that everything was successful. The logs of the daemon don't show any problems The jenkins log I doubt will be helpful. – Eric May 21 '13 at 10:41
    
Maybe the daemon is not properly daemonized, have you tried adding a sleep 10000 to the shell script in Jenkins to see whether it runs longer when the invoking shell isn't closed? Have you tried comparing env output in Jenkins and the shell? If it's a homegrown daemon, it might be affected by things like locale. – Daniel Beck May 21 '13 at 18:43

Jenkins watches for processes spawned by the job and kill them to avoid zombie processes. See https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller

The workaround is to override the BUILD_ID environment variable:

BUILD_ID=dontKillMe
share|improve this answer
2  
Okay, I've tried this in many different incarnations but it doesn't seem to work. I've tried adding the flag to turn off the process tree killer, I've tried changing the build id in the ssh script i'm executing BUILD_ID=dontKillMe sudo service servicename start is in a "execute shell script on remote host over ssh" Am I doing something wrong? – Eric May 22 '13 at 17:29
    
ok, looks like I misread you question. The process three killer only runs on the jenkins node (or the connected slaves) but not on the remote machine, so the source of you problem must be elsewhere. – rcomblen May 23 '13 at 5:14
    
I'm glad you did, your answer just resolved a day long headache of mine. Thanks =) – Kyle Buser Nov 30 '16 at 22:12
up vote 6 down vote accepted

The problem: When executing a build through jenkins, the command to start the daemon process was clearly successfully executing, yet after the build job was done, the daemon would suddenly quit.

The solution: I thought for this whole time that it was jenkins killing the daemon. So I tried many different incarnations and permutations of disabling the ProcessTree module that goes through and cleans up zombie child processes. I tried fooling it by resetting the BUILD_ID environment variable. Nothing worked.

Thanks to this thread I found out that that solution only works for child processes executed on the BUILD machine. I.E. not applicable to my problem.

More searching led me here: Run a persistent process via ssh

The solution? Nohup.

So now the build successfully restarts the daemon by executing the following: sudo nohup service daemonname start

share|improve this answer
    
nohup saved me too... but strangely it didn't work when I appended '&' at the end of the line. I had to omit '&' for the remote script invocation to work. – noumenon Jan 9 '14 at 7:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.