-2

My Jenkins build job compiles the code, then creates two Docker images (each a separate piece of the application) and publishes them.

Then it comes time to deploying them to on-premise Linux servers. I have a docker-compose.yaml file that supposed to get everything up and running.

My problem is, I am looking for a way to do three things:

  1. I need to place the compose file in the deployment server(s) which are not under Jenkins control, allowing perhaps some daemon to process it, as Docker images are already uploaded to Docker repo. I have root access to the deployment servers, ssh credentials are in Jenkins, and I am able to install things on the deployment servers. So, I tried to do that with scp'ing using a special service user, but it does not have permissions to create a directory (eg: /app) - I tried working around by adding that user to the root group (usermod -aG root username) - still same error.
  2. Once the compose file is in, I assume I will be able to deploy (as that same user is in docker group too) with daemon flag, but what about scenario where the app is already running, and I want to just deploy an update (essentially pull newer version of the images), and how to make sure docker will always keep the deployed stack always running, even if something crashes?
  3. If the running container(s) crashes persistently, how to avoid infinite loop, and get notified when say, it tried to restart it 5 times already?
1
  • To the moderators, the response was edited and I think it is worthwhile to give a thought on reopening it. For me at least it is far from being too broad as it is stated now. It describes an exact problem big companies moving to DevOps currently have. At least you can put some comment on how to improve it. Thanks anyway Apr 12, 2019 at 20:36

1 Answer 1

0

I would copy an script in order to stop and start the containers when a new image is available:

  1. Scp should work as it is described in Use ssh credentials in jenkins pipeline with ssh, scp or sftp.
  2. You should copy an script stopping (docker-compose down) and starting the containers (docker-compose up -d) together with the docker-compose file. Then the daemon must execute the script. EDIT: it seems one can use docker stack deploy -c /path/to/docker-compose.yml stack_name --with-registry-auth without needing to stop/start each time!
  3. You can use curl to check that the services are running (as a healthcheck). Retry a number of times returning error if one of the services is not available as It is explained here:

      #!/bin/bash
      url='http://website-to-test'
      attempts=5
      timeout=5
      online=false
    
      echo "Checking status of $url."
    
      for (( i=1; i<=$attempts; i++ ))
      do
          code=`curl -sL --connect-timeout 20 --max-time 30 -w "%{http_code}\\n" "$url" -o /dev/null`
    
          echo "Found code $code for $url."
    
          if [ "$code" = "200" ]; then
              echo "Website $url is online."
              online=true
              break
         else
              echo "Website $url seems to be offline. Waiting $timeout seconds."
              sleep $timeout
         fi
     done
    
     if $online; then
       echo "Monitor finished, website is online."
       exit 0
     else
       echo "Monitor failed, website seems to be down."
       exit 1
     fi
    

UPDATE: Modify the response taking into account your restrictions

13
  • Thanks, on #1 - I already have the compose script in repository. I only need to copy the modified version with the published docker image versions specified on it to the server. On #3 - I actually referred to the running container failing, not the build process. #2 I don't think will work, because I don't commit from git... I mean in the compose I have placeholder like: VERSION, and then replacing it with the built image number using sed on jenkins, then uploading to the servers.. How else would it work? Apr 10, 2019 at 22:12
  • Do you have the repo in GitHub so I can have a look? #1 Check the examples. You can configure your deployment servers as Jenkins slaves. Then you do not need to neither copy nor ssh the script. Jenkins can download your repo into the node and execute the docker-compose #3 yes jenkins detects if the running container fails and stops the pipeline informing about the error. Apr 10, 2019 at 22:20
  • Oh, no sorry it's internal on premise inside the company.. I am not sure I can set the servers as jenkins slaves, we are not admins, there is a whole other team managing the jenkins servers and its plugins.. Look at it simple: I got a final product in jenkins (the docker images which are published also on premise repository) - and I need the best way to deploy/update it on other non-jenkins servers. Apr 10, 2019 at 22:24
  • Ok I get the point. So basically you need to place the compose in the deployment server and some daemon will process it without the control of Jenkins because the images were already uploaded to the docker repo right? Do you have root permissions in the deployment server? And your credentials in Jenkins? Are you able to install things in the deployment server? Any other constraint? Please update the question with those restrictions and I will think about a possible workaround Apr 11, 2019 at 5:46
  • Yes, to all of these. I will update the question in an hour. Apr 11, 2019 at 12:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.