40

I have deployed Jenkins in my CentOS machine, Jenkins was working well for 3 days, but yesterday there was a Disk space is too low. Only 1.019GB left. problem.

How can I solve this problem, it make my master offline for hours?

  • 1
    How much available disk space was there when you first deployed Jenkins? Perhaps you should host Jenkins on a machine with more available disk space. – Bernard Apr 26 '12 at 1:38
27

You can easily change the threshold from jenkins UI (my version is 1.651.3):

[jenkins nodes preventing monitoring page]


Update: How to ensure high disk space

This feature is meant to prevent working on slaves with low free disk space. Lowering the threshold would not solve the fact that some jobs do not properly cleanup after they finish.

Depending on what you're building:

  1. Make sure you understand what is the disk output of your build - if possible - restrict the output to happen only to the job workspace. Use workspace cleanup plugin to cleanup the workspace as post build step.
  2. If the process must write some data to external folders - clean them up manually on post build steps.

Alternative1 - provision a new slave per job (use spot slaves - there are many plugins that integrate with different cloud provider to provision on the fly machines on demand)

Alternative2 - run the build inside a container. Everything will be discarded once the build is finished

|improve this answer|||||
  • 1
    @HarryMoreno Just checked the UI on the current LTS 2.107.2 - looks the same! – orshachar Apr 15 '18 at 10:17
  • 1
    Ah I was looking at [node name] > configure. You described nodes > configure. nvm imgur.com/a/CeTcA – Harry Moreno Apr 16 '18 at 15:10
  • 1
    @TryingToLearnJS I actually agree (and I wrote this answer) - It depends what is the question - if someones wants to keep working on slave with < 1GB - it answers how to configure it on Jenkins. If someone needs some automatic cleanup methodology it doesn't. But the latter depends very much on what you are running on Jenkins. – orshachar Nov 29 '18 at 14:21
  • 1
    but thanks on your feedback - I added a generic section that might answer both concerns – orshachar Nov 29 '18 at 14:37
  • 1
    @orshachar thanks man! Changing the threshold did help me. Since the nodes go offline when dropping below so changing that allowed me to bring them back up and fix some stuff. And thanks for the updated answer :) – pourmesomecode Nov 29 '18 at 15:00
16

Beside above solutions, there is a more "COMMON" way - directly delete the largest space consumer from Linux machine. You can follow the below steps:

  1. Login to Jenkins machine (Putty)
  2. cd to the Jenkins installation path

Using ls -lart to list out hidden folder also, normally jenkin installation is placed in .jenkins/ folder

[xxxxx ~]$ ls -lart

drwxrwxr-x 12 xxxx 4096 Feb 8 02:08 .jenkins/

  1. list out the folders spaces

Use df -h to show Disk space in high level

du -sh ./*/ to list out total memory for each subfolder in current path.

du -a /etc/ | sort -n -r | head -n 10 will list top 10 directories eating disk space in /etc/

  1. Delete old build or other large size folder

enter image description here

Normally ./job/ folder or ./workspace/ folder can be the largest folder. Please go inside and delete base on you need (DO NOT delete entire folder).

rm -rf theFolderToDelete

|improve this answer|||||
7

You can limit the reduce of disc space by discarding the old builds. There's a checkbox for this in the project configuration.

|improve this answer|||||
  • Thank you, this was a very helpful tip. – SimplGy Aug 3 '12 at 14:06
  • 1
    The builds information is saved on the master server. This kind of issue is mostly happen at the salve side, where each job has a workspace contains the source code and build results. – Jirong Hu May 20 '16 at 20:34
3

This is actually a legitimate question so I don't understand the downvotes, perhaps it belongs on Superuser or Serverfault. This is a soft warning threshold not hard limit where the disk is out of space.

Jenkins is the same. The conclusion is for many small projects the system property called hudson.diagnosis.HudsonHomeDiskUsageChecker.freeSpaceThreshold could be decreased.

In saying that I haven't tested it and there is a disclaimer

No compatibility guarantee

In general, these switches are often experimental in nature, and subject to change without notice. If you find some of those useful, please file a ticket to promote it to the official feature.

|improve this answer|||||
  • So set thresholds is to prevent node going down due to out of space? This doesn't help. We are constantly having out of space issue on the slave server due to large number of projects. What's the best solution to fix this issue? – Jirong Hu May 20 '16 at 20:42
  • Add another slave, install more disk space, or more likely you need to get jobs to clean up their workspace when complete. You might find there are unnecessary and unexpectedly large files lying around superuser.com/q/9847/81584 – KCD May 21 '16 at 4:06
3

I got the same issue. My jenkins version is 2.3 and its UI is slightly different. Putting it here so that it may helps someone. Increasing both disk space thresholds to 5GB fixed the issue.

enter image description here

|improve this answer|||||
0

To check the free space as Jenkins Job:

Parameters

  • FREE_SPACE: Needed free space in GB.

Job

#!/usr/bin/env bash

free_space="$(df -Ph . | awk 'NR==2 {print $4}')"

if [[ "${free_space}" = *G* ]]; then
  free_space_gb=${x/[^0-9]*/}

  if [[ ${free_space_gb} -lt ${FREE_SPACE} ]]; then
    echo "Warning! Low space: ${free_space}"
    exit 2
  fi
else
  echo "Warning! Unknown: ${free_space}"
  exit 1
fi


echo "Free space: ${free_space}"

Plugins

Set build description

Post-Build Actions

  • Regular expression: Free space: (.*)
  • Description: Free space: \1

  • Regular expression for failed builds: Warning! (.*)

  • Description for failed builds: \1
|improve this answer|||||
0

I have a cleanup job with the following build steps. You can schedule it @daily or @weekly.

  1. Execute system groovy script build step to clean up old jobs:
    import jenkins.model.Jenkins
    import hudson.model.Job

    BUILDS_TO_KEEP = 5

    for (job in Jenkins.instance.items) {
      println job.name

      def recent = job.builds.limit(BUILDS_TO_KEEP)

      for (build in job.builds) {
        if (!recent.contains(build)) {
          println "Preparing to delete: " + build
          build.delete()
        }
      }
    }

You'd need to have Groovy plugin installed.

  1. Execute shell build step to clean cache directories
rm -r ~/.gradle/
rm -r ~/.m2/

echo "Disk space"
du -h -s /
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.