6

I want to allow running following pipeline in parallel, but I have to limit maximum number of parallel runs due to limited resources.

pipeline {

  agent { label "$JENKINS_AGENT" }
  parameters { .. }
  options { .. }

  stages {
    stage('Checkout') { .. }
    stage('Config') { .. }
    stage('Deploy') { .. }
    stage('Test') { .. }
  }

  post {
    failure { .. }
    success { .. }
    always {
      cleanup()
    }
  }
}

For example if I am limited to 3 deployments, how could I achieve this?

Edit: I have added currently the following to my jenkinsfile. Seems to work, but I'm not sure if that's the correct way to do this. (using Throttle Concurrent Builds Plugin)

properties([
  [
    $class: 'ThrottleJobProperty',
    categories: ['parallel_pipeline'],
    limitOneJobWithMatchingParams: false,
    maxConcurrentPerNode: 3,
    maxConcurrentTotal: 3,
    paramsToUseForLimit: '',
    throttleEnabled: true,
    throttleOption: 'project'
  ],
])

2 Answers 2

4

Strictly speaking, this is not possible at the moment. There is an open ticket in the Jenkins JIRA with a request for this feature.

You may be able to get around this issue with some hacks. There are a couple workarounds in the linked JIRA issue that involve creating pseudo-semaphores in order to lock individual threads until the pseudo-semaphore is free.

Another way that I've been able to handle this in specific use cases is to ensure that each branch of the parallel step is wrapped in a stage block. Since a stage locks a single executor at a time, this means you will run as many branches as you have executor slots available, and then other branches will wait until an executor slot is free to resume execution. This doesn't always work, however - for instance, you may have reasons for not making each branch an independent stage.

0

You may find useful these plugins:

  1. Throttle Concurrent Builds Plugin
  2. Lockable Resources Plugin
2
  • I forgot to add that the configuration snippet that I added was for Throttle Concurrent Builds Plugin.
    – warbaque
    Commented Jun 6, 2018 at 8:06
  • 4
    Downvoting because neither of these plugins can limit the number of concurrently executing branches of a Pipeline parallel step - these are for limiting concurrency at the build level, not at the Pipeline step level. They may happen to work coincidentally for OP because OP is running Pipeline stages within parallel branches, which in some ways behave like independent builds. These plugins don't work in the generic Pipeline parallel case.
    – jayhendren
    Commented Nov 1, 2018 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.