34

It seems like this should be easy to integrate CMake+CTest in jenkins. The cmakebuilder plugin is extremely easy to configure (just set the source tree and the build tree, done!). However I fail to understand how to call the CTest steps.

According to the main xUnit page, since version 1.58 the XML output from CTest is supported, see bug report.

That's about all the documentation I could find. When I search on google or on stackoverflow, I can only find very old documentation requiring manual steps.

I would like to know how to setup a recent jenkins (1.532.1) with xUnit (1.81). Should I create a 'Add build-step' ? Should I create a 'post-build action' ? What do I need to fill in to get CTest to run and to produce proper XML files, so that jenkins can integrate them ?

3 Answers 3

54
+50

Here is a small example that demonstrates how to have xUnit pick up a CTest generated XML test result file. The example consists of a single C++ source file main.cpp

#include <cstdlib>
int main() {
    std::exit(-1);
}

and an accompanying CMakeLists.txt:

cmake_minimum_required(VERSION 2.8)
project(JenkinsExample)
enable_testing()
add_executable(main main.cpp)
add_test(test1 main)
add_test(test2 main)
set_tests_properties(test2 PROPERTIES WILL_FAIL ON)

The CMake list file generates an executable main and runs this executable from two CMake tests, where for demonstration purposes one will always fail and the other one will always succeed.

Using Jenkins, set up a new job and a add new cmakebuilder build step. Point the CMake source directory to the folder that contains the example project on your local machine. The CMake build directory should be set to build. This will make Jenkins create a CMake build folder in the job's workspace directory. It's a good idea to set the Clean Build flag to make the job always start with a clean state.

Then, assuming you are running Unix, add an Execute shell job step and enter the following shell script in the Command box:

cd build
/usr/local/bin/ctest --no-compress-output -T Test || /usr/bin/true

Running ctest with the option -T Test will make CTest generate an XML output file in a sub-folder Testing inside the build folder, which can be picked up by the xUnit plug-in in a post-build action then. The || /usr/bin/true is necessary to prevent Jenkins from aborting the build prematurely (without running the xUnit plug-in) if some tests fail.

If you are using Windows, set up a similar Execute Windows batch command job step instead:

cd build
"C:\Program Files (x86)\CMake 2.8\bin\ctest.exe" --no-compress-output -T Test || verify > NUL

Finally the xUnit plug-in must be configured in the following way:

Add a Publish xUnit test result report post-build action and then use the plug-in's Add button to create a CTest-Version test result report. In the CTest-Version (default) Pattern enter the file pattern build/Testing/**/Test.xml.

7
  • Two comments, 1. you are missing --no-compress-output from CTest execution. Without this flag you'll end up with issues.jenkins-ci.org/browse/JENKINS-21737 2. To have things simplified, I did register a bug for CTest to control the returned value: cmake.org/Bug/view.php?id=14749
    – malat
    Feb 11, 2014 at 13:50
  • @malat Thanks for the --no-compress-output hint. I have updated the answer.
    – sakra
    Feb 11, 2014 at 13:58
  • This answer is why I like Stackoverflow so much. Thanks a lot, great help!
    – MOnsDaR
    Mar 12, 2014 at 10:11
  • 2
    On Ubuntu 14.04, the shell command should be /usr/bin/ctest --no-compress-output -T Test || /bin/true due to different locations of the binaries
    – Sebastian
    Oct 16, 2014 at 10:46
  • 1
    Have you tried setting xUnit's "Fail the build if test results were not updated this run" flag?
    – sakra
    Mar 18, 2016 at 13:38
15

For those wanting to parse CTest output in a Jenkins Declarative Pipeline, you can now do this quite easily using the xUnit plugin, since it can parse CTest output directly.

Add a stage in your Jenkinsfile to run ctest in the build directory, and add a post stage to process the output with xUnit. Here's a skeleton example:

pipeline {
  agent any
  stages {
    stage('Configure') {
      steps {
        dir('build') {
          sh 'cmake ../'
        }
      }
    }
    stage('Build') {
      steps {
        dir('build') {
          sh 'cmake --build .'
        }
      }
    }  
    stage('Test') {
      steps {
        dir('build') {
          sh 'ctest -T test --no-compress-output'
        }
      }
    }
  }
  post {
    always {
      // Archive the CTest xml output
      archiveArtifacts (
        artifacts: 'build/Testing/**/*.xml',
        fingerprint: true
      )

      // Process the CTest xml output with the xUnit plugin
      xunit (
        testTimeMargin: '3000',
        thresholdMode: 1,
        thresholds: [
          skipped(failureThreshold: '0'),
          failed(failureThreshold: '0')
        ],
      tools: [CTest(
          pattern: 'build/Testing/**/*.xml',
          deleteOutputFiles: true,
          failIfNotNew: false,
          skipNoTestFiles: true,
          stopProcessingIfError: true
        )]
      )

      // Clear the source and build dirs before next run
      deleteDir()
    }
  }
}

For examples of how to configure the xUnit plugin in a Declarative Pipeline, the Jenkins Snippets Generator is a very helpful resource.

3

Since CMake 3.21.0 there is also a new option in CTest:

$ man ctest
[...]
  --output-junit <file>        = Output test results to JUnit XML file.

Reference:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.