39

I have just started with Jenkins

My freestyle project used to report JUnit tests results in Slack like this

MyJenkinsFreestyle - #79 Unstable after 4 min 59 sec (Open)
Test Status:
    Passed: 2482, Failed: 13, Skipped: 62

Now I have moved the same to pipeline project, and all is good except that Slack notifications do not have Test Status

done MyPipelineProject #68 UNSTABLE

I understand I have to construct the message to send to Slack, and I have done that above for now.

The only issue is how do I read the test status - the passed count, failed count etc. This is called "test summary" in Jenkins slack-plugin commit, and here is the screenshot testsummaryimage

So how do I access Junit tests count/details in Jenkins Pipeline project ? - so that these are reported in notifications.

UPDATE: In the Freestyle project, the Slack notification itself has the "test summary", and there is no option to opt (or not) for the test summary.

In Pipeline project, my "junit" command to "Publish JUnit test results" is before sending Slack notification.

So in code those lines look like this (this are last lines of the last stage):

bat runtests.bat
junit 'junitreport/xml/TEST*.xml'
slackSend channel: '#testschannel', color: 'normal', message: "done ${env.JOB_NAME} ${env.BUILD_NUMBER} (<${env.BUILD_URL}|Open>)";
1
  • How it worked in the old FreeStyle job: DId the Slack notification build plugin construct the test summary for you? If so, did you run "Publish JUnit test results" before sending Slack notification?
    – izzekil
    Oct 9, 2016 at 3:25

4 Answers 4

78

For anyone coming here in 2020, there appears to be a simpler way now. The call to 'junit testResults' returns a TestResultSummary object, which can be assigned to a variable and used later.

As an example to send the summary via slack:

def summary = junit testResults: '/somefolder/*-reports/TEST-*.xml'
slackSend (
   channel: "#mychannel",
   color: '#007D00',
   message: "\n *Test Summary* - ${summary.totalCount}, Failures: ${summary.failCount}, Skipped: ${summary.skipCount}, Passed: ${summary.passCount}"
)
10
  • This is exactly what I was looking for. The above solutions use a very old interface from Hudson's core. Mar 19, 2020 at 20:01
  • 1
    This was pretty useful! One thing I was hoping to avoid was actually archiving all result xml files; this step seems to do that by default. Not sure if that could be disabled as I am only interested in result counts (logs have rest of info and dev can run tests locally as needed).
    – StaxMan
    Mar 24, 2020 at 22:08
  • 1
    For people using JUnit and maven with the default configurations this value will be **/target/surefire-reports/TEST-*.xml
    – Yeikel
    Jul 1, 2020 at 5:29
  • 3
    Note that as @StaxMan mentioned, as this archives the tests, make sure that you do not have any step that archives them. I was doing so and what I observed is that Jenkins UI was reporting a duplicated count which was very confusing
    – Yeikel
    Jul 1, 2020 at 6:24
  • 1
    Scripted-pipeline version: def summary = junit(testResults: '/somefolder/*-reports/TEST-*.xml') Sep 25, 2020 at 11:22
44

From this presentation of Cloudbees I found that it should be possible via "build" object. It has code like

def testResult = build.testResultAction
def total = testResult.totalCount

But currentBuild does not provide access to testResultAction.

So kept searching and found this post "react on failed tests in pipeline script". There Robert Sandell has given "pro tip"

Pro tip, requires some "custom whitelisting":

AbstractTestResultAction testResultAction =  currentBuild.rawBuild.getAction(AbstractTestResultAction.class)
if (testResultAction != null) {
    echo "Tests: ${testResultAction.failCount} / ${testResultAction.failureDiffString} failures of ${testResultAction.totalCount}.\n\n" 
}

This worked like a charm - just that I had to deselect "Groovy sandbox" checkbox. Now I have these in the build log

Tests: 11  / ±0 failures of 2624

Now I will use this to prepare string to notify in slack with test results.


UPDATE:

Finally, the function I used to get output like the following (Note the "failure diff" after failed tests is very useful)

Test Status:
  Passed: 2628, Failed: 6  / ±0, Skipped: 0

Is the following:

import hudson.tasks.test.AbstractTestResultAction

@NonCPS
def testStatuses() {
    def testStatus = ""
    AbstractTestResultAction testResultAction = currentBuild.rawBuild.getAction(AbstractTestResultAction.class)
    if (testResultAction != null) {
        def total = testResultAction.totalCount
        def failed = testResultAction.failCount
        def skipped = testResultAction.skipCount
        def passed = total - failed - skipped
        testStatus = "Test Status:\n  Passed: ${passed}, Failed: ${failed} ${testResultAction.failureDiffString}, Skipped: ${skipped}"

        if (failed == 0) {
            currentBuild.result = 'SUCCESS'
        }
    }
    return testStatus
}

UPDATE 2018-04-19

Note the above require manual "whitelisting" of methods used. Here is how you can whitelist all the methods in one go

Manually update the whitelist...

Exit Jenkins

Create/Update %USERPROFILE%.jenkins\scriptApproval.xml with the following content

<?xml version='1.0' encoding='UTF-8'?>
<scriptApproval plugin="script-security@1.23">
<approvedScriptHashes>
</approvedScriptHashes>
<approvedSignatures>
<string>method hudson.model.Actionable getAction java.lang.Class</string>
<string>method hudson.model.Cause getShortDescription</string>
<string>method hudson.model.Run getCauses</string>
<string>method hudson.tasks.test.AbstractTestResultAction getFailCount</string>
<string>method hudson.tasks.test.AbstractTestResultAction getFailureDiffString</string>
<string>method hudson.tasks.test.AbstractTestResultAction getSkipCount</string>
<string>method hudson.tasks.test.AbstractTestResultAction getTotalCount</string>
<string>method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild</string>
</approvedSignatures>
<aclApprovedSignatures/>
<approvedClasspathEntries/>
<pendingScripts/>
<pendingSignatures/>
<pendingClasspathEntries/>
</scriptApproval>
  • Restart Jenkins
  • and then verify that the "In script approval" has the above entries approved
  • NOTE: Its the which is important. So if the scriptApproval file is already there, then you will generally need to ensure the contents of tag.
4
  • 1
    Custom whitelisting is very important here. It failed silently until I went to my jenkins.server.example.com:8080/scriptApproval/ and clicked to approve several calls. Note I had to do this a few times between job run attempts, as one approval unblocked the next.
    – MarkHu
    Apr 17, 2018 at 18:28
  • @MarkHu Have updated the answer to show how to update whitelist with all approvals in one step
    – vikramsjn
    Apr 19, 2018 at 14:55
  • 1
    better than just editing the xml, you can just go to [jenkins_url]/script (or run script from manage jenkins), then type this: import jenkins.model.Jenkins def scriptApproval = Jenkins.instance.getExtensionList('org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval')[0] scriptApproval.approveSignature('method hudson.model.Cause getShortDescription')
    – ymajoros
    Mar 7, 2019 at 13:30
  • I'm having this problem stackoverflow.com/questions/59202628/… and can't get it work. Can you please help?
    – Psdet
    Dec 5, 2019 at 21:41
12

To expand upon @vikramsjn's answer, here is what I use to get the test summary in my Jenkinsfile:

import hudson.tasks.test.AbstractTestResultAction
import hudson.model.Actionable

@NonCPS
def getTestSummary = { ->
    def testResultAction = currentBuild.rawBuild.getAction(AbstractTestResultAction.class)
    def summary = ""

    if (testResultAction != null) {
        def total = testResultAction.getTotalCount()
        def failed = testResultAction.getFailCount()
        def skipped = testResultAction.getSkipCount()

        summary = "Test results:\n\t"
        summary = summary + ("Passed: " + (total - failed - skipped))
        summary = summary + (", Failed: " + failed)
        summary = summary + (", Skipped: " + skipped)
    } else {
        summary = "No tests found"
    }
    return summary
}

I then use this method to instantiate my testSummary variable:

def testSummary = getTestSummary()

This will return something similar to:

"Test results:
     Passed: 123, Failed: 0, Skipped: 0"
6
  • Thanks. Your post reminded me to post my function too... which has one more bit there - the failureDiffString
    – vikramsjn
    Mar 1, 2017 at 6:11
  • How did you call that in your JenkinsFile where you need to send slack notification ? Can you please share some code?
    – Psdet
    Dec 5, 2019 at 20:54
  • @Prr I used the slackSend Jenkins plugin, which can be found in Manage Jenkins>Manage Plugins. After you configure the plugin in Manage Jenkins>Manage System>Slack, you can use slackSend in your pipeline like so: slackSend color: 'good', message: 'Message from Jenkins Pipeline' Dec 6, 2019 at 21:22
  • @ChristopherRung - thanks. I used that and I'm able to send messages from slackSend. I'm using declarative pipeline and having troubles with sending build results like number of tests passed, failed etc. The above code you mentioned cannot be written in the JenkinsFile it has to be done in shared library. I'm not able to use shared library. is there an easy way to use the above code in JenkinsFile?
    – Psdet
    Dec 9, 2019 at 1:01
  • @Prr - I was able to use getTestSummary as written above in my Jenkinsfile. Are you referring to security implications? The @NonCPS annotation is used to run the method outside of the script's sandbox (due to the use of currentBuild.rawBuild, a protected object).The build will fail on its first execution because this method hasn't been whitelisted, but there will be a link in the build output that will allow you to whitelist the method. It should then work on subsequent builds. Dec 10, 2019 at 14:57
3

First of all, thank you for above answers. They saved me a lot of time, I used proposed solution in my pipeline. I however didn't use "whitelisting" and it works fine. I use shared libraries for Jenkins pipeline and here is a piece of that shared library with pipeline and using of methods to get counts:

import hudson.model.*
import jenkins.model.*
import hudson.tasks.test.AbstractTestResultAction

def call(Closure body) {
    ...

    def emailTestReport = ""

    pipeline {
        ...

        stages{
            stage('Test'){
                ...
                post {
                    always {
                        junit 'tests.xml'

                        script {
                            AbstractTestResultAction testResultAction =  currentBuild.rawBuild.getAction(AbstractTestResultAction.class)
                            if (testResultAction != null) {
                                def totalNumberOfTests = testResultAction.totalCount
                                def failedNumberOfTests = testResultAction.failCount
                                def failedDiff = testResultAction.failureDiffString
                                def skippedNumberOfTests = testResultAction.skipCount
                                def passedNumberOfTests = totalNumberOfTests - failedNumberOfTests - skippedNumberOfTests
                                emailTestReport = "Tests Report:\n Passed: ${passedNumberOfTests}; Failed: ${failedNumberOfTests} ${failedDiff}; Skipped: ${skippedNumberOfTests}  out of ${totalNumberOfTests} "
                            }
                        }

                        mail to: 'example@email.com',
                        subject: "Tests are finished: ${currentBuild.fullDisplayName}",
                        body: "Tests are finished  ${env.BUILD_URL}\n  Test Report: ${emailTestReport} "
                    }

                }
            }
        }
    }
}

p.s. If I create emailTestRepot as a local variable inside script "section" I get next exception:

an exception which occurred:
    in field locals
    in field parent
    in field caller
    in field e
    in field program
    in field threads
    in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@11cd92de
Caused: java.io.NotSerializableException: hudson.tasks.junit.TestResultAction
...

I struggled a lot with trying to fix that java.io.NotSerializableException. As I understood I needed to use "whitelisting" to prevent NotSerializableException. But I really didn't want to do it and when I moved "def emailTestReport" out of pipeline it worked just fine.

2
  • Thanks ! It works :) It's possible you met security issues, just go to script approval page and allow calls.
    – André DS
    Nov 9, 2018 at 14:17
  • This works without the security approvals. I tried in the Jenkinsfile and ran into the whitelisting problem, which I am not authorized to change. This route avoided the need. Mar 17, 2021 at 20:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.