60

I'm using Jenkins, Maven 3.1, and Java 1.6. I have the following Maven job set up in Jenkins with the following goals and options ...

clean install -U -P cloudbees -P qa

below is my pom.xml surefire configuration ...

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.17</version>
        <configuration>
                <reuseForks>true</reuseForks>
                <argLine>-Xmx2048m -XX:MaxPermSize=512M </argLine>
                <skipTests>false</skipTests>
        </configuration>
</plugin>

However, when my unit tests fail, the Jenkins console output still says "BUILD SUCCESS" and the build is marked as "unstable" instead of outright failing. How do I configure things in Jenkins (or Maven if taht's what it turns out to be) so that my build fails (not becomes unstable or passes) if any of the unit tests fail?

Below is what the console output says

17:08:04   MyProjectOrganizationControllerTest.testRecoverFromError » IllegalState Failed to...
17:08:04   MyProjectOrganizationControllerTest.testVerifyDistrictListPopulated » IllegalState
17:08:04   MyProjectOrganizationControllerTest.testUpdateSchool » IllegalState Failed to loa...
17:08:04   MyProjectOrganizationControllerTest.testDeleteSchool » IllegalState Failed to loa...
17:08:04   MyProjectOrganizationControllerTest.testVerifyOrgListPopulatedPrivateSchoolOrgType » IllegalState
17:08:04   MyProjectOrganizationControllerTest.testSubmitMultipleParams » IllegalState Faile...
17:08:04 
17:08:04 Tests run: 155, Failures: 0, Errors: 154, Skipped: 1
17:08:04 
17:08:04 [ERROR] There are test failures.
17:08:04 
17:08:04 Please refer to /scratch/jenkins/workspace/MyProject/MyProject/target/surefire-reports for the individual test results.
17:08:04 [JENKINS] Recording test results
17:08:07 log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.converters.BooleanConverter).
17:08:07 log4j:WARN Please initialize the log4j system properly.
17:08:14 [INFO] 
17:08:14 [INFO] --- maven-war-plugin:2.4:war (default-war) @ MyProject ---
17:08:15 [INFO] Packaging webapp
17:08:15 [INFO] Assembling webapp [MyProject] in [/scratch/jenkins/workspace/MyProject/MyProject/target/MyProject]
17:08:15 [INFO] Processing war project
17:08:15 [INFO] Copying webapp resources [/scratch/jenkins/workspace/MyProject/MyProject/src/main/webapp]
17:08:15 [INFO] Webapp assembled in [662 msecs]
17:08:15 [INFO] Building war: /scratch/jenkins/workspace/MyProject/MyProject/target/MyProject.war
17:08:20 [INFO] 
17:08:20 [INFO] --- maven-failsafe-plugin:2.17:integration-test (default) @ MyProject ---
17:08:20 [JENKINS] Recording test results
17:08:25 [INFO] 
17:08:25 [INFO] --- maven-failsafe-plugin:2.17:verify (default) @ MyProject ---
17:08:25 [INFO] Failsafe report directory: /scratch/jenkins/workspace/MyProject/MyProject/target/failsafe-reports
17:08:25 [JENKINS] Recording test results[INFO] 
17:08:25 [INFO] --- maven-install-plugin:2.4:install (default-install) @ MyProject ---
17:08:25 
17:08:25 [INFO] Installing /scratch/jenkins/workspace/MyProject/MyProject/target/MyProject.war to /home/jenkins/.m2/repository/org/mainco/subco/MyProject/76.0.0-SNAPSHOT/MyProject-76.0.0-SNAPSHOT.war
17:08:25 [INFO] Installing /scratch/jenkins/workspace/MyProject/MyProject/pom.xml to /home/jenkins/.m2/repository/org/mainco/subco/MyProject/76.0.0-SNAPSHOT/MyProject-76.0.0-SNAPSHOT.pom
17:08:26 Started calculate disk usage of build
17:08:26 Finished Calculation of disk usage of build in 0 seconds
17:08:26 Started calculate disk usage of workspace
17:08:26 Finished Calculation of disk usage of workspace in 0 seconds
17:08:26 [INFO] ------------------------------------------------------------------------
17:08:26 [INFO] BUILD SUCCESS
17:08:26 [INFO] ------------------------------------------------------------------------
17:08:26 [INFO] Total time: 11:00.616s
17:08:26 [INFO] Finished at: Mon Feb 23 17:08:26 UTC 2015
17:08:27 [INFO] Final Memory: 90M/414M
17:08:27 [INFO] ------------------------------------------------------------------------
17:08:27 Waiting for Jenkins to finish collecting data
17:08:28 [JENKINS] Archiving /scratch/jenkins/workspace/MyProject/MyProject/pom.xml to org.mainco.subco/MyProject/76.0.0-SNAPSHOT/MyProject-76.0.0-SNAPSHOT.pom
17:08:28 [JENKINS] Archiving /scratch/jenkins/workspace/MyProject/MyProject/target/MyProject.war to org.mainco.subco/MyProject/76.0.0-  SNAPSHOT/MyProject-76.0.0-SNAPSHOT.war
17:08:31 channel stopped
17:08:31 Started calculate disk usage of build
17:08:31 Finished Calculation of disk usage of build in 0 seconds
17:08:31 Started calculate disk usage of workspace
17:08:31 Finished Calculation of disk usage of workspace in 0 seconds
17:08:31 Finished: UNSTABLE

3 Answers 3

64

You can add -Dmaven.test.failure.ignore=false to the MAVEN_OPTS if you click on Advanced button in the Build section of your Jenkins Job.

See Maven Surefire Plugin - surefire:test options for reference.

5
  • 2
    But do we really have to specify this flag as false ? As per my understanding if we don't specify this flat then default value is "FALSE" only. Please suggest. Thanks!
    – Paresh
    Nov 9, 2015 at 3:51
  • 13
    Jenkins specifies this value to true per default. Have a look at issues.jenkins-ci.org/browse/JENKINS-24655
    – Torsten
    Nov 9, 2015 at 10:47
  • 1
    Keep in mind the documentation tells "Its use is NOT RECOMMENDED".
    – Kai
    Jan 19, 2017 at 9:13
  • 5
    @Kai To be clear, setting -Dmaven.test.failure.ignore=true is what is not recommended based upon the documentation in your link. If Jenkins sets this to a value not recommended then going about changing it back to false complies with the spirit of the warning in the documentation. Jan 25, 2017 at 0:07
  • But this does not show you which tests have failed in the Jenkins UI.
    – vitro
    Dec 6, 2019 at 1:56
10

Use Text Finder plugin. Configure it to look for There are test failures and downgrade the build to FAILED

3
  • 8
    Nice hack, but a hack none the less.
    – Alexey
    Jul 20, 2016 at 12:14
  • In addition to being a hack, I found it failed to downgrade an UNSTABLE to a FAILED build for me. I thought it might be something to do with the documented "the reclassification only applies to builds which returned an overall exit status of zero".
    – bdeniker
    Sep 6, 2016 at 10:48
  • 1
    For Jenkins pipelines I found this hack to be the only manageable solution: Run mvn with -Dmaven.test.failure.ignore=true in a try-finally. In the finally have findText regexp: 'There are test failures', alsoCheckConsoleOutput: true, unstableIfFound: true and junit allowEmptyResults: true, testResults: 'target/surefire-reports/*.xml'
    – Terran
    Jul 3, 2019 at 12:59
-1

Another hack that can be useful is to use groovy post-build to check and set test result.

e.g. this groovy gets build result, appends useful things to build description and sets build result to UNSTABLE in the case that there are no pass or fail but all tests skipped.

def currentBuild = Thread.currentThread().executable
// must be run groovy post-build action AFTER harvest junit xml  if you use junit xml test results
testResult1 = currentBuild.testResultAction

currentBuild.setDescription(currentBuild.getDescription() + "\n pass:"+testResult1.result.passCount.toString()+", fail:"+testResult1.result.failCount.toString()+", skip:"+testResult1.result.skipCount.toString())

// if no pass, no fail all skip then set result to unstable
if (testResult1.result.passCount == 0 && testResult1.result.failCount == 0 && testResult1.result.skipCount > 0) {
   currentBuild.result = hudson.model.Result.UNSTABLE
}

currentBuild.setDescription(currentBuild.getDescription() + "\n" + currentBuild.result.toString())

def ws = manager.build.workspace.getRemote()
myFile = new File(ws + "/VERSION.txt")
desc = myFile.readLines()
currentBuild.setDescription(currentBuild.getDescription() + "\n" + desc)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.