I have Hudson as continuous integration server and I want to use option 'Publish JUnit test result report'. But I don't use xUnit tools for testing, instead of that i have shell scripts which run tests and return results in simple format. I am thinking to make a script which transforms these results to the JUnit format. So i'm interesting how the JUnit file must look?
-
Any reason to not use JUnit? These tests can be automated in a variety of fashions via a variety of tools cmd, UI, etc...– Aaron McIverFeb 7, 2011 at 15:16
-
6@AaronMcIver: Shell scripts are pretty good at running tests on (language that is not Java). How would you use JUnit for that?– Ben VoigtNov 8, 2011 at 19:29
-
1@BenVoigt I had initially assumed the OP had Java involved and was looking to bypass JUnit as the testing harness. This is most likely not the case after reviewing the question. It appears that code.google.com/p/shell2junit may provide some use to the OP after a second look.– Aaron McIverNov 8, 2011 at 19:49
-
1Along the lines of shell2unit here is a JAXB class that I created that can parse/output JUnit XML: gist.github.com/agentgt/8583649– Adam GentJan 23, 2014 at 18:06
8 Answers
I did a similar thing a few months ago, and it turned out this simple format was enough for Hudson to accept it as a test protocol:
<testsuite tests="3">
<testcase classname="foo1" name="ASuccessfulTest"/>
<testcase classname="foo2" name="AnotherSuccessfulTest"/>
<testcase classname="foo3" name="AFailingTest">
<failure type="NotEnoughFoo"> details about failure </failure>
</testcase>
</testsuite>
This question has answers with more details: Spec. for JUnit XML Output
-
Please make a correction to this answer, because xunit plugin rejects the 'classname' attribute and accepts just 'class'– andhoNov 5, 2011 at 7:45
-
12i had the opposite issue.
class
was rejected and onlyclassname
worked. Feb 9, 2012 at 18:35 -
1This started failing for me when I upgraded the xUnit plugin to 1.60. I found that the validator became stricter and I had to add
<testsuite tests="(number of tests)">
ex.<testsuite tests="10">
. Jun 7, 2013 at 12:17 -
2Thanks @KevinBrotcke, I'll update the answer to include that attribute. Jun 7, 2013 at 12:35
-
3Also note that to have Hudson organize your tests by package/suite, you must specify a package in the classname attribute. ex:
<testcase classname="foo.bar" name="ATest" />
This will put the bar class in a foo package on Jenkins making your test collection more organized.– jluzwickSep 11, 2013 at 16:18
I just grabbed the junit-4.xsd that others have linked to and used a tool named XMLSpear to convert the schema to a blank XML file with the options shown below. This is the (slightly cleaned up) result:
<?xml version="1.0" encoding="UTF-8"?>
<testsuites disabled="" errors="" failures="" name="" tests="" time="">
<testsuite disabled="" errors="" failures="" hostname="" id=""
name="" package="" skipped="" tests="" time="" timestamp="">
<properties>
<property name="" value=""/>
</properties>
<testcase assertions="" classname="" name="" status="" time="">
<skipped/>
<error message="" type=""/>
<failure message="" type=""/>
<system-out/>
<system-err/>
</testcase>
<system-out/>
<system-err/>
</testsuite>
</testsuites>
Some of these items can occur multiple times:
- There can only be one
testsuites
element, since that’s how XML works, but there can be multipletestsuite
elements within thetestsuites
element. - Each
properties
element can have multipleproperty
children. - Each
testsuite
element can have multipletestcase
children. - Each
testcase
element can have multipleerror
,failure
,system-out
, orsystem-err
children.
-
1Is there a document that describes the valid values of certain attributes, like testcase's status or error's type? Jun 27, 2012 at 21:39
-
1@EricCope I can recommend looking at the source code svn.apache.org/viewvc/ant/core/trunk/src/main/org/apache/tools/…. Basically it's just a string.– SulthanNov 14, 2012 at 16:21
-
4
-
settings mirror : imgur.com/quneFJf alt:
Rootelement
:testsuites
,Max recursive de...
:2
,Max Repeat factor
:2
,include optional elements
: (yes=ticked),include optional attributes
: (yes=ticked)– n611x007Apr 22, 2015 at 13:20 -
1
The top answer of the question Anders Lindahl refers to an xsd file.
Personally I found this xsd file also very useful (I don't remember how I found that one). It looks a bit less intimidating, and as far as I used it, all the elements and attributes seem to be recognized by Jenkins (v1.451)
One thing though: when adding multiple <failure ...
elements, only one was retained in Jenkins. When creating the xml file, I now concatenate all the failures in one.
Update 2016-11 The link is broken now. A better alternative is this page from cubic.org: JUnit XML reporting file format, where a nice effort has been taken to provide a sensible documented example. Example and xsd are copied below, but their page looks waay nicer.
sample JUnit XML file
<?xml version="1.0" encoding="UTF-8"?>
<!-- a description of the JUnit XML format and how Jenkins parses it. See also junit.xsd -->
<!-- if only a single testsuite element is present, the testsuites
element can be omitted. All attributes are optional. -->
<testsuites disabled="" <!-- total number of disabled tests from all testsuites. -->
errors="" <!-- total number of tests with error result from all testsuites. -->
failures="" <!-- total number of failed tests from all testsuites. -->
name=""
tests="" <!-- total number of successful tests from all testsuites. -->
time="" <!-- time in seconds to execute all test suites. -->
>
<!-- testsuite can appear multiple times, if contained in a testsuites element.
It can also be the root element. -->
<testsuite name="" <!-- Full (class) name of the test for non-aggregated testsuite documents.
Class name without the package for aggregated testsuites documents. Required -->
tests="" <!-- The total number of tests in the suite, required. -->
disabled="" <!-- the total number of disabled tests in the suite. optional -->
errors="" <!-- The total number of tests in the suite that errored. An errored test is one that had an unanticipated problem,
for example an unchecked throwable; or a problem with the implementation of the test. optional -->
failures="" <!-- The total number of tests in the suite that failed. A failure is a test which the code has explicitly failed
by using the mechanisms for that purpose. e.g., via an assertEquals. optional -->
hostname="" <!-- Host on which the tests were executed. 'localhost' should be used if the hostname cannot be determined. optional -->
id="" <!-- Starts at 0 for the first testsuite and is incremented by 1 for each following testsuite -->
package="" <!-- Derived from testsuite/@name in the non-aggregated documents. optional -->
skipped="" <!-- The total number of skipped tests. optional -->
time="" <!-- Time taken (in seconds) to execute the tests in the suite. optional -->
timestamp="" <!-- when the test was executed in ISO 8601 format (2014-01-21T16:17:18). Timezone may not be specified. optional -->
>
<!-- Properties (e.g., environment settings) set during test
execution. The properties element can appear 0 or once. -->
<properties>
<!-- property can appear multiple times. The name and value attributres are required. -->
<property name="" value=""/>
</properties>
<!-- testcase can appear multiple times, see /testsuites/testsuite@tests -->
<testcase name="" <!-- Name of the test method, required. -->
assertions="" <!-- number of assertions in the test case. optional -->
classname="" <!-- Full class name for the class the test method is in. required -->
status=""
time="" <!-- Time taken (in seconds) to execute the test. optional -->
>
<!-- If the test was not executed or failed, you can specify one
the skipped, error or failure elements. -->
<!-- skipped can appear 0 or once. optional -->
<skipped/>
<!-- Indicates that the test errored. An errored test is one
that had an unanticipated problem. For example an unchecked
throwable or a problem with the implementation of the
test. Contains as a text node relevant data for the error,
for example a stack trace. optional -->
<error message="" <!-- The error message. e.g., if a java exception is thrown, the return value of getMessage() -->
type="" <!-- The type of error that occured. e.g., if a java execption is thrown the full class name of the exception. -->
></error>
<!-- Indicates that the test failed. A failure is a test which
the code has explicitly failed by using the mechanisms for
that purpose. For example via an assertEquals. Contains as
a text node relevant data for the failure, e.g., a stack
trace. optional -->
<failure message="" <!-- The message specified in the assert. -->
type="" <!-- The type of the assert. -->
></failure>
<!-- Data that was written to standard out while the test was executed. optional -->
<system-out></system-out>
<!-- Data that was written to standard error while the test was executed. optional -->
<system-err></system-err>
</testcase>
<!-- Data that was written to standard out while the test suite was executed. optional -->
<system-out></system-out>
<!-- Data that was written to standard error while the test suite was executed. optional -->
<system-err></system-err>
</testsuite>
</testsuites>
JUnit XSD file
<?xml version="1.0" encoding="UTF-8" ?>
<!-- from https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd -->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="failure">
<xs:complexType mixed="true">
<xs:attribute name="type" type="xs:string" use="optional"/>
<xs:attribute name="message" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
<xs:element name="error">
<xs:complexType mixed="true">
<xs:attribute name="type" type="xs:string" use="optional"/>
<xs:attribute name="message" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
<xs:element name="properties">
<xs:complexType>
<xs:sequence>
<xs:element ref="property" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="property">
<xs:complexType>
<xs:attribute name="name" type="xs:string" use="required"/>
<xs:attribute name="value" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="skipped" type="xs:string"/>
<xs:element name="system-err" type="xs:string"/>
<xs:element name="system-out" type="xs:string"/>
<xs:element name="testcase">
<xs:complexType>
<xs:sequence>
<xs:element ref="skipped" minOccurs="0" maxOccurs="1"/>
<xs:element ref="error" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="failure" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="system-out" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="system-err" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="name" type="xs:string" use="required"/>
<xs:attribute name="assertions" type="xs:string" use="optional"/>
<xs:attribute name="time" type="xs:string" use="optional"/>
<xs:attribute name="classname" type="xs:string" use="optional"/>
<xs:attribute name="status" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
<xs:element name="testsuite">
<xs:complexType>
<xs:sequence>
<xs:element ref="properties" minOccurs="0" maxOccurs="1"/>
<xs:element ref="testcase" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="system-out" minOccurs="0" maxOccurs="1"/>
<xs:element ref="system-err" minOccurs="0" maxOccurs="1"/>
</xs:sequence>
<xs:attribute name="name" type="xs:string" use="required"/>
<xs:attribute name="tests" type="xs:string" use="required"/>
<xs:attribute name="failures" type="xs:string" use="optional"/>
<xs:attribute name="errors" type="xs:string" use="optional"/>
<xs:attribute name="time" type="xs:string" use="optional"/>
<xs:attribute name="disabled" type="xs:string" use="optional"/>
<xs:attribute name="skipped" type="xs:string" use="optional"/>
<xs:attribute name="timestamp" type="xs:string" use="optional"/>
<xs:attribute name="hostname" type="xs:string" use="optional"/>
<xs:attribute name="id" type="xs:string" use="optional"/>
<xs:attribute name="package" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
<xs:element name="testsuites">
<xs:complexType>
<xs:sequence>
<xs:element ref="testsuite" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="name" type="xs:string" use="optional"/>
<xs:attribute name="time" type="xs:string" use="optional"/>
<xs:attribute name="tests" type="xs:string" use="optional"/>
<xs:attribute name="failures" type="xs:string" use="optional"/>
<xs:attribute name="disabled" type="xs:string" use="optional"/>
<xs:attribute name="errors" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
</xs:schema>
-
How do you get the failures to look nice then? I would like to manually add new line characters but they don't show up in Jenkins. Aug 6, 2015 at 21:37
-
That's a disadvantage with my approach. I remember struggling with that a well. Try to add something like <br/> - I forgot how this was resolved (and we're not using this anymore), but this seems something worth trying.– parvusAug 13, 2015 at 12:57
-
1I found a way around it. Since we are using c++ I am just reporting the number of failures in the failure message and using the "Stack Trace" to report the actual failures. Since the stack trace is reported from the text withing the body of the failure element, new lines are supported correctly. Aug 13, 2015 at 18:44
I couldn't find any good information on this, so I did some trial and error. The following attributes and fields (and only these) are recognized by Jenkins (v1.585).
<?xml version="1.0" encoding="UTF-8"?>
<testsuite>
<!-- if your classname does not include a dot, the package defaults to "(root)" -->
<testcase name="my testcase" classname="my package.my classname" time="29">
<!-- If the test didn't pass, specify ONE of the following 3 cases -->
<!-- option 1 --> <skipped />
<!-- option 2 --> <failure message="my failure message">my stack trace</failure>
<!-- option 3 --> <error message="my error message">my crash report</error>
<system-out>my STDOUT dump</system-out>
<system-err>my STDERR dump</system-err>
</testcase>
</testsuite>
(I started with this sample XML document and worked backwards from there.)
-
Later versions of the Jenkins plugin can handle more fields in the jUnit.XML results specification.– MarkHuOct 25, 2021 at 18:15
-
I've decided to post a new answer, because some existing answers are outdated or incomplete.
First of all: there is nothing like JUnit XML Format Specification
, simply because JUnit doesn't produce any kind of XML or HTML report.
The XML report generation itself comes from the Ant JUnit task/ Maven Surefire Plugin/ Gradle (whichever you use for running your tests). The XML report format was first introduced by Ant and later adapted by Maven (and Gradle).
If somebody just needs an official XML format then:
- There exists a schema for a maven surefire-generated XML report and it can be found here: surefire-test-report.xsd.
- For an ant-generated XML there is a 3rd party schema available here (but it might be slightly outdated).
Hope it will help somebody.
-
Thank you for your clarification. I am trying to send JUnit test summaries from an old Jenkins 1.6 instance to Slack - maybe you help? Where would I place such a XML file?– JJDApr 26, 2017 at 15:33
-
@JJD Sorry, I don't get you. What do you exactly mean by such XML file? But I assume you already run your JUnit tests with ant/ maven/ gradle, yes? If yes, these tools, after tests execution, generate the nice summary report. Version of Jenkins doesn't matter here. May 2, 2017 at 11:30
-
Yes, my build runs via Gradle. I would like to send a JUnit test summary to a Slack channel while using Jenkins 1.6. Reading the GitHub discussion I thought I have to put a configuration XML file somewhere to let the Slack plugin pick up the test summary. Maybe I misunderstand.– JJDMay 2, 2017 at 12:21
-
1Please check that correctly generated test reports exist after Gradle finishes launching your JUnit tests. Then Slack plugin should be able to use these reports. May 4, 2017 at 14:37
-
1Finally, your advice pushed me into the right direction: I had to configure the correct path to look for the XML files. For my Android project with multiple Gradle product flavors, the following works:
**/build/test-results/**/TEST-*.xml
. Thank you very much!!!– JJDMay 9, 2017 at 17:13
Basic Structure Here is an example of a JUnit output file, showing a skip and failed result, as well as a single passed result.
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="JUnitXmlReporter" errors="0" tests="0" failures="0" time="0" timestamp="2013-05-24T10:23:58" />
<testsuite name="JUnitXmlReporter.constructor" errors="0" skipped="1" tests="3" failures="1" time="0.006" timestamp="2013-05-24T10:23:58">
<properties>
<property name="java.vendor" value="Sun Microsystems Inc." />
<property name="compiler.debug" value="on" />
<property name="project.jdk.classpath" value="jdk.classpath.1.6" />
</properties>
<testcase classname="JUnitXmlReporter.constructor" name="should default path to an empty string" time="0.006">
<failure message="test failure">Assertion failed</failure>
</testcase>
<testcase classname="JUnitXmlReporter.constructor" name="should default consolidate to true" time="0">
<skipped />
</testcase>
<testcase classname="JUnitXmlReporter.constructor" name="should default useDotNotation to true" time="0" />
</testsuite>
</testsuites>
Below is the documented structure of a typical JUnit XML report. Notice that a report can contain 1 or more test suite. Each test suite has a set of properties (recording environment information). Each test suite also contains 1 or more test case and each test case will either contain a skipped, failure or error node if the test did not pass. If the test case has passed, then it will not contain any nodes. For more details of which attributes are valid for each node please consult the following section "Schema".
<testsuites> => the aggregated result of all junit testfiles
<testsuite> => the output from a single TestSuite
<properties> => the defined properties at test execution
<property> => name/value pair for a single property
...
</properties>
<error></error> => optional information, in place of a test case - normally if the tests in the suite could not be found etc.
<testcase> => the results from executing a test method
<system-out> => data written to System.out during the test run
<system-err> => data written to System.err during the test run
<skipped/> => test was skipped
<failure> => test failed
<error> => test encountered an error
</testcase>
...
</testsuite>
...
</testsuites>
There are multiple schemas for "JUnit" and "xUnit" results.
- XSD for Apache Ant's JUnit output can be found at : https://github.com/windyroad/JUnit-Schema (credit goes to this answer: https://stackoverflow.com/a/4926073/1733117)
- XSD from Jenkins xunit-plugin can be found at : https://github.com/jenkinsci/xunit-plugin/tree/master/src/main/resources/org/jenkinsci/plugins/xunit/types (under
model/xsd
)
Please note that there are several versions of the schema in use by the Jenkins xunit-plugin (the current latest version is junit-10.xsd
which adds support for Erlang/OTP Junit format).
Some testing frameworks as well as "xUnit"-style reporting plugins also use their own secret sauce to generate "xUnit"-style reports, those may not use a particular schema (please read: they try to but the tools may not validate against any one schema). Python unittests in Jenkins? gives a quick comparison of several of these libraries and slight differences between the xml reports generated.
Good answers here on using python: (there are many ways to do it) Python unittests in Jenkins?
IMHO the best way is write python unittest tests and install pytest (something like 'yum install pytest') to get py.test installed. Then run tests like this: 'py.test --junitxml results.xml test.py'. You can run any unittest python script and get jUnit xml results.
https://docs.python.org/2.7/library/unittest.html
In jenkins build configuration Post-build actions Add a "Publish JUnit test result report" action with result.xml and any more test result files you produce.