-2

We have a TestNG tests suite for our enterprise product. It was developed more than 8 years ago and it is not supports to single ant command to run all tests. So, we need to run single test suite at once.

Now, I have changed the structure of test suite and updated testng.5.4-jdk.jar with testng.6.8.jar. As a enterprise application changing single jar also matters a lot as we need to make sure all other component should work with this update.

We are planing to have different Jenkins jobs (60 diff Jobs as each test suite maintained by different teams) for each individual suite.

So, I need a answer for management:

  1. Why single command to drive a test suite is good idea?
  2. How maintaining 60 different Jenkins Jobs is not a god idea?
  3. Basically the older testng.5.4 jar didn't have ITestContext class, so how this will help us in future, some code snippet will be appreciated.

4 Answers 4

1

Answers to such questions can only be opinion-based.

So:

  1. Single command to run all tests could be extremely helpful for developers. Before commit/push they could verify product state by single command (if your tests are good). From other hand, if there are so many suites (60) they could take a lot of time to execute, and eventually devs could begin skipping testing at all. Personally, I prefer to have one command to run tests per module (which is extremely easy with maven: mvn test or mvn verify for integration tests).
  2. The main problem here could be team communication. If a team maintains only one suite, will they be aware if fixes to their suite not broken down other suites? Anyway, I would recommend to keep both separate test suite run jobs and all-in-one job as a smoke test.
  3. Just read api docs. It seems it could help to get tests' meta-info.
1

Answers to such questions can only be opinion-based.

So:

1 : Gradle, Maven, and Ant handle all the dependencies downloading automatically. You want to use that to launch your test suites because you don't want to store all of those .jar artifacts in your local Git repository. That would be really really messy. If you need to do something like that, you could syncronize your downloaded libraries in Gradle using the Sync directive. Then you could benefit from having all your .jar libraries locally in your project, without polluting your source control repository and still be able to launch standalone command line by including those .jar files in a -classpath argument to the test run.

2 : Jenkins easily can handle 100+ jobs. There is nothing unusual about that and I would recommend it.

3 : The TestNG ITestContext really helps you mainly in the @Before and @After configuration methods so that you can get to the test context from outside the regular @Test block, and that includes the DataProvider method signature.

@BeforeMethod
public void beforeMethod(ITestContext iTestContext){
    String testName = iTestContext().getName();
1
  • Yeah ! I know the pain to store the .jar(any binary files) in source control systems. History rewinding simply becomes too hard if binary files are in repo :( Apr 21, 2015 at 5:54
1

Sometimes it is helpful to be able to look at the state of ONE jenkins job to see if all tests passed. The Dashboards plugin is a prime example where you have limited visual space and want to be able to view the state of your entire system at a glace (for managers that like giant monitors in public places that have updates / status messages).

Just keep in mind that each jenkins job will download the entire repository into its workspace. So if you have 60 different jenkins jobs, then there will be 60 different copies of the entire workspace, which could clutter your jenkins server and wastes space on your build nodes. You get skirt this problem by putting unit test info into a dedicated repo for unit tests. But then you need to pass the compiled artifacts URL into the test workspace so the unit tests know what to run against.

0

Just to summerize what I get is :-

  • Storage :- If I used 60 different jobs, it will clone 60 git repositories, projects, tests outputs.
  • Jenkins Management :-

  • To manage single job is always good over so many because I also have to hook other test suites.
  • Also, We should have to increase number of slave machine Executors (infrastructure cost) to run this many tests at once.
  • Having many machine need more maintenance(Plugins, tools and dependency software installed on it and that needs to be updated with time)

  • Single Dashboard :- For management perspective it's good to look at central global dashboard.
  • Too Many notifications :- Too many email will surely create spam and instead of that one mail with link to TestNG dashboard is always good.
  • ITestContext :- This is really very helpful to drive your test as below

The most usefulness of ITestContext is you can iterate over test tags and run multiple tests/suites and can pass HashMap as a param to drive your tests(Data driven testing)

Most of time if you have a some web app, the functionalists are same but your operations may vary based on input data. So, you can have some java code which can re-utilize through TestNG xml files.

<!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
<suite name="Some Tests" verbose="0">

    <test name="First Test">
        <parameter name="testConfigFile"
            value="TestConfig.xml" />
        <parameter name="testConfigFile"
            value="TestConfig1.xml" />
        <parameter "as many params "..... />
        <classes>
            <class name="com.testframework.FirstTests">
            </class>
            <class name="com.testframework.FoobarTests">
            </class>
        </classes>
    </test>

    <test name="Second Test">
        <parameter name="testConfigFile"
            value="SecondTestConfig.xml" />
        <classes>
            <class name="com.testframework.SecondTests">
            </class>
        </classes>
    </test>
</suite>

You can get all test data by this function

/**
     * This method parse input TestNG xml file and based on each <test></test>
     * input tags and iterate over all tests.
     * @param context
     * @return HashMap of all input tests params required for the tests.
     */
    @DataProvider(name = "DataFile")
    public Object[][] testdata(ITestContext context) {
        Map<String, String> parameters = (((ITestContext)context).getCurrentXmlTest())
                .getAllParameters();
        return new Object[][] { { parameters } };
    } 

And Finally, your test can keep iterating till your test complet

@Test(dataProvider="DataFile")
public void testImplementation(Map parameter) { 
 // here your unit test will go
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.