0

We run behave BDD tests in our pipeline. We run the tests in the docker container as part of the jenkins pipeline. Currently it takes ~10 minutes to run all the tests. We are adding a lot of tests and in few months, it might go upto 30 minutes. It is outputting a lot of information. I believe that if I reduce the amount of information it outputs, I can get the tests to run faster. Is there a way to control the amount of information behave outputs? I want to print the information only if something fails. I took a look at behave-parallel. Looks like it is in python 2.7. We are in python3. I was looking at various options behave provides.

behave -verbose=false folderName (I assumed that it will not output all the steps) behave --logging-level=ERROR TQXYQ (I assumed it will print only if there is an error) behave --logging-filter="Test Step" TQXYQ (I assumed it will print only the tests that has "Test Step" in it) None of the above worked.

The current output looks like this

Scenario Outline: IsError is populated correctly based on Test Id -- @1.7 # TestName/Test.feature:187 Given the test file folder is set to /TestName/steps/ # common/common_steps.py:22 0.000s And Service is running # common/common_steps.py:10 0.000s Given request used is current.json # common/common_steps.py:26 0.000s And request is modified to set X to q of type str # common/common_steps.py:111 0.000s And request is modified to set Y to null of type str # common/common_steps.py:111 0.000s And request is modified to set Z to USD of type str # common/common_steps.py:111 0.000s
When make a modification request # common/common_steps.py:37 0.203s Then it returns 200 status code # common/common_steps.py:47 0.000s And transformed result has IsError with 0 of type int # common/common_steps.py:92 0.000s And transformed result has ErrorMessages contain [] # common/common_steps.py:52 0.000s

I want to print only all these things only if there is an error. If everything is passing, I don't want to display this information.

2
  • 1
    As you can see from the output the latency comes from a request while all other steps finish within a millisecond. So, don't expect the speed to increase significantly if you reduce the output.
    – Klaus D.
    Nov 18, 2018 at 15:26
  • Thanks Klaus. I can also run the tests in separate containers to see whether it reduces the time.
    – Bala
    Nov 18, 2018 at 15:49

2 Answers 2

1

I think the default log level INFO will not impact the performance of your tests.
I am also using docker container to run the regression suite and it takes about 2 hours to run 2300 test scenarios. It took nearly a day before and here is what I did :
1. Run all test suite parallel.
This is the most important reason that will reduce the execution time.
We spent a lot of efforts to turn the regression suite to be parallel-able.
- make atomic, autonomous and independent tests so that you can run all your tests in parallel effectively.
- create a parallel runner to run tests on multiple processes. I am using multiprocessing and subprocessing libraries to do this.
I would not recommend behave-parallel because it is no longer active supported.
You can refer to this link :
http://blog.crevise.com/2018/02/executing-parallel-tests-using-behave.html?m=1
- using Docker Swarm to add more nodes into Selenium Grid.
You can scale up to add more nodes and the maximum numbers of nodes depend on the number of cpus. The best practice is number of node = number of cpu.
I have 4 PCs , each has 4 cores so I can scale up to 1 hub and 15 nodes.

2. Optimize synchronization in your framework.
Remove time.sleep()
Remove implicitly wait. Use explicitly wait instead.

Hope it helps.

6
  • Hi Thach Hoang. Thanks for your response. I am starting with running the same test in multiple containers to check whether it works ok. Unfortunately the containers become slower when running in parallel. We have 2 services (one for running the test and the other for actual service to be tested). I have 4 containers (2 containers for each service). First test container talks to first service. second test container talks to second service. They are running in parallel. Have you come across this? I read that, there is a chance this can happen as containers share the same kernel space.
    – Bala
    Dec 3, 2018 at 15:47
  • Hi @Bala , I assume that you have 4 containers on the same docker host. 2 containers for testing and the others for AUT. In this scenario, you need to create 2 separate network for each application using user-defined bridge networks. $ docker network create --driver bridge custom-net1 . The first test container and the first app container will be assigned to this network. The second test container and second app container need to be assigned in different network like custom-net2. So test1 can only reach app1, test2 only reaches app2. Hope it helps. Dec 7, 2018 at 6:11
  • Also, please make sure your PC performance is still good when running all containers, check your CPU percentages. If you have other PCs available , I would suggest to use Docker Swarm for more resources. Dec 7, 2018 at 6:11
  • reference link for you: docs.docker.com/network/network-tutorial-standalone/… Dec 7, 2018 at 6:12
  • Thank you so much Thach. That makes sense. I missed the part where we need to create different networks. I will do that and let you know. Thanks again.
    – Bala
    Dec 7, 2018 at 20:31
0

Well I have solved this in a traditional way, but I m not sure how effective it could be. I just started this yesterday and now trying to work on building the reports out of it. Approach as below, suggestions welcome

this solves the parallel execution at the example driven as well.

parallel_behave.py Run command (mimics all the params of behave command) py parallel_behave.py -t -d -f ......

initial_command = 'behave -d -t <tags>'
'''
the above command returns the eligible cases. may not be the right approach,                  but works well for me
'''

r = subprocess.Popen(initial_command.split(' '), stdout=subprocess.PIPE, bufsize=0)
finalsclist = []

_tmpstr=''
for out in r.stdout:
    out = out.decode('utf-8')
    # print(out.decode('utf-8'))
    if shellout.startswith('[') :
        _tmpstr+=out
    if shellout.startswith('{') :
        _tmpstr+=out
    if shellout.startswith(']'):
        _tmpstr+=out
        break

scenarionamedt = json.loads(_tmpstr)

for sc in scenarionamedt:
    [finalsclist.append(s['name']) for s in sc['elements']]

now the finalsclist contains the scenario name
ts = int(timestamp.timestamp)
def foo:
   cmd = "behave -n '{}' -o ./report/output{}.json".format(scenarioname,ts)

pool = Pool(<derive based on the power of the processor>) 
pool.map(foo, finalsclist)

this will create that many processes of individual behave calls and generate the json output under report folder

*** there was a reference from https://github.com/hugeinc/behave-parallel but this is at the feature level. I just extended it to the scenarios and example ****

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.