10

We've been using protractor for end-to-end testing for a while.

Now we are trying to cover several corner cases, which involve modifying the response from the API endpoint requests - for this we are using protractor-http-mock which provides an easy-to-use way to replace HTTP responses with pre-defined mocks.

But, what if we want to test a situation when a sudden connection loss happens? What are our options in this case?

In other words, we want to achieve a case when requests to specific endpoints would produce a network connection error and see how our application would react.


I'm open to any suggestions, I am currently thinking of the following strategies:

  • see if there are third-party nodejs libraries similar to protractor-http-mock
  • mock $http angularjs service
  • fire up a proxy and somehow control it during the tests (grunt-connect-proxy looks pretty mature though I'm not sure if it is possible to dynamically change the behavior of the proxy from spec to spec)
  • control it on a browser level - e.g. with a Network Throttling google chrome feature (though I'm pretty sure it is something that selenium cannot control, Network throttling with chrome and selenium) (browser addon/extension?)
11
  • Unit testing is so chic that developers often over test. I think you are reaching too far in your test effort. We could definitely work through ways to simulate a lost connection, but a lost connection should be an integration test. The best way to test lost connections is with to manually cut your connection or stop your server and observe the results. It's not a test you want repeated automatically, but rather with manual effort. The give-away that you are over-reaching for a unit test is when your mock / scenario is so complex it is in need of unit testing and debugging. Apr 4, 2015 at 3:32
  • @DaveAlperovich well, this is really an end-to-end integration test (not a unit test) we are asking about (using protractor). Though, I accept the point that it could be not worth the effort. Thank you!
    – alecxe
    Apr 4, 2015 at 4:01
  • 1
    take a look at sinon FakeXMLHttpRequest. You would have to implement a mock server response, which is an investment, and it's transactional, so it falls short in the sense that you ideally would want a random number of transactions or random time range for ideal testing. But, this is the best I can imagine to simulating a connection. Apr 4, 2015 at 15:45
  • 1
    Another thought, you could connect to blocked port. I suspect you would prefer to create a robust Server mocking pattern instead. PS, while I criticize your perfectionism, I admire it too. This kind of open-ended issue is fun once you start wrapping your mind around it. Apr 4, 2015 at 21:16
  • 2
    If you are going to mock a http service not to work then why mock it at all? Just make your tests run on invalid uri's, for that test. Apr 7, 2015 at 22:08

4 Answers 4

4

Seems like the Comcast tool would provide the majority of the functionality you need:

Comcast is a tool designed to simulate common network problems like latency, bandwidth restrictions, and dropped/reordered/corrupted packets.

It works by wrapping up some system tools in a portable(ish) way. On BSD-derived systems such as OSX, we use tools like ipfw and pfctl to inject failure. On Linux, we use iptables and tc. Comcast is merely a thin wrapper around these controls.

https://github.com/tylertreat/Comcast

Example on Linux:

comcast --device=eth0 --latency=250 --target-bw=1000 --default-bw=1000000 --packet-loss=10% --target-addr=8.8.8.8,10.0.0.0/24 --target-proto=tcp,udp,icmp --target-port=80,22,1000:2000

You can change the settings on the fly and put packet loss up to 100% as needed.

2
  • haha, good name, Simulating s****y network connections so you can build better systems.
    – lyjackal
    Apr 9, 2015 at 2:29
  • the name of this library rules.
    – looshi
    Oct 24, 2015 at 16:06
2
+200

I hope this helps you to define your best way to implement the mocks.

The project explains the general idea, there are semantics and translation problems with the readme.md, i'll working on that soon please be patient. Please forget my English/misspelling let me known of any correction

I was working on this Demo project of how to implement Protractor + CucumberJS + sugar-step . Right now it contains a very very simple Angular APP and 3 E2E Test.

The demo is working, but is unfinished yet, still need some features which i think are useful for E2E Testing.

  • The first test check a service against a default service (Dev environment mock)
  • The Second test check that the service request returns an error with status code 404
  • The Third test check that the service request returns an error with status code 500

How can i set different Mock responses for each Scenario?

  • Using Cucumber's Around Feature, which can be used to inject a angular.module('mock-Service-response-x',fn...) before any feature or scenario, and remove it after the feature or scenario runs

What can be tested this way?

  • Any kind of scenarios, responses with extreme delays, response any Status Code, response special data for test very particular scenarios which otherwise will be impossible/hard to obtain from a normal back-end service, like data inconsistency on the back-end

Why implement your UI-Mocks in AngularJS and not with a proxy?

  • UI Control, (of course depending of your particular needs) using Angular you can do things like: disable ngAnimations or jQuery animations entirely, Mock your local browser Date, to emulate behaviors which depends on date in a automatically.

  • Cost, (depending of your project) the cost of implement a local proxy for all the web services on every dev environment can be expensive (in hardware or effort or hours of implementation).

  • Dependency, if your dev teams are working in parallel, it's more likely to be depending from each-other, if the UI team is working in a feature that depends on a X Web Service, knowing the implementation specs the UI team can create simple mocks to continue with his work making more probable the deliver on time of the hole project

All those arguments depends of which project are you working on, and doesn't mean at all that this is the best solution, but i'm more in favor of keep the project layers independent from each other, and be able to release a new feature in one layer no matter if other layer is delayed in his own new features.

To anyone who reads this, any comment, correction or suggestion will be more than welcome

4
  • Thank you for the answer,. We currently simulate 404 and 500 server errors with protractor-http-mock, but need to simulate connection errors - haven't found an example in the project you've mentioned..
    – alecxe
    Apr 8, 2015 at 14:08
  • The project i posted doesn't implement that scenario yet, but please check this file. (Can you use protractor-http-mock for protractor for automation test on the UI ? I think only is useful on karma test) Apr 8, 2015 at 14:25
  • By "connection error" what do you mean? 444 No Response or 499 Client Closed Request Or you are talking of a service that never respond? If is the first case, the only thing to do is change the status code returned. If is the second case, is ver simple only have to do is something like: $httpBackend.whenPOST('/services/login').respond(function() { // never return, so the client will wait until detects any timeout fallback }); Apr 8, 2015 at 14:25
  • It is the second case, I'll try what you've suggested and come back to you, thanks.
    – alecxe
    Apr 8, 2015 at 14:35
1

That's also depends on how are going to do that? Should it be programmatically?

If so the best solution is proxy depending, and the best tool could be used with all selenium/webdriver tools is browserMobProxy,

If you want to do it from the outside, I would recommend to use any external Traffic shapers/proxies also. Eg: http://vaurien.readthedocs.org/en/1.8/ and

vaurien --protocol http --proxy nonexistingproxy.com:8000 --backend website.com:80

Why shouldn't you use 9n0browser capabilities? First of all, you will need to reload the page to see them in action, second one is you will not redefine the proxy settings in protractor with them. As for the plugins, it's not so easy to interact with them programmatically and I see no any benefits, while you should append a lot of the for different browsers.

1

There is a tool from Facebook that called ATC (A tool to simulate network conditions).

It allows you to control bandwith, latency, packet loss and packet corruption rates in your network.

Since it's open source, I think you can extend the functionality in order to use different "error" profiles, or set up integration with Webdriver.

You can find general information about ATC here. ATCs github repo is here.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.