86

I am looking for ways to run test suites in parallel.

I am aware of .testrunconfig setting. This allows you to multiplex on the number of CPUs.

I want to run 1000 tests in parallel. This makes sense because I am testing a web service, so 90% of the time spent in a test is waiting for the service to respond.

Any ideas on how to pull this off ? The tests are written for VS, but I am open to running them outside of VS.

Later edit: the Visual Studio test team have added this in VS 2015 Update 1. See Mark Sowul's answer bellow.

6
  • Your talking here about 1000 threads. Oct 12, 2010 at 16:53
  • Right, I am wondering if there is a pre-build framework to manage this. Or if someone build their own framework. Oct 12, 2010 at 17:13
  • 2
    You need a load simulator. MSTest isn't meant to be used as a load test. 1000 threads is insane by the way, what you need to do is look into different testing technologies, this is outside the bounds of a unit test.
    – Aren
    Oct 12, 2010 at 17:47
  • 1
    I am not looking to perform load tests. VS has a pretty good module for that. I just want to run unit tests in parallel. Ok, 1000 too much ? 100 then :) Oct 13, 2010 at 8:55
  • Duplicate of stackoverflow.com/questions/921953/…
    – Pat
    Jun 22, 2012 at 21:42

6 Answers 6

69

Most of the answers on this page forget to mention that MSTest parallelizes tests in separate assemblies. You have to split your unit tests into multiple .dll's to parallelize it.

But! The recent version - MSTest V2 - now CAN parallelize "in-assembly" (yay!). You just need to install a couple of NuGet packages in your test project - TestFramework and TestAdapter - like described here https://blogs.msdn.microsoft.com/devops/2018/01/30/mstest-v2-in-assembly-parallel-test-execution/

And then simply add this to your test project

[assembly: Parallelize(Workers = 4, Scope = ExecutionScope.ClassLevel)]

EDIT: You can also disable parallel execution for a specific test using the [DoNotParallelize] attribute on a test method.

2
  • 4
    Where does one place this snippet?
    – A X
    Apr 21, 2020 at 23:35
  • 6
    @Abr usually assembly-wide attributes are placed in AssemblyInfo.cs but actually you put this in any file, up to you. Apr 22, 2020 at 6:50
30

You can get up to 5 by using the method from the Visual Studio Team Test Blog

Keep in mind that there may be concurrency issues using this, as MSTest doesn't completely isolate each test (statics carry over, for example, making things interesting for code meant to run once).

(No idea why the limit is 5, but MSTest will not run them in parallel if parallelTestCount is set to more than 5. As per the comments below, this rule apparently changes with Visual Studio 2013)

4
  • Yup, just found that the hard way :) - I ran my suite on a 8 core and all tests were aborted... Nov 4, 2010 at 15:23
  • It shouldn't be limited to 5 in total, but 5 hung tests. Bruce Taimana's comment: "Should that clean up "hang" or take too long, we let it continue to cleanup but we kick off the next test in parallel. If you reach 5 "hung" tests, we abort." Read the whole comment here.
    – Anttu
    Mar 20, 2013 at 5:59
  • @Anttu I would want to see a solution that is set up to run more than 5 at once and have it actually run them instead of failing. Each time I've tried to have more than 5 for that setting it would just fail. Maybe the time for a hang is considered to be too small? I don't really know.
    – Rangoric
    Mar 20, 2013 at 13:24
  • I'm using MSTest 12.0.21005.1 (Visual studio 2013) and there is no thread limit.
    – Nir
    Jul 23, 2015 at 13:08
30

Visual Studio 2015 Update 1 adds this. https://learn.microsoft.com/visualstudio/releasenotes/vs2015-update1-vs#misc

For Update 2, there is a UI toggle button in the toolbar at the top of the Test Explorer pane (between the 'grouping' and 'search' boxes).

For Update 1, Set the following in the .runsettings

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration>
    <MaxCpuCount>0</MaxCpuCount>
   </RunConfiguration>
</RunSettings>

The value for MaxCpuCount has the following semantics:

• ‘n’ (where 1 <= n <= number of cores) : upto ‘n’ processes will be launched.

• ‘n’ of any other value : The number of processes launched will be as many as the available cores on the machine.

Note also for MSTest V2, you can apply parallelism at the class level, either with assembly directives:

[assembly: Parallelize(Workers = 3, Scope = ExecutionScope.ClassLevel)]

or the .runsettings:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <!-- MSTest adapter -->
  <MSTest>
    <Parallelize>
      <Workers>4</Workers>
      <Scope>ClassLevel</Scope>
    </Parallelize>
  </MSTest>
</RunSettings>

An assembly, class, or method can opt out with [DoNotParallelize]

See https://github.com/Microsoft/testfx-docs/blob/master/RFCs/004-In-Assembly-Parallel-Execution.md

5
  • 3
    We have test DLLs with hundreds of tests, some can run parallel, some can't. Is there a way of attributing safe and unsafe test classes/methods? I typically want to select one or more of these huge DLLs and let it rip. Apr 5, 2016 at 14:23
  • 3
    One thing to keep in mind is that the parallelism is on the class level, not the test level. I don't see a built-in way to do what you want, but you could do it yourself, e.g. with named Mutexes in ClassInitialize/ClassCleanup. Unfortunately you could end up with those tests blocking the parallelism for a while (e.g. all four 'parallel tests' are ones that use the mutex), but that's as good as I think you can get.
    – Mark Sowul
    Apr 5, 2016 at 14:58
  • 4
    Tests are parallelized at the container level (in case of C#, that's Assembly), so it will run tests from different assemblies in parallel. Tests in the same assembly are executed in sequence in the new model. May 9, 2016 at 11:59
  • @jessehouwing I have observed parallelism at the class level with massive speed differences. I think there is some hidden logic that decides how to parallelize the tests.
    – Gusdor
    May 25, 2016 at 7:14
  • In some cases the file is not picked up automatically even it it is there and named correctly but than than you can configure to be picked up from Visual studio menu Test -> Configure Run Settings-> Select Solution Wide runsettings file. (Browse to file) May 7, 2021 at 15:33
8

What I found is that C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe will run parallel tests with a .testsettings file which looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="TestSettings1" id="21859d0f-7bdc-4165-b9ad-05fc803c9ee9" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description>These are default test settings for a local test run.</Description>
  <Deployment enabled="false" />
  <Execution parallelTestCount="8">
    <TestTypeSpecific>
      <UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
        <AssemblyResolution>
          <TestDirectory useLoadContext="true" />
        </AssemblyResolution>
      </UnitTestRunConfig>
    </TestTypeSpecific>
    <AgentRule name="Execution Agents">
    </AgentRule>
  </Execution>
</TestSettings>

Reference can be found here http://msdn.microsoft.com/en-us/library/vstudio/jj155796.aspx

2
  • 1
    It also works if you use the Test Explorer when within Visual Studio. You only need to pick the .testsettings configured with parallelTestCount. More details here: ardalis.com/… By the way: to pick the .testsettings, click the TEST menu in VS 2013 then Test Settings => Select Test Settings File. Jul 17, 2014 at 22:47
  • I've never seen such a beast. In 2013 I had to create my own test settings file (Add Item at the Solution level), then modify it for parallel support as per Igor's instructions. Aug 4, 2014 at 18:43
6

The above answers definitely helped clarify things for me, but, this point from John Koerner's blog: https://johnkoerner.com/vs2015/parallel-test-execution-in-visual-studio-2015-update-1-might-not-be-what-you-expect/ was the bit we were missing.

"Parallel test execution leverages the available cores on the machine, and is realized by launching the test execution engine on each available core as a distinct process, and handing it a container (assembly, DLL, or relevant artifact containing the tests to execute), worth of tests to execute."

--> "The separate container bit is the piece I was missing. In order to get my tests to run in parallel, I needed to split up my tests into separate test assemblies. After doing that, I saw that the tests in different assemblies were running in parallel."

So yeah, we got the tests running in parallel in VSTS by using their handy 'run in parallel' flag, but it wasn't enough, we had to split our tests up into separate test projects. Logically grouped of course, not a project-per-test which would be ridiculous

0
1
  1. Ensure the first column in your DataTable is a unique Id.
  2. Create a AsyncExecutionTask delegate that accepts a DataRow and returns nothing.
  3. Create a static class (ParallelTesting) with a AsyncExecutionContext method that accepts a DataRow and an AsyncExecutionTask delegate.
  4. In the static class add a static BatchStarted property.
  5. In the static class add a static AsyncExecutionTests Dictionary property.
  6. In the AsyncExecutionContext method add the following:

    public static void AsyncExecutionContext(DataRow currentRow, AsyncExecutionTask test) 
    {
        if(!BatchStarted)
        {
            foreach(DataRow row in currentRow.Table)
            {
                Task testTask = new Task(()=> { test.Invoke(row); });
                AsyncExecutionTests.Add(row[0].ToString(), testTask);
                testTask.Start();
            }
            BatchStarted = true;
        }
        Task currentTestTask = AsyncExecutionTests[row[0].ToString()];
        currentTestTask.Wait();
        if(currentTestTask.Exception != null) throw currentTestTask.Exception;
    }
    
  7. Now use the class like so:

    [TestMethod]
    public void TestMethod1()
    {
        ParallelTesting.AsyncExecutionContext(TestContext.DataRow, (row)=>
            {
                //Test Logic goes here.
            }
        );
    }
    

Note: You will have to do some tinkering with exceptions to get them to bubble correctly (you may have an aggregate exception here, you'll need the first exception from it). The amount of time displayed that each test takes to execute will no longer be accurate. You will also want to cleanup the ParallelTesting class after the last row is completed.

How it works: The test logic is wrapped in a lambda and passed to a static class that will execute the logic once for each row of test data when it is first called (first row executed). Successive calls to the static class simply wait for the prestarted test Task to finish.

In this way each call the test framework made to the TestMethod simply collects the test results of the corresponding test that was already run.

Possible Improvements:

  • Make AsyncExecutionContext take a maxSynchronousTasks parameter.
  • Look into how the framework moves complete stacktraces across unmanaged code to see if the Task.Exception can be passed to the visual studio test framework without rethrowing and destroying the stacktrace.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.