24

Ok, this is annoying.

MSTest executes all of my tests simultaneously which causes some of them to fail. No this is not because my tests are fragile and susceptible to build order rather it is because this is a demo project in which I use a Db4o object database running from a file.

So I have a couple of DataAccess tests checking that my repositories work correctly and boom, MSTest blows up. Since it tries to run all its tests at the same time it gets an error when a test tries to access the database file while other tests are using it.

Can anyone think of a quick way around this? I don't want to ditch MSTest (ok I do but another story) and I sure as heck don't want to run a full-blown database service so I'll take any way to force MSTest not to run simultaneously or tricks with opening files.

Anyone have any ideas?

3 Answers 3

34

You might want to try using a Monitor and entering in TestInitialize and exiting on TestCleanup. If your test classes all depend on the external file, you'll need to use a single lock object for all of them.

public static class LockClass
{
    public static object LockObject = new object();
}

...

[TestInitialize]
public void TestSetup()
{
     Monitor.Enter(LockClass.LockObject);
}

[TestCleanup]
public void TestCleanup()
{
     Monitor.Exit(LockClass.LockObject);
}

This should force all of your tests to run serially and as long as all of your tests pass/fail they should run. If any of them throws an unexpected exception, though, all the rest will hang since the Exit code won't be run for the test that blows up.

5
  • What do you mean by "an unexpected exception"? I tried throwing .NET and native (C++) exceptions and TestCleanup() appears to always be called.
    – mhenry1384
    Jul 28, 2009 at 16:34
  • Without checking, I think it is possible to have an exception (say a stack overflow) that would cause the thread to exit without being able to run the clean up code. In that case the Monitor would never exit.
    – tvanfosson
    Jul 28, 2009 at 20:22
  • 1
    On .Net 2.0 or latest, a StackOverflowException will crash the mstest.exe process. IOW the whole thing will simply shutdown and disappear from the list of running processes. In this case, releasing the Monitor or not is a moot point. Oct 22, 2009 at 20:23
  • For posterity: see answer from @user3202543 he makes a good point. Jan 16, 2014 at 16:02
  • Let's be clear, this is for unit tests. Failing tests should be fixed so I'm ok with it failing hard when test clean up isn't called. It could be an issue for continuous integration, though. Better make sure things are passing before check in.
    – tvanfosson
    Jan 16, 2014 at 17:57
5

I had a try using locks in this manner. What I experienced, however, was that VS2010 does not execute the tests in parallel by default, but executes them sequencially, in a single thread. (parallel execution could be switched on, however. But this would not prevent the problem completely)

What I find very disturbing is, that the sequencial execution will take place in arbitrary order, even across test classes!

So for example an execution order may look like this:

  • Class A - TestInitialize: Lock will be established
  • Class A - TestMethod1: Will execute, OK
  • Class B - TestInitialize: Lock will be established => Thread will be blocked => Complete UnitTests will be blocked! The cause is that there are no other Threads which would go on executing methods of Class A. So the Montor.Exit() will never be reached.

I do not understand why MS is doing so. Other UnitTest frameworks (e.g. JUnit) execute the test methods class-wise. Otherwise there will be some interleaving of SetUp/TearDown method which would cause the chaos described...

Is there anybody out there knowing how to prevent MSTest jumping between test classes? (Currently I use Resharpers test runner, which behaves as expected, executing all tests methods of one classe before proceeding with the next class)

4
  • 8
    Seven - I think one of my co-workers put it best: It's because MSTest is not a Unit Testing framework. It's a very generalist testing framework that tries to do too many things for too many people and fails at all of them. May 17, 2010 at 13:43
  • Jep, maybe this is just the all-in-one-thing-solution-paradigm M$ always drives, resulting in things capable of everything, but usable for nothing. So, probably we do need to switch to NUnit, as some of my colleagues do not use resharper.
    – Seven
    May 17, 2010 at 15:08
  • 1
    "that tries to do too many things for too many people and fails at all of them." applies to Microsoft as a whole, not just their unit testing "framework"
    – Jeff
    Sep 4, 2011 at 21:19
  • 6
    While I agree with you that mstest.exe messes up the order, as a test author, you should not depend on what order the tests are run. Every test should be runnable independent of any other test, and this is the only way you can produce effective test cases. Sep 16, 2011 at 18:52
3

Use an Ordered Test

http://msdn.microsoft.com/en-us/library/ms182630(v=VS.90).aspx

5
  • 1
    Sorry, doesn't answer the question. Jun 9, 2013 at 19:36
  • It sure does, an Ordered test runs all the tests in a sequential sequence which is exactly what you want! Oct 17, 2014 at 6:30
  • It does, but you have to explain why. Otherwise you are not really answering the question but just giving a suggestion
    – jgauffin
    Jan 15, 2015 at 8:52
  • Indeed, ordered tests are the way to go.
    – Sebazzz
    Aug 26, 2016 at 19:33
  • 1
    Again, it doesn't answer the issue. I'm running VS2015 and tests don't run by the order of the entries in Alphabetical.orderedtest file. I have several test classes and the order is respected only for tests inside a given class. Oct 16, 2017 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.