256

I am going to be starting up a new project at work and want to get into unit testing. We will be using Visual Studio 2008, C#, and the ASP.NET MVC stuff. I am looking at using either NUnit or the built-in test projects that Visual Studio 2008 has, but I am open to researching other suggestions. Is one system better than the other or perhaps easier to use/understand than the other?

I am looking to get this project set up as kind of the "best practice" for our development efforts going forward.

0

18 Answers 18

99

Daok named all the pro's of Visual Studio 2008 test projects. Here are the pro's of NUnit.

  • NUnit has a mocking framework.
  • NUnit can be run outside of the IDE. This can be useful if you want to run tests on a non-Microsoft build server, like CruiseControl.NET.
  • NUnit has more versions coming out than visual studio. You don't have to wait years for a new version. And you don't have to install a new version of the IDE to get new features.
  • There are extensions being developed for NUnit, like row-tests, etc.
  • Visual Studio tests take a long time to start up for some reason. This is better in Visual Studio 2008, but it is still too slow for my taste. Quickly running a test to see if you didn't break something can take too long. NUnit with something like Testdriven.Net to run tests from the IDE is actually much faster. Especially when running single tests. According to Kjetil Klaussen, this is caused by the Visual Studio testrunner. Running MSTest tests in TestDriven.Net makes MSTest performance comparable to NUnit.
11
  • 19
    Can't you just use mstest.exe to run MSTest tests outside the IDE? Apr 29, 2009 at 20:26
  • 13
    Nunit having a mocking framework is not much of an advantage. I've been using VS 2008 unit test projects with the Moq framework which uses a novel approach, leveraging LINQ expression trees: code.google.com/p/moq
    – dso
    May 27, 2009 at 19:49
  • 5
    One more plus for Nunit, for me anyway, is that Resharper has a very nice UI wrapped around it which is much faster than the VS testing components. It even hyperlinks the stack trace of failed tests to your code.
    – Jeff Putz
    Jul 27, 2009 at 0:00
  • 7
    Unit testing is included in the professional version of VS 2008.
    – user179700
    Oct 5, 2009 at 22:20
  • 3
    @Jeff Putz: Resharper can run the Visual Studio unit-tests too, even outside of a test project.
    – Paul Ruane
    Aug 3, 2010 at 15:38
64

The unit-testing framework doesn't actually matter much, because you can convert test classes with separate project files and conditional compilation (like this, Visual Studio → NUnit):

 #if !NUNIT
  using Microsoft.VisualStudio.TestTools.UnitTesting;
 #else
  using NUnit.Framework;
  using TestClass = NUnit.Framework.TestFixtureAttribute;
  using TestMethod = NUnit.Framework.TestAttribute;
  using TestInitialize = NUnit.Framework.SetUpAttribute;
  using TestCleanup = NUnit.Framework.TearDownAttribute;
  using TestContext = System.String;
  using DeploymentItem = NUnit.Framework.DescriptionAttribute;
 #endif

The TestDriven.Net plugin is nice and not very expensive... With only plain Visual Studio 2008 you have to find the test from your test class or test list. With TestDriven.Net you can run your test directly from the class that you are testing. After all, unit tests should be easy to maintain and near the developer.

10
  • 12
    I voted this one down because NUnit has a richer syntax than MSTest, which means you can go from MSTest -> NUnit, but not vice versa unless you are VERY careful. History shows at least one of us aren't. Feb 9, 2009 at 11:43
  • 2
    I agree with Thomas. This will work assuming you are using the most basic assert statements, but the NUnit constraint model is very powerful and reason enough to choose NUnit over MSTest.
    – Mark
    Apr 23, 2009 at 13:57
  • I believe, this approach is taken by the ms pattern and practice group in the EntLib tests.
    – robi-y
    May 17, 2009 at 19:54
  • 1
    @Dan Neely if you're testing privates you're doing it wrong :(
    – JDPeckham
    Mar 28, 2011 at 2:30
  • 2
    @JDPeckham I'm not saying that conventions of what should/shouldn't be done with a tool aren't important, but at the end of the day getting the job done is the most important thing. If that means pounding a nail with the back side of a hatchet because the tool vendors don't sell hammers then the hatchet manufacturers are just going to have to live with being outraged. Mar 28, 2011 at 4:24
34

Benefits/changes of the Visual Studio 2008 built-in unit testing framework:

  1. The 2008 version now is available in professional editions (before it required expensive versions of Visual Studio, and this is just for developer unit testing) that left a lot of developers with the only choice of open/external testing frameworks.
  2. Built-in API supported by a single company.
  3. Use the same tools to to run and create tests (you may run them using the command line also MSTest).
  4. Simple design (granted without a mock framework, but this is a great starting point for many programmers).
  5. Long term support granted (I still remember what happened to NDoc, and I don't want to commit to a testing framework that might not be supported in five years, but I still consider NUnit a great framework).
  6. If using Team Foundation Server as your backend, you can create work items or bugs with the failed test data in a simple fashion.
4
  • 4
    I do think it's still telling about Microsoft's view of testing that it's NOT in the Standard version, just Professional and above.
    – J Wynia
    Mar 2, 2009 at 15:27
  • Agree, I would love to see it in standard and up. In express versions it will be overkill for a beginner.
    – Simara
    Mar 5, 2009 at 22:48
  • 1
    @J Wynia: Reading their decision to only include it in Professional and above as saying something about their view on testing is reading way too much into it. It's more likely a business decision than it is a philosophical one.
    – jason
    Nov 9, 2010 at 2:39
  • @Simara Testing shoud be inherent to the development lifecycle. It shoud be provided in express editions.
    – eastender
    Sep 17, 2013 at 6:00
33

I have been using NUnit for two years. All is fine, but I have to say that the unit testing system in Visual Studio is pretty nice, because it's inside the GUI and can more easily do a test for private function without having to mess around.

Also, the unit testing of Visual Studio lets you do covering and other stuff that NUnit alone can't do.

11
  • 44
    You shouldn't touch your privates. All kidding aside, one school of thought is that all you need the test are you'r public methods. Calling all your public methods should call all your private methods. If a private method is not being called through a public one, the private method is redundant. Mar 8, 2009 at 10:30
  • 7
    @Lieven: if you are testing privates though the public you are really doing an intergration test and not a unit test. (Of course I am not a TDD zealot and I would probably just test the publics... but I feel like helping to start a fight) Jul 8, 2009 at 20:52
  • 11
    @Matthew, integration testing is testing two or more units together. Testing private methods is just a (huge) violation of encapsulation and may lead to brittle unit tests that have to be modified everytime the implementation changes. Aug 19, 2009 at 10:15
  • 3
    You can also use [assembly:InternalsVisibleTo(...)] with a compiler directive to take it out when you're making an RTM build. Nov 26, 2009 at 10:03
  • 1
    I touch my privates all the time :) I think there is value in specifically testing private members to avoid the overhead from testing callers that require a lot of complex setup. May 20, 2010 at 20:15
14

One slight annoyance of Visual Studio's testing framework is that it will create many test run files that tend to clutter your project directory - though this isn't that big of a deal.

Also, if you lack a plugin such as TestDriven.NET, you cannot debug your NUnit (or MbUnit, xUnit, etc.) unit tests within the Visual Studio environment, as you can with the Microsoft Visual Studio testing framework, which is built in.

4
  • 3
    You can debug NUnit testing within Visual Studio 2005. Sep 24, 2008 at 17:21
  • You can also debug xunit but it's non-obvious how to set that up (properties page)
    – annakata
    Jan 19, 2009 at 10:40
  • 1
    You can easily debug NUnit by attaching the debugger to the running NUnit process like grimus said. No real disadvantage here. Nov 19, 2010 at 8:58
  • 1: Number of tests configurable in VS settings - I set it to one. 2: Agree with above comments - possible but awkward, 3: Overall, I prefer the built in VS testing environment.
    – RaoulRubin
    Jul 5, 2013 at 18:05
14

Slightly off-topic, but if you go with NUnit I can recommend using ReSharper - it adds some buttons to the Visual Studio UI that make it a lot easier to run and debug tests from within the IDE.

This review is slightly out-of-date, but explains this in more detail:

Using ReSharper as an essential part of your TDD toolkit

3
11

xUnit is another possibility for a greenfield project. It's got perhaps a more intuitive syntax, but it is not really compatible with the other frameworks.

0
11

My main beef with Visual Studio unit tests over NUnit is the Visual Studio test creation tends to inject a bunch of generated code for private member access.

Some might want to test their private methods, and some may not. That's a different topic.

My concern is when I'm writing unit tests they should be extremely controlled so I know exactly what I'm testing and exactly how I'm testing it. If there's auto generated code I'm losing some of that ownership.

11

I have done some TDD using both and (maybe I'm a little dumb) NUnit seems to be a lot faster and simpler to use to me. And when I say a lot, I mean a lot.

In MSTest, there is too many attributes, everywhere - the code that do the real tests is the tiny lines you may read here and there. A big mess. In NUnit, the code that do the test just dominates the attributes, as it should do.

Also, in NUnit, you just have to click on the tests you want to run (only one? All the tests covering a class? An assembly? The solution?). One click. And the window is clear and large. You get clear green and red lights. You really know what happens in one sight.

In VSTS, the test list is jammed in the bottom of the screen, and it's small and ugly. You have to look twice to know what happened. And you cannot run just one test (well, I did not find out yet!).

But I may be wrong, of course - I just read about 21 blog posts about "How to do simple TDD using VSTS". I should have read more; you are right.

For NUnit, I read one. And I was TDDing the same day. With fun.

By the way, I usually love Microsoft products. Visual Studio is really the best tool a developer can buy - but TDD and Work Item management in Visual Studio Team System sucks, really.

9

First I want to correct a wrong statement: you can run MSTest outside of Visual Studio using the command line. Although several CI tools, such as TeamCity, have better support for NUnit (probably would change as MSTest becomes more popular).

In my current project we use both and the only big difference we found that MSTest always runs as a 32 bit while NUnit runs as either 32 bit or 64 bit tests which only matters if your code uses native code that is 32/64 bit dependent.

9

I got messages that "NUnit file structure is richer than VSTest"... Of course, if you prefer the NUnit file structure, you can use this solution to the other way, like this (NUnit → Visual Studio):

 #if !MSTEST
     using NUnit.Framework;
 #else
     using Microsoft.VisualStudio.TestTools.UnitTesting;
     using TestFixture = Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute;
     using Test = Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute;
     using SetUp = Microsoft.VisualStudio.TestTools.UnitTesting.TestInitializeAttribute;
     using TearDown = Microsoft.VisualStudio.TestTools.UnitTesting.TestCleanupAttribute;
 #endif

Or any other conversion... :-) This use here is just an alias to the compiler.

2
  • 1
    I don't understand what you are saying here. Feb 11, 2010 at 19:30
  • no fixture level setup/teardown :( or do they assume we use ctor and dtor?
    – JDPeckham
    Mar 28, 2011 at 2:33
8

I started with MSTest, but I switched for one simple reason. MSTest does not support inheritance of test methods from other assemblies.

I hated the idea of writing the same test multiple times. Especially on a large project where test methods can easily run into 100's of tests.

NUnit does exactly what I need. The only thing that is missing with NUnit is a Visual Studio addin which has can display the red/green status (like VSTS) of each test.

7

If you are considering either MSTest or NUnit, then I recommend you look at MbUnit. My reasons are

  1. TestDriven.Net compatibility. Nothing beats have TestDriven.Net.ReRunWithDebugger bound to a keyboard combination.
  2. The Gallio framework. Gallio is a test runner like NUnit's. The only difference is it doesn't care if you wrote your tests in NUnit, MSTest, xUnit or MbUnit. They all get run.
  3. Compatibility with NUnit. All features in NUnit are supported by MbUnit. I think you don't even need to change your attributes (will have to check that), just your reference and usings.
  4. Collection asserts. MbUnit has more Assert cases, including the CollectionAssert class. Basically you no longer need to write your own tests to see if two collections are the same.
  5. Combinatorial tests. Wouldn't it be cool if you could supply two sets of data and get a test for all the combinations of data? It is in MbUnit.

I originally picked up MbUnit because of its [RowTest ....] functionality, and I haven't found a single reason to go back. I moved all my active test suites over from NUnit and never looked back. Since then I've converted two different development teams over to the benefits.

6

As far as I know, there are four frameworks available for unit testing with .NET these days:

NUnit has always been out in front, but the gap has closed in the last year or so. I still prefer NUnit myself, especially as they added a fluent interface a while back which makes tests very readable.

If you're just getting started with unit testing it probably doesn't make much difference. Once you're up to speed, you'll be in a better position to judge which framework is best for your needs.

6

I don't like the Visual Studio's built-in testing framework, because it forces you to create a separate project as opposed to having your tests as part of the project you're testing.

1
  • 3
    You can actually fool Visual Studio by manually editing the project files and adding in the ProjectTypeGuids values that it uses as to identify test projects: <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>
    – Paul Ruane
    Aug 3, 2010 at 16:09
5

MSTest is essentially NUnit slightly reworked, with a few new features (such as assembly setup and teardown, not just fixture and test level), and missing some of the best bits (such as the new 2.4 constraint syntax). NUnit is more mature, and there is more support for it from other vendors; and of course since it's always been free (whereas MSTest only made it into the Professional version of Visual Studio 2008, and before that it was in way more expensive SKUs), and most ALT.NET projects use it.

Having said that, there are some companies who are incredibly reluctant to use something which does not have the Microsoft label on it, and especially so OSS code. So having an official Microsoft test framework may be the motivation that those companies need to get testing; and let's be honest, it's the testing that matters, not what tool you use (and using Tuomas Hietanen's code, you can almost make your test framework interchangeable).

4
  • I love NUnit's contraint syntax but I think you should read this regarding the SetUp and TearDown attributes: jamesnewkirk.typepad.com/posts/2007/09/why-you-should-.html
    – Nobody
    Dec 2, 2010 at 16:34
  • What is "SKU"? I suppose it is not Young Communist League of Sweden (Sveriges Kommunistiska Ungdomsförbund). Aug 23, 2020 at 19:39
  • @PeterMortensen it's a Stock Keeping Unit, apparently, for tracking inventory - inventory of software products in this case. So each version of Visual Studio has its own SKU - one for Professional, one for Enterprise. Sep 21, 2020 at 0:24
  • But maybe young Swedish communists like Microsoft products more than most people? Sep 21, 2020 at 0:25
4

With the release in .NET 4.0 of the Code Contracts system and the availability of a static checker, you would need to theoretically write fewer test cases and a tool like Pex will help identify those cases. Relating this to the discussion at hand, if you need to do less with your unit tests because your contracts are covering your tail, then why not just go ahead and use the built-in pieces since that is one less dependency to manage. These days, I am all about simplicity. :-)

See also:

3

I would prefer to use MS's little test framework, but for now am sticking with NUnit. The problems with MS's are generally (for me)

  • Shared "tests" file (pointless) that must be maintained
  • Tests lists cause conflicts with multiple developers / VCSs
  • Poor integrated UI - confusing setup, burdensome test selection
  • No good external runner

Caveats

  • If I were testing an aspx site, I would definitely use MS's
  • If I were developing solo, also MS would be fine
  • If I had limited skill and couldn't configure NUnit :)

I find it much easier to just write my tests and fire up NUnitGUI or one of the other front ends (testDriven is far far far far overpriced). Setting up debugging with the commandline version is also pretty easy.

1
  • Right now I'm using Resharper to run the MS Tests, so I don't mind them much. I prefer CodeRush+RefactorPro, though it's not what we use here. Maybe they have a decent runner for MS Test now too.
    – Andrew
    Apr 16, 2012 at 7:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.