This begins a series of posts in which I will compare the major testing frameworks available for unit testing (as opposed to UI testing frameworks like Selenium or Cucumber) available for the .NET Framework. The major players here are MSTest, NUnit, and Xunit. There are, of course, others, such as MSpec, but these three are definitely the biggest competitors on the market and for good reason.
Framework Philosophy
MSTest
The Visual Studio Unit Testing Framework is usually referred to by the name of its command line executable, MSTest. While I am not qualified to discuss what went on at Microsoft during the business and development decisions that led to MSTest, I think it is fair to say that it feels to the community like a product which is only supported begrudgingly. Microsoft’s own .NET projects (Roslyn, for example) use Xunit, not MSTest. It is difficult to find a discussion of MSTest on, say, StackOverflow without running into complaints about it (and it is not uncommon to find comments to the effect of “Upvoted for the complaints about MSTest”). MSTest was released in March 2005. The current version ships with Visual Studio 2017 Community Edition and is open source. Microsoft seems to be trying to improve it, but their attempts seem half-hearted at best.
NUnit
NUnit was around before MSTest. NUnit 2.2 released in August 2004. NUnit is free, open source, and time-tested. Its main goal (initially) was to be a friendly port of JUnit for .NET developers who were more experienced in Java.
xUnit.net
xUnit.net was written some time later by James Newkirk and Brad Wilson (the same folks who designed NUnit). Its purpose is to address what the authors thought of as the shortcomings of NUnit. In particular, its goal is to emphasize TDD/BDD and proper unit testing practices. For instance, the [Test] attribute from NUnit has been replaced in Xunit with [Fact]. It may seem at first glance that this is a pedantic change done just to be different, but it was done because a test in TDD is not a mechanism to prevent or catch bugs or regressions or defects–it is a specification or fact about the behavior of the system. Hence, it is called a [Fact] in Xunit. We will see throughout this series some of the ways that Xunit encourages TDD by its naming and restrictions.
I should also mention that when I refer to Xunit, I am referring specifically to xUnit.net, whereas Xunit is also the name of the test framework philosophy which underlies all of these testing frameworks. All of them are said to be “Xunit-style testing”, but I use the term xUnit specifically to refer to the xunit.net framework. If you are curious about Xunit-style testing or about the underlying philosophies of TDD and BDD, I suggest TDD by Kent Beck.
Installation and Use
MSTest: Excellent in Visual Studio, Poor Otherwise
For installation and getting up and running with a test project in Visual Studio, nothing will ever beat MSTest. This is because MSTest ships with Visual Studio itself and is a built-in feature. NUnit and Xunit and anything else you may invent is, at best, an add-on requiring a separate installation and download.
Its integration with Visual Studio is certainly its greatest strength. There is no need for other NuGet packages or frameworks or extensions or plugins. You simply go to File >> New Project >> Visual C# >> Test >> Unit Test Project (.NET Framework), and you’re up and running. The Test Explorer in Visual Studio supports MSTest tests natively. This makes it very easy to get started with MSTest.
However, its greatest strength is also its greatest weakness. Running MSTest from the command line requires either Visual Studio or MSBuild to be installed; there is no easy-to-get NuGet package to enable this. Also, MSTest does not always integrate nicely with third party GUI Test Explorers (like Resharper or Rider). It has, as of this time, been less than a year since Rider added support for MSTest, and even now the support is languid. As such, although you can start a new test project very quickly with MSTest in Visual Studio, the same cannot be said for other development environments. Therefore, most developers prefer to use NUnit or Xunit, as the extra installation and download are not burdensome, and they both come with excellent support in modern versions of Visual Studio that all but eliminate reasons to use MSTest in greenfield projects. Still, I will include MSTest in my comparisons in this series on account of its popularity in legacy projects.
NUnit/Xunit: Very Easy in All Frameworks
Xunit and NUnit are extremely similar here. You create a test project by adding a new Class Library project and then installing the appropriate NuGet package to that project.
> Install-Package nunit -Project MyProject.Tests
> Install-Package xunit -Project MyProject.Tests
The tests for both frameworks will show up by default in the Visual Studio Test Explorer (with or without Resharper). This was not always the case, so your mileage may vary on older versions of Visual Studio. However, if you are not using Resharper, even though they show up, they may not be able to run. For that, you will have to install another package:
> Install-Package NUnit3TestAdapter -Project MyProject.Tests
> Install-Package xunit.runner.visualstudio -Project MyProject.Tests
Note that having the NUnit3TestAdapter installed on any project in the solution is enough to be able to run NUnit tests from every project in the solution under the target framework. Similarly, having the xunit.runner.visualstudio package installed in a single project in the solution is enough to be able to run tests in any project in the entire solution.
Rider’s Test Explorer was designed to discover and run NUnit and xUnit tests automatically.
The tests can also be run from the command line by installing another package.
> Install-Package NUnit.Console -Project MyProject.Tests
> Install-Package xunit.runner.console
The console runner is installed to a specific project in the solution and can be viewed in that project’s packages.config file. You can find it by going to the solution and looking in the packages folder. You want either packages/NUnit.ConsoleRunner.3.8.0/tools/nunit3-console.exe or packages/xunit.runner.console.2.3.1/tools/net452/xunit.console.exe depending on which framework you are using. Running it with a command line of --help will display the options for running it, or the NUnit docs are here. Note that the xUnit version installs separate runners for .NET Framework and .NET Core. Also, older versions of xUnit installed the console runner to the solution instead of to a specific project.
Framework support
To get NUnit to run in a .NET Core project, you will also have to install another NuGet package.
> Install-Package Microsoft.NET.Test.Sdk -Project MyProject.Tests
No such addition is necessary for Xunit to run on .NET Core.
Neither can run on a class library compiled for .NET Standard. See the explanation here.
Mixed
It is, of course, possible to mix and match. There is no conflict in having a test from all three frameworks in the same class. For that matter, it is possible to mark the same method as a test in multiple frameworks. They will run naturally in the Test Explorer.
NUnit and xUnit tests can also be run as part of MSBuild. See here. Otherwise, a build server that needs to support multiple test frameworks will usually run them separately from the command line.
Summary
And that’s it! You’re now set up with a test project in the framework of your choice. The second installment in this series will show you how to start writing tests in your new test project. If that’s old hat to you, don’t lose hope–the Part Three will cover data-driven tests, Part Four will cover assertions in depth (including separate assertion frameworks like FluentAssertions), and Part Five will cover advanced features and extensibility points of all three frameworks. For more information on the plan for this and other series, refer to my schedule.
The Developing Coder 