The first post in this series described the three major testing frameworks in .NET: MSTest, NUnit, and xUnit. The second post described how to get started and declare tests in each of them. Here I’ll be discussing data-driven tests.
Motivation
Suppose you are writing a function to determine the sum of two integers. Your function might have this signature:
public static int Sum(int x, int y)
You begin by writing a test.
Once it passes, you write the next.
And you need to make sure addition is commutative.
And don’t forget negative numbers! And a thousand other things. Soon you begin to wonder if there’s a better way than writing all these tests by hand. You might consider a solution like this:
That’s fine, but it has the issue that–in a less contrived and more complex example–you’re just running the same test logic over and over, duplicated each time. The DRY principle would probably result in something more like this:
This works well, until you look at the Test Explorer.
If the test fails, we will have to look at the stack trace to know which sum failed. We will not know how many of the cases failed or which they were; we will only get an error on the first one that failed and any that come after it will not even be run.
Enter data-driven tests. The idea here is that you can generate a test case with data. Let’s see how it works.
MSTest
Data-driven tests are supported by MSTest, but jumping through the necessary hoops is rather painful, especially if the tests are shared with other developers. As such, this will be covered in Part Five of this series, which will discuss a variety of more advanced concepts. Besides this, data-driven testing is about as close to an “advanced feature” as MSTest has, so it’s appropriate to cover it there I think.
NUnit
You can write the same test situation in NUnit like this.
Or this.
If the logic is a bit more complicated than what you want to express inline, you could delegate it to a class member, which can be a field, property, or parameterless method.
A similar outcome could be achieved by specifying each parameter’s values independently.
When specifying parameters independently, it is important to know how they will be combined. In this case, that is specified with the [Sequential] attribute. This makes the test run use the values in order: 2, 2, 4, then 2, 3, 5, then 3, 2, 5.
Instead, we could omit [Sequential] and specify the optional [Combinatorial] attribute. This uses the Cartesian product of the sets. So then it would run 2, 2, 4, then 2, 2, 5, then 2, 3, 4, then 2, 3, 5, then 3, 2, 5, then 3, 3, 4, then 3, 3, 5. There is also a [Pairwise] attribute that can be used in place of [Sequential] and [Combinatorial].
Also, the Values parameter attribute can be replaced with Range or Random.
Similarly to the [TestCaseSource] above, it is possible to specify a member which can supply the data for a given parameter.
The YThings and SumThings members would be defined analagously.
I will cover NUnit theories and further extensibility in Part Five.
Before we move on, though, there is one more NUnit feature worth covering. You can instantiate the same test fixture class multiple times and provide constructor arguments like so.
You can even use generic fixtures this way.
If the type arguments can be deduced from the parameters, you can even omit them.
If the type arguments cannot be deduced and may conflict with arguments to be passed to the constructor itself, you can always distinguish them explicitly.
This can be useful if, for example, you are writing a contract test. Say you want to make sure that all of your implementations of ICloneable return an object of the same type and have all properties set to the same values. You can write those tests easily enough, but constructing one instance of each type of ICloneable is a little hairy. But with a generic test fixture, you can just add another [TestFixture] attribute to indicate your new implementation and now you have a new fixture which can work with that type.
xUnit
Where a parameterless xUnit test is a [Fact], a parameterized one is a [Theory]. This is in fact enforced at compile time. A test like this
[Theory]public static void SumsAreCorrect(int x, int y, int sum) ...
will cause xUnit to give a compile-time error, saying “Theory methods must have test data”. There are three ordinary ways to provide such data.
The first is the [InlineData] attribute.
The second is the [MemberData] attribute.
The compiler warning on the name is actually telling me to use the C# 6 nameof operator to reference the SumCases member.
Moreover, xUnit makes it easy to extend. Let us return to our discussion of tests to make sure that all implementations of ICloneable satisfy some contracts. All you have to do is derive from DataAttribute, say, like this:
Then it can be used like so.
The output from the test explorer would look like this:
Summary
We now have data-driven tests in NUnit and xUnit. Apologies to those expecting MSTest; it will come later, I promise.
The NUnit section was much longer than the xUnit section, and this is indicative of the philosophies of the two frameworks. NUnit is meant for tests, while xUnit is meant for TDD. If you need to write complex scenarios that mix and match test parameters in strange ways, if you need generic test fixtures and all kinds of extensibility patterns, perhaps NUnit is better suited to your needs. If you are writing TDD, then you are writing concrete specifications and implementing concrete behaviors of your SUT one at a time, and the extensibility of xUnit’s [DataAttribute] is likely more than enough.
In Part Four, we will look at assertion frameworks.
The Developing Coder 