Unit Test Frameworks in C#, Part Three: Data-Driven Tests

The first post in this series described the three major testing frameworks in .NET: MSTest, NUnit, and xUnit. The second post described how to get started and declare tests in each of them. Here I’ll be discussing data-driven tests.

Motivation

Suppose you are writing a function to determine the sum of two integers. Your function might have this signature:

public static int Sum(int x, int y)

You begin by writing a test.

2and2is4

Once it passes, you write the next.

2and3is5

And you need to make sure addition is commutative.

3and2is5

And don’t forget negative numbers! And a thousand other things. Soon you begin to wonder if there’s a better way than writing all these tests by hand. You might consider a solution like this:

inlinetests

That’s fine, but it has the issue that–in a less contrived and more complex example–you’re just running the same test logic over and over, duplicated each time. The DRY principle would probably result in something more like this:

drytests

This works well, until you look at the Test Explorer.

testexplorer

If the test fails, we will have to look at the stack trace to know which sum failed. We will not know how many of the cases failed or which they were; we will only get an error on the first one that failed and any that come after it will not even be run.

Enter data-driven tests. The idea here is that you can generate a test case with data. Let’s see how it works.

MSTest

Data-driven tests are supported by MSTest, but jumping through the necessary hoops is rather painful, especially if the tests are shared with other developers. As such, this will be covered in Part Five of this series, which will discuss a variety of more advanced concepts. Besides this, data-driven testing is about as close to an “advanced feature” as MSTest has, so it’s appropriate to cover it there I think.

NUnit

You can write the same test situation in NUnit like this.

nunittestcase

Or this.

nunitexpected

If the logic is a bit more complicated than what you want to express inline, you could delegate it to a class member, which can be a field, property, or parameterless method.

testcasesource

A similar outcome could be achieved by specifying each parameter’s values independently.

nunitvalues

When specifying parameters independently, it is important to know how they will be combined. In this case, that is specified with the [Sequential] attribute. This makes the test run use the values in order: 2, 2, 4, then 2, 3, 5, then 3, 2, 5.

Instead, we could omit [Sequential] and specify the optional [Combinatorial] attribute. This uses the Cartesian product of the sets. So then it would run 2, 2, 4, then 2, 2, 5, then 2, 3, 4, then 2, 3, 5, then 3, 2, 5, then 3, 3, 4, then 3, 3, 5. There is also a [Pairwise] attribute that can be used in place of [Sequential] and [Combinatorial].

Also, the Values parameter attribute can be replaced with Range or Random.

Similarly to the [TestCaseSource] above, it is possible to specify a member which can supply the data for a given parameter.

nunitvaluesource

The YThings and SumThings members would be defined analagously.

I will cover NUnit theories and further extensibility in Part Five.

Before we move on, though, there is one more NUnit feature worth covering. You can instantiate the same test fixture class multiple times and provide constructor arguments like so.

testfixture

You can even use generic fixtures this way.

genericfixtures

If the type arguments can be deduced from the parameters, you can even omit them.

genericfixturesopen

If the type arguments cannot be deduced and may conflict with arguments to be passed to the constructor itself, you can always distinguish them explicitly.

genericfixturestypeargs

This can be useful if, for example, you are writing a contract test. Say you want to make sure that all of your implementations of ICloneable return an object of the same type and have all properties set to the same values. You can write those tests easily enough, but constructing one instance of each type of ICloneable is a little hairy. But with a generic test fixture, you can just add another [TestFixture] attribute to indicate your new implementation and now you have a new fixture which can work with that type.

xUnit

Where a parameterless xUnit test is a [Fact], a parameterized one is a [Theory]. This is in fact enforced at compile time. A test like this

[Theory]public static void SumsAreCorrect(int x, int y, int sum) ...

will cause xUnit to give a compile-time error, saying “Theory methods must have test data”. There are three ordinary ways to provide such data.

The first is the [InlineData] attribute.

xunitinline

The second is the [MemberData] attribute.

xunitmemberdata

The compiler warning on the name is actually telling me to use the C# 6 nameof operator to reference the SumCases member.

Moreover, xUnit makes it easy to extend. Let us return to our discussion of tests to make sure that all implementations of ICloneable satisfy some contracts. All you have to do is derive from DataAttribute, say, like this:

xunitmydata

Then it can be used like so.

xunitmydatatest

The output from the test explorer would look like this:

xunitcloneableoutput

Summary

We now have data-driven tests in NUnit and xUnit. Apologies to those expecting MSTest; it will come later, I promise.

The NUnit section was much longer than the xUnit section, and this is indicative of the philosophies of the two frameworks. NUnit is meant for tests, while xUnit is meant for TDD. If you need to write complex scenarios that mix and match test parameters in strange ways, if you need generic test fixtures and all kinds of extensibility patterns, perhaps NUnit is better suited to your needs. If you are writing TDD, then you are writing concrete specifications and implementing concrete behaviors of your SUT one at a time, and the extensibility of xUnit’s [DataAttribute] is likely more than enough.

In Part Four, we will look at assertion frameworks.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s