Unit Test Frameworks in C#, Part Four: Basic Assertions

You want to write a test for your code. The IsPrime function should return true when given an input of 2. There’s just one problem: C# doesn’t offer a keyword that says “this thing should happen”. So how will you record the fact that the test should fail or not depending on some condition?

In C# unit testing frameworks, the answer is always the same: the test passes if it runs without throwing an exception, and the test fails if it did throw an exception. As such, you want some logic that looks like this: if (!IsPrime(2)) throw new Exception();. This is pretty awkward, though, and it throws a generic exception with no custom message. Generating the message starts to feel like boilerplate after…well, the first one you try to roll on your own. Therefore, there are assertion frameworks to help with this task. They can even be mixed and matched, because any exception getting thrown will cause the test to fail. (this is not technically true in the case where you want to assert that some piece of code does throw a certain exception, but I’ll get to that later)

There are, to my knowledge, five assertion frameworks worth knowing about in C#. I’ll go through each of them in some detail here and in the next post. The MSTest, NUnit, and xUnit.net assertion frameworks are included with their respective test frameworks. However, there are two other assertion frameworks that deserve mention: FluentAssertions and Shouldly. These two are similar, but they have key differences.

I’ll evaluate each of the frameworks on four axes: Ease of Use, Error Messages, Breadth of Assertions, and Extensibility. I’ll cover the first two and part of the third in this post, and the rest in the next.

However, I’ll start by walking through some of the basic types of assertions so as to build your familiarity with the styles of each.

To browse my findings more thoroughly, check out this example project.

Asserting on Identity

A few things are worthy of note in this category.

The double, bool, float, DateTime, and TimeSpan primitives may be nullable except in MSTest and Shouldly.
Xunit will not accept nullable DateTime but will accept the nullable versions of the others.
MSTest does not have approximate equality comparisons for DateTime or TimeSpan.
Approximate equality assertion works on double as well as float on every framework, but it also works with decimal in every framework except MSTest.
If you want to assert that two values are not approximately equal, Shouldly can do it if the values are DateTime or TimeSpan but not if they are numeric.
MSTest does not even support approximate equality (let alone inequality) for dates or times.
FluentAssertions prefers to indicate approximate equality with a different method, called BeApproximately or BeCloseTo (depending on the type being compared), while Shouldly just uses an overload of Be.

For this segment, I considered 15 cases and tried to find an in-framework way to assert each one. Here are the results.

Assert equality: all frameworks pass.
Assert inequality: all frameworks pass.
Assert false: all frameworks pass.
Assert true: all frameworks pass.
Approximate equality of numbers: all frameworks pass.
Not approximate equality of numbers: only Shouldly fails.
Approximate equality of DateTime: only MSTest fails.
Not approximate equality of DateTime: only MSTest and XUnit fail.
Approximate equality of TimeSpan: only MSTest and XUnit fail.
Not approximate equality of TimeSpan: only MSTest and XUnit fail.
Reference equality: all frameworks pass.
Reference inequality: all frameworks pass.
Assert a reference is null: all frameworks pass.
Assert a reference is not null: all frameworks pass.
Assert.Fail: only MSTest and NUnit have this feature.

Based on this, I give the following ratings for breadth of assertions in the basic category. (the other categories will be considered later in a more holistic fashion)

NUnit 15/15
FluentAssertions 14/15
Shouldly 13/15
XUnit 11/15
MSTest 11/15

Asserting on Order

I considered six cases here. MSTest does not support any of them.

Assert a result is greater than a threshold: only XUnit fails.
Assert a result is greater than or equal to a threshold: only XUnit fails.
Assert a result is less than a threshold: only XUnit fails.
Assert a result is less than or equal to a threshold: only XUnit fails.
Assert a result is in a given range: all pass.
Assert a result is not in a given range: all pass.

Another note: only FluentAssertions does not support the use of a custom IComparer<T> for the ordering. That said, I give the following ratings for breadth of assertions in this section.

NUnit 6/6
FluentAssertions 6/6
Shouldly 6/6
XUnit 2/6
MSTest 0/6

Asserting on Type

I considered eight cases here. Since these are about checking whether an instance is of a particular type, it is natural to ask whether it is possible to make such an assertion and then get the value casted back out. Indeed it is, but not in MSTest or NUnit.

Assert an instance is exactly a given static type. Only MSTest fails.
Assert an instance’s type is exactly a given type reference. Only MSTest fails.
Assert an instance is assignable to a given static type. Only MSTest fails.
Assert an instance is assignable to a given type reference. All frameworks pass.
Assert an instance is not exactly a given static type. Only MSTest fails.
Assert an instance’s type is not exactly a given type reference. Only MSTest fails.
Assert an instance is not assignable to a given static type. Only MSTest and XUnit fail.
Assert an instance is not assignable to a given type reference. Only XUnit fails.

There is a minor note here: Fluent Assertion’s error message has a minor bug in the “instance is assignable to a given static type” case: iModel = act.Should().BeAssignableTo<IModel>().Which; generates the error Expected iModel = act to be assignable to IModel, instead of correctly parsing that act, not iModel = act, was the name of the variable under test.

As such, I give the following ratings for breadth of assertions in this section.

NUnit 8/8
FluentAssertions 8/8
Shouldly 8/8
XUnit 6/8
MSTest 2/8

Asserting on Exceptions

I considered eight cases here, which are generated by the Cartesian product of three boolean conditions: the delegate to be checked is synchronous or asynchronous, the thrown exception is either exactly the specified type or merely assignable to the specified type, and the specified type is a generic parameter or a type reference.

The resulting table would be a mess, so it might be easier to conceptualize in terms of what each framework supports.

FluentAssertions and MSTest only support using a generic parameter. A type reference is not permitted.
MSTest only checks that the exception is exactly of the given type; it has no mechanism for checking assignability.
NUnit checks for exact type equality with Throws.TypeOf and checks for assignability with Throws.InstanceOf. However, NUnit does not support asynchronous exception checking.
XUnit checks for exact type equality with Throws and for assignability with ThrowsAny.
FluentAssertions checks for exact type equality with ThrowsExactly and checks for assignability with Throws.
Shouldly checks for assignability on methods with the generic type parameter and for exact type equality in methods that take a type reference. In other words, there is no way to specify your preferred behavior.

Also, Shouldly has a problem with its error message in the asynchronous case. The details of the test cases follow.

Assert a synchronous delegate throws an exception exactly of a static type. Only Shouldly fails.
Assert a synchronous delegate throws an exception exactly of a type reference. Only MSTest and Fluent fail.
Assert a synchronous delegate throws an exception assignable to a static type. Only MSTest fails.
Assert a synchronous delegate throws an exception assignable to a type reference. Only NUnit passes.
Assert an asynchronous delegate throws an exception exactly of a static type. Only NUnit and Shouldly fail.
Assert an asynchronous delegate throws an exception exactly of a type reference. Only MSTest and Fluent fail.
Assert an asynchronous delegate throws an exception assignable to a static type. Only MSTest and NUnit fail.
Assert an asynchronous delegate throws an exception assignable to a type reference. All frameworks fail.

These are the exception scores:

XUnit 6/8
Fluent 4/8
Shouldly 4/8
NUnit 4/8
MSTest 2/8

We have one more type of test to consider in this post.

Assert on String Comparisons

I considered 18 cases here.

String contains. All pass.
String not contains. Only MSTest fails.
String starts with. All pass.
String not starts with. Only MSTest and XUnit fail.
String ends with. All pass.
String not ends with. Only MSTest and XUnit fail.
String matches regex pattern. Only MSTest fails.
String not matches regex pattern. Only MSTest fails.
String matches regex object. Only MSTest and XUnit pass.
String not matches regex object. Only MSTest and XUnit pass.
String contains case insensitive. Only MSTest fails.
String not contains case insensitive. Only MSTest fails.
String starts with case insensitive. Only MSTest fails.
String not starts with case insensitive. Only MSTest and XUnit fail.
String ends with case insensitive. Only MSTest fails.
String not ends with case insensitive. Only MSTest and XUnit fail.
String equality case insensitive. Only MSTest fails.
String inequality case insensitive. Only NUnit and XUnit pass.

A few notes follow.

The default comparison is case-sensitive in all frameworks except Shouldly. ShouldBe defaults to case-sensitive, but the StartsWith, EndsWith and the like default to case-insensitive.
MSTest and XUnit support most positive assertions but very few negative assertions.
Shouldly’s errors do not mention the case-insensitivity of the comparison on ShouldStartWith, ShouldNotStartWith, ShouldEndWith, andShouldNotEndWith.

The scores are, therefore:

NUnit: 16/18
FluentAssertions: 15/18
Shouldly: 15/18
XUnit: 14/18
MSTest: 5/18

Error Messages

MSTest gets points for taking format string arguments in every single one of its method overloads. XUnit loses points for not taking a user-defined message of any type in any assert other than Assert.True and Assert.False.

Shouldly and FluentAssertions both have a useful feature: the exception thrown will contain the text of the variable or expression upon which the extension method was called. So, for example, in Fluent, if the assertion actualValue.Should().Be(expectedValue); fails, the message will be “Expected actualValue to be 3, but found 4.” or the like. Note the name of the variable in your caller code, actualValue, is copied to the exception message. This is very powerful and allows much easier and quicker debugging. In case the reader is curious, both frameworks achieve this by generating a stack trace, then loading the text of your file from disk using the path it finds in the stack trace.

Shouldly’s messages are the most useful, despite the bug in async exception handling. Fluent’s are also nice, but it attempts to weave the entire message into a single line of English prose, which requires the user to parse a sentence instead of being easily able to see the interesting quantities at a glance.

FluentAssertions has another nice feature–it attempts to detect the test framework you are running and to throw the exception type that framework is expecting. So, for example, if it detects NUnit, it will throw NUnit.Framework.AssertionException. This is because each test framework detects its own assertion exception type and displays the message for it a little more cleanly than it displays a vanilla exception. The framework is even configurable. However, while the feature seems like a nice touch, I have never found myself missing it in Shouldly, so I do not award extra points for it. My Error Message scores stand as follows.

Shouldly: 9/10
Fluent: 8/10
MSTest: 6/10
NUnit: 5/10
XUnit: 3/10

Ease of Use

NUnit is the loser here. When your framework forces you to write something like Is.Not.EqualTo(3.14).Within(1.5).Percent and that’s just one argument in your method call, most people will long since have done the calculations themselves and asserted on a boolean rather than try to decipher your syntax.

MSTest loses a small amount of points for having separate StringAssert and CollectionAssert classes that most people are probably not even aware of. Don’t make me hunt far and wide for basic functionality, and if you do make me hunt for it, I had better find a gold mine when I get there, not the bare minimum.

XUnit, despite not supporting the breadth of assertions that I would like, at least wins points for having everything under a single Assert class, though it seems to value terseness at the expense of capturing useful information.

Shouldly is similar to Fluent here, but loses points for its biggest sin: cluttering up the intellisense on every single object you want to dereference. Fluent avoids this by having a single Should() extension and making its various assertions extend off of that.

Fluent: 10/10
Shouldly: 8/10
XUnit: 6/10
MSTest: 4/10
NUnit: 2/10

Summary

I’ll cover extensibility and finish up breadth of assertions next time, including assertions that are unique to each framework, plus asserting on collections. The scores so far stand thusly.

Fluent: 88.5%
Shouldly: 84.5%
Xunit: 53.6%
NUnit: 53.0%
MSTest: 45.5%