It’s been over ten years since Kent Beck created SUnit, the first unit testing framework for SmallTalk. Since then, hundreds of unit testing frameworks have been created in its image, one for nearly every platform and programming language. JUnit, the open-source Java unit testing framework, begat NUnit, the popular .NET port, which inspired Visual Studio Team Test, which was ported to the Silverlight Unit Testing Framework, and it was good.
For the most part NUnit’s and VSTT’s API’s are the same as those of JUnit. This is probably a good thing because it makes it much easier for Java developers to adopt .NET. It was also a natural decision given how similar C# and Java were when NUnit was created. However .NET programming languages have evolved rapidly in the last few years. Have you ever stopped to wonder whether the API’s for the .NET unit testing frameworks should still be so similar to those of JUnit?
Nowadays C# and VB.NET are capable of some very powerful code transformations. Despite the fact that .NET developers are conditioned to think of API’s and programming languages as separate the reality is not so simple. The capabilities of programming languages tend to influence the design of API’s. The question is this: Can we use the advanced features of .NET programming languages to create better unit testing API’s?
“What’s wrong with Unit Testing API’s? They’re pretty straightforward.”
There are three things wrong with the assertion API’s used by the major unit testing frameworks:
1. You Can’t Use Expressions
Let’s take a look at a typical Visual Studio Team Test.
[TestMethod] public void TestStackIsEmptyWhenCreated() { Stack emptyStack = new Stack(); Assert.AreEqual(emptyStack.Count, 0, "Stack is not empty."); }
Ask yourself this question: Is the AreEqual method really necessary? Our programming languages already have a rich set of comparison operators. Most of us are accustomed to writing debug assertions this way:
Debug.Assert(emptyStack.Count == 0, "Stack is not empty.");
Operators make common operations more recognizable to the human eye. If comparison operators are easier to read than the Assert methods why do we use the latter in our unit tests?
The reason is that we want descriptive failure messages in the event our test fails. If we fail the test above by adding an item to the stack we get a failure message much like this:
TestStackIsEmptyWhenCreated failed. Values are not equal. Expected 0, got 1. Stack is not empty.
Notice that in addition to including what type of comparison operation that failed the framework also provides the values being tested. If we were to specify an expression (as in the Debug.Assert example above) it would get compiled into executable code and by the time the UT framework got our expression it would be a black box. As a result it would be impossible to determine what kind of comparison we were attempting or the values of the arguments. Therefore Assert methods are necessary, despite the fact that they are ugly.
2. Assert.AreEqual vs. Assert.AreSame
Another problem caused by the inability to use expressions is that we must be explicit about whether we want to use value or reference comparisons. Have you ever used Assert.AreEqual when you should’ve used Assert.AreSame or vice versa? In some circumstances this can lead to false positives or negatives. It is especially annoying for C# and F# developers as we are used to our compilers choosing the appropriate comparison type for us based on context.
3. Redundant Assertion Messages
It is generally considered good practice to add a custom assertion message to provide more context in the event a test fails. This is especially important if a test makes multiple calls to the same Assert method. Let’s say we left the assertion message out of our unit test:
[TestMethod] public void TestStackIsEmptyWhenCreated() { Stack emptyStack = new Stack(); Assert.AreEqual(emptyStack.Count, 0); }
The resulting message would be:
TestStackIsEmptyWhenCreated failed. Values are not equal. Expected 0, got 1.
Of course this is a very simple test so you could probably infer what went wrong. It is a little cryptic though. In a larger test it would be much more difficult to isolate the issue.
So what’s wrong with assertion messages? They’re superfluous. Ask yourself this question: If you didn’t have to specify a message for the Unit Test Framework to display on failure, would you add that message as a comment above the line of code?
// Testing whether stack is empty Assert.AreEqual(emptyStack.Count, 0);
The answer is usually no. Assuming you’ve properly named your variables it is rarely necessary to comment assertions. In an ideal world our Unit Testing Framework would output the line of code that failed, just as our compilers do when they encounter syntax errors. This would make most assertion messages redundant.
The Root Problem
All of these issues are caused by the same root problem. Conceptually Unit Testing frameworks are an extension of your compiler. Compilers report static errors such as malformed code and type mismatches and unit testing frameworks report run-time errors such as unexpected values. The reason why Unit Testing API’s are so clunky is that our Unit Testing frameworks don’t have access to as much information as our compiler does. Compilers have access to the expression trees generated by our code. They can analyze the code and determine what type of comparison we are attempting, and whether it is a value or reference comparison. If the compiler encounters an error it can print the exact line of code or expression responsible.
What if there was a way to make our Unit Testing Frameworks as smart as our compilers?
Introducing Test.Assert()
Turns out it is possible to express assertions as expressions and get all the feedback provided by the Assert methods when a test fails (and even more).
Assertions as Expressions
Let’s rewrite our unit test using the Test.Assert() method:
using Unfold.Testing.VSTT; // snip... [TestMethod] public void TestStackIsEmptyWhenCreated() { Stack emptyStack = new Stack(); emptyStack.Push("an item that shouldn't be here."); Test.Assert(() => emptyStack.Count == 0); }
When we run this test and it fails we’ll get a message resembling this:
TestStackIsEmptyWhenCreated failed. Values are not equal. Expected 0, got 1. emptyStack.Count = 0
Notice that we get the same failure message as Assert.AreEqual in addition to the expression responsible for the failure!
Compound Expressions
Now that we can express assertions as expressions you may be wondering whether it is possible to express several assertions in a single compound expression. With Test Extensions it’s as easy as using the “and” operator.
Test.Assert(() => customer.Name != null && customer.Name != “”);
Now our assertions are much easier to read as well as shorter!
“Hold on. Won’t using compound expressions make it harder to figure out which expression failed?”
No. Test.Assert doesn’t display the line of code that fails, it displays the expression that fails. For example if, in the test above, the customer’s name was an empty string we’d get the following failure messages:
TestCustomerDefaultPropertyValues failed. Values are same. Expected not “”, got “”. customer.Name != “”
“Okay, but what if I still want to specify a custom assertion message?”
No problem. Test Assert has an overload for that:
Test.Assert(() => customer.Name != null && customer.Name != “”, "Customer name is null or empty.");
“Is it available for my Unit Testing Framework?”
Yes – assuming you’re using NUnit or VSTT :-). I’m making the source available under MS-PL so you’re free to port it to your unit testing framework of choice. You’ll find the port is trivial if your unit testing framework uses the familiar Assert methods.
“What about the Silverlight Unit Testing Framework?”
Don’t worry, I haven’t forgotten about Silverlight developers :-). In fact Test.Assert() has been integrated into the Silverlight Unit Testing Framework and will ship with the next version.
“How does it work?”
Test.Assert() improves your Unit Testing Framework by giving it access to the same information your compiler has: the expression tree. Test.Assert is similar to Linq to SQL. It converts your assertion expression into data, analyzes it, and then invokes the appropriate Assert methods.
Like Linq to SQL, Test.Assert() only supports a subset of the functionality provided by the underlying API. For certain assertions, particularly those that don’t use operators, you will still need to use the Assert methods.
“Which Assert Methods are supported?”
Here’s a table that shows which expressions map to which Test.Assert methods.
|
Expression |
NUnit |
VSTT/Silverlight Unit Testing Framework |
| value == true | Assert.IsTrue | Assert.IsTrue |
| value == false | Assert.IsFalse | Assert.IsFalse |
| value == 2 | Assert.AreEqual | Assert.AreEqual |
| value != 2 | Assert.AreNotEqual | Assert.AreNotEqual |
| value == “Jim” | Assert.AreSame | Assert.AreSame |
| value != “John” | Assert.AreNotSame | Assert.AreNotSame |
| “Jim” is string | Assert.IsInstanceOf | Assert.IsInstanceOfType |
| !(“Jim” is string) | Assert.IsNotInstanceOf | Assert.IsNotInstanceOfType |
| value < 2 | Assert.Less | N/A |
| value <= 2 | Assert.LessOrEqual | N/A |
| value > 2 | Assert.Greater | N/A |
| value >= 2 | Assert.GreaterOrEqual | N/A |
| double.IsNaN(value) | Assert.IsNaN | N/A |
| string.IsNullOrEmpty(“Jim”) | Assert.IsNullOrEmpty(“Jim”) | N/A |
| !string.IsNullOrEmpty(“Jim”) | Assert.IsNotNullOrEmpty(“Jim”) | N/A |
If Test.Assert doesn’t recognize any patterns in your expression it will fallback to the Assert.IsTrue method and pass it your entire expression. Often this - in addition including the expression in the failure message - is adequate. If you want to ensure a specific Assert method is used you can always call that method directly as you would have before.
“What about VB.NET support?”
The expression in the failure message is actually language agnostic and is the produced by calling the ToString() method of the System.Linq.Expressions.Expression object. Sometimes it looks like VB.NET, sometimes it looks like C#. There is no reason why you can’t use Test.Assert() in VB.NET.
“What about F# support?”
Although the F# compiler is not capable of converting code into Linq expression trees, F# does have it’s own expression tree format which can be produced using the quotations language syntax.
let expressionTree = <@@ value = “” @@>
You can convert F# expression trees to Linq expression trees using the F# Power Pack available with the latest version of F#. It should be trivial to write a wrapper method which does the conversion and forwards it to Test.Assert().
“Okay, okay. Where can I get it?”
You can download the source project (MS-PL) here. It includes the VSTT and NUnit versions. Test.Assert() will be integrated into the next release of the Silverlight Unit Testing Framework. Enjoy.
