Daniel has posted a reply to my post, titling it:  Are you smart enough to do without TDD. I more or less expected to get responses like that, which was why I was hesitant to  post it. Contrary to popular opinion, I don’t really enjoy being controversial.

There are two main points that I object to in his post:

You see, Ayende appears to say that if you're smart enough, you'll just know what code to write, just like that. Ergo, if you don't know, maybe you're not that smart and hence you would need this technique for losers called Test Driven Design/Development.

That is not what I said, please don’t put words in my mouth. What I said was: “The idea behind TDD is to use the tests to drive the design. Well, in this case, I don’t have any design to drive.” Combine this with my concepts & features architecture, where the main tenets is: “A feature creation may not involve any design activity.” and it should be clear why TDD simply doesn’t work for my scenario.

And his attack on Rhino Mocks:

Moq vs Rhino Mocks: he [Ayende, it seems] read the (useless IMO) literature on mocks vs stubs vs fakes, had apparently a clear idea of what to do, and came up with Rhino's awkward, user unfriendly and hard to learn API with a myriad of concepts and options, and a record-replay-driven API (ok, I'm sure it was not his original idea, but certainly it's his impl.) which two years ago seemed to him to stand at the core of mocking. Nowadays not only he learned what I've been saying all along, that "dynamic, strict, partial and stub... No one cares", but also is planning to remove the record / playback API too.

This is just full of misinformation. Let me see how:

• Rhino Mocks is 5 years old.
• Rhino Mocks came out for .NET 1.0.
• Rhino Mocks actually predate most of the mocks vs. stubs debate.

I keep Rhino Mocks updated as new concepts and syntax options comes. Yes, AAA is easier, but AAA relies on having the syntax options that we have in C# 3.0. Rhino Mocks didn’t start from there, it started a lot earlier, and it is a testament to its flexibility that I was able to adapt it to any change along the way.

Oh, and Rhino Mocks was developed with TDD, fully. Still is, for that matter. So I find it annoying that someone attacks it on this grounds without really understanding how it worked.

I originally titled this blog post: Separate the scenario under test from the asserts. I intentionally use the terminology scenario under test, instead of calling it class or method under test.

One of the main problem with unit testing is that we are torn between competing forces. One is the usual drive for abstraction and eradication of duplication, the second is clarity of the test itself. Karl Seguin does a good job covering that conflict.

I am dealing with the issue by the simple expedient of forbidding anything but asserts in the test method. And no, I don’t mean something like BDD, where the code under test is being setup in the constructor or the context initialization method.

I tend to divide my tests code into four distinct parts:

• Scenario under test
• Scenario executer
• Test model, represent the state of the application
• Test code itself, asserting the result of a specific scenario on the test model

The problem is that a single scenario in the application may very well have multiple things that we want to actually test. Let us take the example of authenticating a user, there are several things that happen during the process of authentication, such as the actual authentication, updating the last login date, resetting bad login attempts, updating usage statistics, etc.

I am going to write the code to test all of those scenarios first, and then discuss the roles of each item in the list. I think it will be clearer to discuss it when you have the code in front of you.

We will start with the scenarios:

public class LoginSuccessfully : IScenario
{
    public void Execute(ScenarioContext context)
    {
        context.Login("my-user","swordfish is a bad password");
    }
}

public class TryLoginWithBadPasswordTwice : IScenario
{
    public void Execute(ScenarioContext context)
    {
        context.Login("my-user","bad pass");
        context.Login("my-user","bad pass");
    }
}

public class TryLoginWithBadPasswordTwiceThenTryWithRealPassword : IScenario
{
    public void Execute(ScenarioContext context)
    {
        context.Login("my-user","bad pass");
        context.Login("my-user","bad pass");
        context.Login("my-user","swordfish is a bad password");
    }
}

And a few tests that would show the common usage:

public class AuthenticationTests : ScenarioTests
{
    [Fact]
    public void WillUpdateLoginDateOnSuccessfulLogin()
    {
        ExecuteScenario<LoginSuccessfully>();
        
        Assert.Equal(CurrentTime, model.CurrentUser.LastLogin);
    }
    

    [Fact]
    public void WillNotUpdateLoginDateOnFailedLogin()
    {
        ExecuteScenario<TryLoginWithBadPasswordTwice>();
        
        Assert.NotEqual(CurrentTime, model.CurrentUser.LastLogin);
    }

    [Fact]
    public void WillUpdateBadLoginCountOnFailedLogin()
    {
        ExecuteScenario<TryLoginWithBadPasswordTwice>();
        
        Assert.NotEqual(2, model.CurrentUser.BadLoginCount);
    }

    [Fact]
    public void CanSuccessfullyLoginAfterTwoFailedAttempts()
    {
        ExecuteScenario<TryLoginWithBadPasswordTwiceThenTryWithRealPassword>();
        
        Assert.True(model.CurrentUser.IsAuthenticated);
    }
}

As you can see, each of the tests is pretty short and to the point, there is a clear distinction between what we are testing and what is being tested.

Each scenario represent some action in the system which we want to verify behavior for. Those are usually written with the help of a scenario context (or something of the like) with gives the scenario access to the application services required to perform its work. An alternative to the scenario context is to use a container in the tests and supply the application service implementations from there.

The executer (ExecuteScenario<TScenario>() method) is responsible for setting the environment for the scenario, executing the scenario, and cleaning up afterward. It is also responsible for any updates necessary to get the test model up to date.

The test model represent the state of the application after the scenario was executed. It is meant for the tests to be able to assert against. In many cases, you can use the actual model from the application, but there are cases where you would want to augment that with test specific items, to allow easier testing.

And the tests, well, the tests simple execute a scenario and assert on the result.

By abstracting the execution of a scenario into the executer (which rarely change) and providing an easy way of building scenarios, you can get very rapid feedback into test cycle while maintaining testing at a high level.

Also, relating to my previous post, note what we are testing here isn’t a single class. We are testing the system behavior in a given scenario. Note also that we usually want to assert on various aspects of a single scenario as well (such as in the WillNotUpdateLoginDateOnFailedLogin and WillUpdateBadLoginCountOnFailedLogin tests).

Let us get a few things out of the way:

• I am not using TDD.
• I am not using BDD.
• I am not using Test After.
• I am not ignoring testing.

I considered not posting this post, because of the likely response, but it is something that I think it worth at least discussion. The event that made me decide to post this is the following bug:

public bool IsValid
{
    get { return string.IsNullOrEmpty(Url); }
}

As you can probably guess, I have an inverted conditional here. The real logic is that the filter is valid if the Url is not empty, not the other way around.

When I found the bug, I briefly considered writing a test for it, but it struck me as a bad decision. This is code that I don’t see any value in testing. It is too stupid to test, because I won’t have any ROI from the tests.  And yes, I am saying that after seeing that the first time I wrote the code it had a bug.

The idea behind TDD is to use the tests to drive the design. Well, in this case, I don’t have any design to drive. In recent years, I have moved away from the tenets of TDD toward a more system oriented testing system.

I don’t care about testing a specific class, I want to test the entire system as whole. I may switch some parts of the infrastructure (for example, change the DB to in memory one), for perf sake, but I usually try to test an entire component at a time.

My components may be as small as a single class or as big as the entire NH Prof sans the actual UI pieces.  I have posted in the past, showing how I implement features for NH Prof, including the full source code for the relevant sections. Please visit the link, it will probably make more sense to you afterward. It is usually faster, easier and more convenient to write a system test than to try to figure out how to write a unit test for the code.

Now, let us look at why people are writing tests:

• Higher quality code
• Safe from regressions
• Drive design

Well, as I said, I really like tests, but my method of designing software is no longer tied to a particular class. I have the design of the class handed to me by a higher authority (the concept), so that is out. Regressions are handled quite nicely using the tests that I do write.

What about the parts when I am doing design, when I am working on a new concept?

Well, there are two problems here:

• I usually try several things before I settle down on a final design. During this bit of churn, it is going to take longer to do things with tests.
• After I have a design finalized, it is still easier to write a system level test than write unit tests for the particular implementation.

As a matter of fact, in many cases, I don’t really care about the implementation details of a feature, I just want to know that the feature works. As a good example, let us take a look at this test:

public class CanGetDurationOfQueries : IntegrationTestBase
{
    [Fact]
    public void QueriesSpecifyTheirDuration()
    {
        ExecuteScenarioInDifferentAppDomain<SelectBlogByIdUsingCriteria>();
        var first = model.RecentStatements
            .ExcludeTransactions()
            .First();
        Assert.NotNull(first.DurationViewModel.Inner.Value);

    }
}

NH Prof went through three different ways of measuring the duration of a query. The test didn’t need to change. I have a lot of tests that work in the same manner. Specifying the final intent, rather than specifying each individual step.

There are some parts in which I would use Test First, usually parts that I have high degree of uncertainty about.  The “show rows from query” feature in NH Prof was develop using Test First, because I had absolutely no idea how to approach it.

But most of the time, I have a pretty good idea where I am and where I am going, and writing unit tests for every miniscule change is (for lack of a better phrase) hurting my style.

Just about any feature in NH Prof is covered in tests, and we are confident enough in our test coverage to release on every single commit.

But I think that even a test has got to justify its existence, and in many cases, I see people writing tests that have no real meaning. They duplicate the logic in a single class or method. But that isn’t what I usually care about. I don’t care about what a method or a class does.

I care about what the overall behavior is. And I shaped my tests to allow me to assert just that. I’ll admit that NH Prof is somewhat of a special case, since you have a more or less central location that you can navigate form which to everything else. In most systems, you don’t have something like that.

But the same principle remains, if you setup your test environment so you are testing the system, it is going to be much easier to test the system. It isn’t a circular argument. Let us take a simple example of an online shop and wanting to test the “email on order confirmed” feature.

One way of doing this would be to write a test saying that when the OrderConfirmed message arrive, a SendEmail message is sent. And another to verify that SendEmail message actually send an email.

I would rather write something like this, however:

[Fact]
public void WillSendEmailOnOrderConfirmation()
{
    // setup the system using an in memory bus
    // load all endpoints and activate them
    // execute the given scenario
    ExecuteScenario<BuyProductUsingValidCreditCard>();

    var confirmation = model.EmailSender.EmailsToSend.FirstOrDefault(x=>x.Subject.Contains("Order Confirmation");
    Assert.NotNull(confirmation);
}

I don’t care about implementation, I just care about what I want to assert.

But I think that I am getting side tracked to another subject, so I’ll stop here and post about separating asserts from their scenarios at another time.

I was just reviewing someone’s code, and as I was browsing through the tests, I run into the following anti pattern, just about all the asserts were specified as:

Assert.True(pet.Owner == "Baz");

The problem with that is that this is a perfect case of “let’s create a test that will tell us nothing when it is failing”.

It seems like a minor thing, to use the more explicit version:

Assert.Equal("Baz", pet.Owner);

But it isn’t, when this assert fails, we are going to get a lovely error message telling us exactly what went wrong. From maintainability standpoint, being able to see why a test failed without debugging it is not a plus, it is essential.

Assert.True is a tool of last resort, don’t use it if you have any choice at all.

With Rhino Queues, I have a very serious problem. While all the tests would run if they were run individually, running them in concrete would cause them to fail. That is a pretty common case of some test affecting the global state of the system (in fact, the problem is one test not releasing resources and all other tests failing because they tried to access a locked resource).

The problem is finding out which. By the way, note that I am making a big assumption here, that there is only one test that does so, but for the purposes of this discussion, it doesn’t matter, the technique works for both options.

At the heart of it, we have the following assumption:

If you have tests that cause failure, and you don’t know which, start by deleting tests.

In practice, it means:

1. Run all unit tests
2. If the unit tests failed:
• Delete a unit test file with all its tests
• Go to first step
3. If the unit tests passed:
• Revert deletion of a unit test file in reverse order to their deletion
• Go to step 1

When you have only a single file, you have pinpointed the problem.

In my case, unit tests started to consistently pass when I deleted all of those ( I took a shortcut and delete entire directories ):

So I re-introduced UsingSubQueues, and… the tests passed.

So I re-introduced Storage/CanUseQueue and… the tests passed.

This time, I re-introduced Errors, because I have a feeling that QueueIsAsync may cause that, and… the tests FAILED.

We have found the problem. Looking at the Errors file, it was clear that it wasn’t disposing resources cleanly, and that was a very simple fix.

The problem wasn’t the actual fix, the problem was isolating it.

With the fix… the tests passed. So I re-introduced all the other tests and run the full build again.

Everything worked, and I was on my way :-)

On one of my machines, I can consistently get the following test failure, because of an exception throw here:

As you can see, we get an ObjectDisposedException here, which cause the test to fail. There is only one problem with that:

We only dispose the queue manager on the dispose method, and that can only be called if the test itself was completed.

On the face of it, it looks like I am doing things correctly, but there is a bug here. The CLR’s threading semantics are not broken, even though on first glance it appears so.

As a side note, people keep asking me how I blog so much. This post is a perfect example. When I started writing this blog post, I was truly stamped. By the time I finished, I already noticed where the bug was and fixed it.

Can you tell?

Dror from TypeMock has managed to capture the essence of my post about unit tests vs. integration tests quite well:

Unit tests runs faster but integration tests are easier to write.

Unfortunately, he draws an incorrect conclusion out of that;

There is however another solution instead of declaring unit testing as hard to write - use an isolation framework which make writing unit test much easier.

And the answer to that is… no, you can’t do that. Doing something like that put you back in the onerous position of unit test, where you actually have to understand exactly what is going on and have to deal with that. With an integration test, you can assert directly on the end result, which is completely different than what I would have to do if I wanted to mock a part of the system. A typical integration test looks something like:

public class SelectNPlusOne : IScenario
{
    public void Execute(ISessionFactory factory)
    {
        using (var session = factory.OpenSession())
        using (var tx = session.BeginTransaction())
        {
            var blog = session.CreateCriteria(typeof(Blog))
                .SetMaxResults(1)
                .UniqueResult<Blog>();//1
            foreach (var post in blog.Posts)//2
            {
                Console.WriteLine(post.Comments.Count);// SELECT N
            }
            tx.Commit();
        }
    }
}


[Fact]
public void AllCallsToLoadCommentsAreIdentical()
{
    ExecuteScenarioInDifferentAppDomain<SelectNPlusOne>();
    var array = model.Sessions[0]
        .SessionStatements
        .ExcludeTransactions()
        .Skip(2)
        .ToArray();
    Assert.Equal(10, array.Length);
    foreach (StatementModel statementModel in array)
    {
        Assert.Equal(statementModel.RawSql, array[0].RawSql);
    }
}

This shows how we can assert on the actual end model of the system, without really trying to deal with what is actually going on. You cannot introducing any mocking to the mix without significantly hurting the clarity of the code.

I can sum it up pretty easily:

Unit tests:
65 passed, 0 failed, 5 skipped, took 7.49 seconds.

Integration tests:
45 passed, 0 failed, 0 skipped, took 284.43 seconds.

Except, there is one thing that you aren’t seeing here. Writing an integration tests is about as straightforward as they come. Execute the user scenario, assert the result. There is no real knowledge required to write the test, and they are very easy to read and understand.

Unit tests, however, often require you to understand how the software work in much finer detail, and they often make no sense to someone who isn’t familiar with the software.

When using NHibernate we generally want to test only three things, that properties are persisted, that cascade works as expected and that queries return the correct result. In order to do all of those, we generally have to talk to a real database, trying to fake any of those at this level is futile and going to be very complicated.

We can either use a standard RDBMS or use an in memory database such as SQLite in order to get very speedy tests.

I have a pretty big implementation of a base class for unit testing NHibernate in Rhino Commons, but that has so many features that I forget how to use it sometimes. Most of those features, by the way, are now null & void because we have NH Prof, and can easily see what is going on without resorting to the SQL Profiler.

At any rate, here is a very simple implementation of that base class, which gives us the ability to execute NHibernate tests in memory.

public class InMemoryDatabaseTest : IDisposable
{
	private static Configuration Configuration;
	private static ISessionFactory SessionFactory;
	protected ISession session;

	public InMemoryDatabaseTest(Assembly assemblyContainingMapping)
	{
		if (Configuration == null)
		{
			Configuration = new Configuration()
				.SetProperty(Environment.ReleaseConnections,"on_close")
				.SetProperty(Environment.Dialect, typeof (SQLiteDialect).AssemblyQualifiedName)
				.SetProperty(Environment.ConnectionDriver, typeof(SQLite20Driver).AssemblyQualifiedName)
				.SetProperty(Environment.ConnectionString, "data source=:memory:")
				.SetProperty(Environment.ProxyFactoryFactoryClass, typeof (ProxyFactoryFactory).AssemblyQualifiedName)
				.AddAssembly(assemblyContainingMapping);

			SessionFactory = Configuration.BuildSessionFactory();
		}

		session = SessionFactory.OpenSession();

		new SchemaExport(Configuration).Execute(true, true, false, true, session.Connection, Console.Out);
	}

	public void Dispose()
	{
		session.Dispose();
	}
}

This just set up the in memory database, the mappings, and create a session which we can now use. Here is how we use this base class:

public class BlogTestFixture : InMemoryDatabaseTest
{
	public BlogTestFixture() : base(typeof(Blog).Assembly)
	{
	}

	[Fact]
	public void CanSaveAndLoadBlog()
	{
		object id;

		using (var tx = session.BeginTransaction())
		{
			id = session.Save(new Blog
			{
				AllowsComments = true,
				CreatedAt = new DateTime(2000,1,1),
				Subtitle = "Hello",
				Title = "World",
			});

			tx.Commit();
		}

		session.Clear();


		using (var tx = session.BeginTransaction())
		{
			var blog = session.Get<Blog>(id);

			Assert.Equal(new DateTime(2000, 1, 1), blog.CreatedAt);
			Assert.Equal("Hello", blog.Subtitle);
			Assert.Equal("World", blog.Title);
			Assert.True(blog.AllowsComments);

			tx.Commit();
		}
	}
}

Pretty simple, ah?

To be truthful, I never thought that I would have a following for this post 4 years later, but I run into that today.

The following is a part of an integration test for NH Prof:

Assert.AreEqual(47, alerts[new StatementAlert(new NHProfDispatcher())
{
	Title = "SELECT N+1"
}]);

I am reviewing all our tests now, and I nearly choked on that one. I mean, who was stupid enough to write code like this?  I mean, yes, I can understand what it is doing, sort of, but only because I have a dawning sense of horror when looking at it.

I immediately decided that the miscreant that wrote that piece of code should be publically humiliated and  chewed on by a large dog.

SVN Blame is a wonderful thing, isn’t it?

Hm… there is a problem here.

Actually, there are a couple of problems here. One is that we have a pretty clear indication that we have a historic artifact here. Just look at the number of version that are shown in just this small blame window. This is good enough reason to start doing full fledged ancestory inspection. The test has started life as:

[TestFixture]
public class AggregatedAlerts:IntegrationTestBase
{
	[Test]
	public void Can_get_aggregated_alerts_from_model()
	{
		ExecuteScenarioInDifferentAppDomain<Scenarios.ExecutingTooManyQueries>();

		var alerts = observer.Model.Sessions[1].AggregatedAlerts;
		Assert.AreEqual(47, alerts["SELECT N+1"]);
		Assert.AreEqual(21, alerts["Too many database calls per session"]);
	}
}

Which I think is reasonable enough. Unfortunately, it looks like somewhere along the way, someone had taken the big hammer approach to this. The code now looks like this:

Assert.AreEqual(47, alerts.First(x => x.Key.Title == "SELECT N+1"));

Now this is readable.

Oh, for the nitpickers, using hash code evaluation as the basis of any sort of logic is wrong. That is the point of this post. It is a non obvious side affect that will byte* you in the ass.

* intentional misspelling