+972 52-548-6969
The Record/Replay/Verify model
Daniel and I are having an interesting discussion about mock frameworks, and he just posted this: What's wrong with the Record/Reply/Verify model for mocking frameworks.
Daniel also pulled this quote from the Rhino Mocks documentation:
Record & Replay model - a model that allows for recording actions on a mock object and then replaying and verifying them. All mocking frameworks uses this model. Some (NMock, TypeMock.Net, NMock2) use it implicitly and some (EasyMock.Net, Rhino Mocks) use it explicitly.
Daniel go on to say:
I find this record/replay/verify model somewhat unnatural.
I suggest that you would read the entire post, since this is a response to that. I just want to point out that I still hold this view. Mock frameworks all use this model, because verification is a core part of mocking.
The problem that I have with what Daniel is saying is that we seem to have a fundamental difference in opinion about what mocking is. What Daniel calls mocks I would term stubs. Indeed, Daniel's library, by design, is not a mock framework library. It is a framework for providing stubs.
Going a little bit deeper than that, it seems that Daniel is mostly thinking about mocks in tests as a way to make the test pass. I am thinking of those in terms of testing the interactions of an object with its collaborators.
This project does a good job of showing off how I think mocking should be used, and it represent the accumulation of several years of knowledge and experience in testing and using mocks. It also represent several spectacular failures in using both (hint: mocking the database when going to production may not be a good idea), from which I learned quite a bit.
Mocking can be abused to cause hard to change tests, so can other methods. Deciding to throw the baby with the bath water seems to be a waste to me. There is a lot of guidance out there about correct application of mocking, including how to avoid the over specified tests. The first that comes to mind is Hibernating Rhinos #1 web cast, which talks about Rhino Mocks and its usage.
Rhino Mocks can be used in the way that Daniel is describing. I would consider this appropriate at certain times, but I think that to say that this is the way it should be is a mistake. The Rhino Mocks web cast should do a good job in not only showing some of the more interesting Rhino Mocks features, but also where to use them.
To conclude, the record, replay, verify model stands at the core of mocking. You specify what you think that should happen, you execute the code under test and then you verify that the expected happened. Taking this to the limit would produce tests that are hard to work with, but I am in the opinion that taking a worst practice and starting to apply conclusions from that is not a good idea.
Comments
Now this looks like a job for me...
I'm not that into mocking. The thing that interests me is testing. I'm not that interested in testing interactions, since they tend to change a lot. What I am interested in testing is the state changes an object goes through as a result of methods/events that it gets sent.
Maybe that puts me in the "stubs" camp, but I don't care. After doing unit testing and TDD on several large-scale projects (you know what I mean by large), I've found that state-based testing has the highest return on investment. I've also found that it's the easiest for new developers to get up to speed with - but I grant that my colored opinion probably influenced that.
A year and a half ago I started formalizing this process, as I posted here:
http://udidahan.weblogs.us/2006/05/06/behavior-specification-the-next-generation-of-unit-testing/
In which the state-based tests can serve as part of the contract of a class - complementing the static definition found in an interface.
Again, if interfaces are a core part of your design methodology (as they are in mine), you'll find this style quite natural - and the Design-by-Contract nature of TDD will become apparent.
But that's just my 10 cents - my 2c are free.
@ Ayende
When I try to view the Hibernating Rhinos video I get this error:
500 An error occured:ComponentActivator: could not proxy Cuyahoga.Modules.Downloads.DownloadsModule
@Udi
Yeah I pretty much agree, the main reasons I prefer state testing are:
1) They tend to be good examples.
2) They are less prone to break when you change some encapsulated implementation detail.
3) I find them useful when driving my (domain) design using TDD.
So overall I'm interested in the externally observable behavior. Not to say interaction testing isn't sometimes useful, particularly between layers.
I have not been using Mocks as of yet and am not fully convinced they are significantly useful. I understand that they can be used to isolate specific portions of code so you can do a true unit test BUT at what point to you stop isolating the test and get the full picture?
In production none of these pieces of code are run in isolation and the combined execution is what you are working with (database and all). If you never have unit tests that run through the full layered application, you will potentially miss interaction bugs that only appear through the combination of interactions rather than a single unit.
Do people have a method to deal with this such as a testing flag to indicate use mocks vs interactions?
I am still on the fence as to whether they provide enough value to the client to invest the time learning. I looked at the sample in quoted project above (Binsor.Presentation.Framework.Tests - ApplicationContextFixture) and in this I just don't see what value it provides other than ensuring that a developer does not mistakenly change the order of or remove an initializer. I just don't see any real value in this and could add code comments to the DefaultApplicationContext.Start method to ensure that developers are aware of the dependencies. If there is a design change that requires these to change then you have to change the test that appears only to remind the developer of a dependency.
Don't get me wrong, I do see some value in isolating tests so you can ensure you test the code you want but in the end you really need to test the code and what it relies on in one unit to ensure integration works for the unit like it will in production.
Thoughts?
Fred,
Those tests are there to assert the contract of the system.
They are also very fast tests, able to run during the dev build process.
Having integration tests is important, absolutely. But they are usually much longer and not as well focused.
@Ayende
I was wondering how you mix the contract, interaction and state tests for different parts of your system and in particular for your domain/service layers.
I've never been clear on the value of interaction testing for domain layers but some BDD seems to emphasize its use as a design technique. I guess you could use it to show the high level collaborations of some entities/services but I'm skeptical about how useful it is when you get into implementation details (such as an encapsulated object that an entity uses when doing some complex logic).
IMO both state and interaction based testing are useful tools. I tend to use whichever 'fits' best for the test I want to write.
With a strict TDD approach I try to write the very simplest code to make a test pass, often hard coding a return rather than interacting with a collaborator. Until I can write a test that requires me to perform the interaction I will avoid it, if this test can most easily be expressed as a state test then I will usually default to this, if it is easier (or more logical) to write this test using an interaction test I'll do that.
It seems a shame to me that effective mocking takes time (and getting burnt) to learn properly. I recently switched to using dynamic mocks as my default (rather than the standard mock) as it tends to encourage me to only care about interaction testing when the interaction is important. It would be good if the large community using Rhino Mocks (including myself) could collaborate on best practice for its use, probably on the wiki.
It turns out to be an implementation detail (one I'm not too fond of, you can tell ;))
See http://www.clariusconsulting.net/blogs/kzu/archive/2008/02/02/NewMoqfeaturesformockverificationandcreation.aspx for more info.
Comment preview