So someone raised a great point about my last post, noticing that it is usually hard to pinpoint the exact line of code where your error is when a test fails. Usually, you know which line in the test failed, but not why. That happens when you have integration tests, and not unit tests.
When I say "Pin point breakages and causes of failures to the line level", I actually do mean that. It is quite possible to do this with true unit tests. This comes back to the definition of a unit test. It should be testing one unit or method, and not testing things across classes. So if you want to isolate the cause of the breakage, we mock out anything that is not this class to truly test just this class.
Consider this example, you have a class House which has a few methods like enterHouse() and lockHouseAndLeave(). A house has some members like door, rooms, etc. When we test enterHouse(), we mock out the door and rooms, and assume those work perfectly (Since we test those individually as well for correctness). We then test just enterHouse(), and have the mocks return what we expect it should in the correct behavior. That way, if anything breaks in a test for enterHouse(), you cna be totally assured it is something because of enterHouse(), and rarely have to dig in deep.
Now if you have an integration test, where you don't have mocks for Doors and rooms, you are going to have trouble pinpointing the point of breakage, but with mocks, and proper unit testing, you know exactly what is being tested and what is causing the test to fail.
Though getting to this point where you can test pieces in isolation either requires you start designing with testability in mind or refactoring stuff to get it where you can inject mocks.
In my next post, I plan to talk about code coverage, one of the easiest ways to figure out what is untested and some caveats.
Saturday, July 12, 2008
Subscribe to:
Post Comments (Atom)
Posts
2 comments:
First of all, thanks for the answer on my blog and addressing my comment in a full blog post here. Let my structure this comment in two parts. First I will present example-driven testing, an alternative to mock-based testing. An alternative which I personally find more pragmatic. Ads second, I will show why we are still not at the level of source-line assertion failures.
1)
Given your House with a Kitchen, you propose to use a mock kitchen to test the house. However, setting up a mock can be quite some work. And why do that work if somewhere else we already do all the work of creating a working kitchen? Yes, if only there would be a return value at the end of the kitchen tests we could use that instead of the mock.
This is exactly what we do in JExample (a JUnit extension): we avoid all the mocks, because our test methods can have return values and dependency injections. Let me give an example
@Test
public Kitchen testKitchen() {
Kitchen k = new Kitchen;
// ... assertions
return k;
}
@Test
@Depends("testKitchen")
public void testHouse(Kitchen k) {
House h = new House(k);
// ... assertions
return h;
}
When executing testKitchen, the framework caches the return value: an example kitchen! When executing testHouse this example is injected as invokation argument in the test call.
If however testKitchen fails, the framework reports testKitchen as red and testHouse as white (ie failed due to dependency. Thus pin-pointing us to kitchen as the source of the failure!
Example injection is also very handy when testing different states of the same instance under test. You just inject the example from one test method to the next, thus testing all different states.
2)
Still, whether using mocks or examples, we are still not level of source-line assertion failures. Why? Even if you know the MUT (method under test) there is no bijective, ie 1:1, relation between methods and test-methods.
Given the method under test delegates to four private methods (ie 1:n), where is the bug the test fails?
Given two methods under test delegate to the same private method (ie n:1), where is the bug if only one fails?
EzUnit by Friedrich Steimann addresses these issues by analysing both the static and runtime coverage of tests and comparing it to each other.
But alas, their tool has no example injection :)
@akuhn: Thanks for the great reply. Example driven testing seems like an interesting approach, especially if you prefer to do TDD or EDD. But still, as you mentioned, the House test being shown as white due to a failed dependency means you wouldn't know if House works or not. There might be additional failures which you wouldn't catch until you fix the problem in Kitchen. With mocks on the other hand, Kitchen test fails, we know kitchen has a problem, while the House which uses a mock, and thus probably wouldn't be having the bug allows the House test to pass. Just my thought on the subject.
Well, yes, sadly we are not yet at the level of line level assertion failures. Thought generally, I try not to have too much same class method dependencies. Usually those kind of methods are private as well. And technically, you could mock out public methods with EasyMock, I believe.
Post a Comment