Coverage isn't everything

A few posts ago, I mentioned Emma and code coverage, and how it could be useful. Code coverage, in layman’s terms, is the lines of code covered by your tests. So generally, you want a high code coverage, implying that most of your code is well tested.

But since my post, I have talked with a few friends and colleagues, and one point has been brought up again and again. First, it was a discussion with my friend on his homework assignment. It was a hypothetical situation about a manager wanting 100 % code coverage before he launches his product. That discussion quickly devolved into a rant about why code coverage isn’t the alpha and the omega.

And then, just one or two days ago, I had another discussion with a colleague, who was telling / reminding me of the fact that a project having great code coverage doesn’t mean that they don’t need to test it anymore. I politely agreed and nodded my head, and both of us ended up repeating the same facts to each other. Which was that code coverage, like a lot of other things, can be faked.

The trick with code coverage is that a high code coverage number doesn’t actually tell you anything in and by itself. For it is just as easy to get a high code coverage with a single line of test which exercises the entire system as it is to do it the hard way and write a lot of small unit tests. A single unit test which executes main will most likely result in a high code coverage number, even without having any assertions.

Similarly, consider the following example :

void someMethod(int a, String b) {

if (a < someThreshold || isValid(b)) {
// Do first thing
} else if (a == someThreshold && isNumber(b)) {
// DO second thing
}
}

Now, it is possible to to write just two unit tests, and still attain 100% coverage. Here, if I call this test with a < a ="="">

And these are exactly the type of things which can demean the value of code coverage. Any manager worth his salt should understand this aspect of code coverage. It is possible and not too difficult to attain a very high code coverage even with a sub par quality product. The trick is in understanding how the tests are written, and how comprehensive they are.

Code coverage is a great tool to find out spots which are completely untested, and projects which develop their code in a Test Driven fashion often end up with high code coverage. But that does not imply that projects with a high code coverage are of the greatest quality. Careful consideration of their testing practices and comprehensiveness is essential in these cases. Track stuff like bugs, amount of test code written compared to amount of production code, etc to get a overall picture, rather than relying on just one metric.