The Testability Explorer cometh....

So last time, we explored on how to find hotspots and untested code in your code base. But then you start looking at your code, and then you realize there is a reason why you didn't test the darn thing. The code's untestable. Whoo hoo..

Well, there is no such thing as untestable code. Or rather, all untestable code can be refactored to make it much nicer and easier to test, through a variety of techniques. The first and foremost reason for untestable code ends up being "Constructor doing work." If the constructor of a class does anything more than stuff like "this.x = x" or if it tries to call a constructor itself or use a *GASP* static factory, bingo, you have a problem.

But fixing that isn't the target of this post. That will be covered in a later one, cause its a doozie. No, in this post, I want to talk about how to find these untestable code snippets without any manual effort. Every code base has atleast a few of these gems, which turn up being a nightmare to test, and in turn make everything depending on it a nightmare as well. Well, fear not, for the Testability Explorer cometh...

The Testability Explorer (http://www.testabilityexplorer.org) is an open source tool which looks at classes and does cyclomatic complexity analysis on it. What does that mean ? Well, it looks for things which make testing hard, like conditionals, and recursively dives in to objects it instantiates to find their testability score. In that respect, it is a static and recursive analysis of a code base. It takes all these into consideration and assigns a score to each class. Based on these scores, a class is either

• An excellent to test class
• A Good class but could use some work
• A horrible class to test, needs a lot of work.

The following image, taken from the Testability Explorer website, shows a sample report generated by the tool :

As you can see, it generates html reports with bar graphs and pie charts. It can even, depending on the options you specify, allow you to dig in deep into the problem classes and find the method and line which causes you the most problem to test. This can give you great insight on deciding what classes need refactoring first to make it testable. A lot of times, fixing the most problematic one causes a ripple effect, which fixes a bunch of problems in classes depending on it.

Another great thing is that the Testability Explorer can take jar files, so if you don't want to expose your code directly to it, you have an option. Though sadly, the Testability Explorer is currently only available for Java code. It stands to reason that something similar could be done for C++, though you Javascript guys are out on your own.

All in all, a great tool. But don't depend solely on it. It is great as one tool in a repertoire of tools, but not just by itself. Testability Explorer is also a great way to notice trends, of whether your code is growing more testable or untestable, and other great things, just like code coverage. Though leading you to nothing more than testable code, you would be surprised at how much positive impact increased testability and tests can have on the quality of a project.

So go check it out. And enjoy.

Of EMMA's and Eclipse's

So last time I covered the joys of testing. Now if you do Test Driven Development, then you never have to worry about what you have tested and what is untested, but what about the scores of projects which aren't developed in a TDD fashion ? How do I figure out which among my thousands of classes needs tests most urgently right now ? Is the class which usually gets the most bug fixes my prime target, or are there even worse classes that should be tested ?

One solid easy way to identify this is to run some code coverage analysis on your code. What is code coverage, you ask ? It is one of the single most brilliant things which gives you coverage information about your code. Coverage information in this case basically provides you with knowledge on how much of your code and classes are executed by your tests, and which hotspots in your code are completely ignored by the test. Though you can run it without having a suite of tests and running your program manually, it provides the most bang for your buck (especially considering it is free) when you run it along with the tests. Code coverage tools generally provide information on a per package, per class, per method, per block and per line basis, so you can dig in as deep as you like.

EMMA is a free, open source tool which allows you to generate code coverage information for Java code. And if you have some issues with providing some random tool with your source code, fear not, for you can provide it with a jar file which it can instrument and generate code coverage information for. And it generates nice Html reports if you prefer, which you can again dig into as deep as you like.

The above image, grabbed from EMMA's official website, shows a sample html report for a single class. Notice how nicely it highlights the class. The green lines represent code which was covered by one test or the other, the red ones were lines not covered at all and the yellow ones represent code which was partially covered. EMMA is smart enough to distinguish partial matches, as in the mutli condition statement above.

This image, also grabbed from EMMA's official website, shows code coverage information on a package level. Notice how it breaks down the information to a method, block and line level. So this report can be used to easily identify classes lacking in testing and allows surveyors to tackle these hotspots.

To make it even easier for developers, EMMA is available as a plugin to most IDEs, including Eclipse. Available at EclEmma. This tool can be run along with the tests to generate code coverage information. SO you can instant feedback on any new test that you have written, instead of having to come out and run EMMA separately. This can also help give you feedback on your test, to ensure that you are testing the code paths that you intended.

Generally, it has been found that projects with code coverage less than 50 - 60% generally tend to have much more bugs and fixes than projects with higher coverage. And projects which are developed using TDD tend to end with high code coverage numbers, generally above 80%.

But this comes with a few caveats. Even if you do attain 100% code coverage, it does not mean that your job is done. In the end, code coverage is a statistic and can be bent or twist by a knowledgeable person. A high code coverage number ensures you are hitting a lot of your code paths and your tests exercise a lot of the system, but it does not necessarily mean that you covered all possible cases nor does it mean that your source code itself is testable or maintainable. It is also entirely possible to write as few tests as possible which exercise bigger amounts of systems and don't provide much value, rather than writing small fast unit tests which exercise just a small part of the system and still end up with good code coverage numbers.

But that said, code coverage is an excellent tool when used as part of a greater set of tools to evaluate your project and can reveal startling trends about your projects. Maintaining a historical trend of how your code coverage grows is an interesting metric and can reveal the practices of your developers as well. And considering how it is free for Java, I don't see any reason to not start using this for your projects.

Getting to the root of a problem...

So someone raised a great point about my last post, noticing that it is usually hard to pinpoint the exact line of code where your error is when a test fails. Usually, you know which line in the test failed, but not why. That happens when you have integration tests, and not unit tests.

When I say "Pin point breakages and causes of failures to the line level", I actually do mean that. It is quite possible to do this with true unit tests. This comes back to the definition of a unit test. It should be testing one unit or method, and not testing things across classes. So if you want to isolate the cause of the breakage, we mock out anything that is not this class to truly test just this class.

Consider this example, you have a class House which has a few methods like enterHouse() and lockHouseAndLeave(). A house has some members like door, rooms, etc. When we test enterHouse(), we mock out the door and rooms, and assume those work perfectly (Since we test those individually as well for correctness). We then test just enterHouse(), and have the mocks return what we expect it should in the correct behavior. That way, if anything breaks in a test for enterHouse(), you cna be totally assured it is something because of enterHouse(), and rarely have to dig in deep.

Now if you have an integration test, where you don't have mocks for Doors and rooms, you are going to have trouble pinpointing the point of breakage, but with mocks, and proper unit testing, you know exactly what is being tested and what is causing the test to fail.

Though getting to this point where you can test pieces in isolation either requires you start designing with testability in mind or refactoring stuff to get it where you can inject mocks.

In my next post, I plan to talk about code coverage, one of the easiest ways to figure out what is untested and some caveats.

The Joys of Testing

Yes, you read it right. The title does say "The Joys of Testing". And no, I am not trying to be cute or imply anything other than what the title says. It actually can (and should) be a pleasure and joy to test your code.

When I first joined Google as a Software Engineer in Test, fresh out of college, all naive and innocent, I had a very limited exposure (read almost none) to testing. I wasn't even sure if I could do justice to my role there. But over the past year, I have sunk into my role with relish and learned far more about testing (and coding and engineering) than I seemed to have in my 4 years at college. This is not to say that I didn't learn anything at college, because I sure did, from Data Structures and Algorithms to OOPS to AI, but none of them ever seemed to cover or focus on testing.

And then I joined this industry where everyone I talk to generally says that you should test your code, and every class you write should be accompanied by tests. So I learnt JUnit and figured out what they meant. But then I started wondering, how many people actually mean "Do as I say, not as I do." when they say this. For I have seen code snippets of classmates where I go "What the heck was this guy thinking ?" . I have come to realize that many people just don't know how to write tests. Or understand the importance of the same. Oh sure, they say they have tests, and they do, but how many of the tests actually provide any value.

Testing done right is an art. A joy in itself to behold. I still feel shivers and goosebumps when I see a nice beautiful class which is nice and easy to test. There is actually something known as Design for Testability which I will cover in some of my later posts, but the importance of that is underrated. There are so many great tools and techniques to make testing easier which are often left unmentioned. I hope to cover a great many of those tools and approaches.

And we finally come to the question of "Why test ?". I mean, who cares if some code has tests or not. I can clearly see it is working. But to those, I raise this question. Say your mythical application which you see is working. Say you introduce a slew of changes to it, across various parts of the application. And then when you fire up your Oh so perfect application, its broken. Now you manually have to go over whatever you changed in your codebase to find out what broke it. If you had tests around your application, preferably unit tests, you wouldn't have had to wait to compile, build your application, and then go and check the various parts. Just hit a button (in say eclipse), and BAM!!! AssertionFailed. You know exactly what line your code is breaking at, and even why, if you have unit tested properly.

Unit tests when done right, provide multiple benefits, including the following :

• Pin point breakages and causes of failures to the line level
• Reduce the amount of manual testing you have to do for your application
• Help you design your code beautifully (Design for testability)
• Give you instant feedback on whether something works as intended
• Quality Assurance metric for your code
  
• Reduce the amount of code paths you have to test manually.

By the last point, consider the following case. Assume that each part of the app is a class, for ease of understanding. An app, whose frontend is a single class with a single method talks to this middle tier which has again, say one method, which does some things and finally talks to the back end layer, which has, you guessed it, one method. Now each method has 5 possible code paths through it. Now you have two options :

1. You can manually test the whole app, which leaves you with 5 * 5 * 5 code paths to consider, which is 125 possible tests
2. Or you can test each part individually with unit tests. Thats 5 unit tests for the frontend, 5 for the middle tier and 5 for the back end. And then you can have one large scale integration test which makes sure that everything's connected properly.

You just cut down from 125 manual tests to say 16-20 quick small tests which do the exact same thing. You know each individual piece works on its own, and you know that each part is connected properly to its neighbors. Thus, the whole application works.

In my next post, I intend to cover the most basic, fundamental technique to writing testable code, Dependency Injection. I will have a few Java based examples to give an idea, along with a few mentions of way to make it easier.