Code Coverage Metrics

Code coverage reports are great, I love the information they give me. I also love the idea of failing a build if you code coverage metrics drop below a certain point. But I think it's generally accepted that code coverage numbers are can be very misleading. A low % of lines coverage is certainly bad, but a high % of lines covered doesn't necessarily mean you've done a good job either. You could have a bunch of tests that really don't exercise many edge cases, but they hit all the lines of code. It doesn't really mean that its been tested well.

I find this sort of thing happens with integration tests. One medium sized integration test, could "cover" lots and lots of code, I could remove a number of unit tests and still have the same coverage % because I have a lot of integration tests. These days I'm not as interested in the coverage % (okay I still want close to 100) but I'm really interested to know if I run Emma (for example) on FooTest, is Foo 100% covered? In my ideal world each Test would cover its related class 100%. I find the Emma plugin for eclipse really helpful to do that kind of analysis. And I'd love a tool that would give me that kind of report.

Sadly that tool doesn't exist. In the current world coverage metrics are great, but they leave something to be desired. After discovering the moreunit plugin I've realized how a tool like that could help enhance coverage metrics. I want to know for every public method in Foo is there a corresponding test method in FooTest. If you had this kind of metric in combination with code coverage % this could put a confidence value on how good your code coverage is. Sadly that tool doesn't exist either.

But what would be even greater than two tools that don't exist a third tool that combines the two. I would love to know that my FooTest.bar*() methods give me 100% coverage on Foo.bar() method. Having something like that would give me very high confidence in my code coverage metrics. I'm guessing as things move along in code quality metrics we'll start seeing tools like that being developed.

One issue with the moreunit tool is that to do it's analysis it requires test method naming conventions, that in my eyes seem to be in continual development. Google around a little bit... lots of people argue for really long names, junit3 required test to start off the test method names, http://blog.jayfields.com/2008/05/testing-value-of-test-names.html kind of believes there should be no test method names, I personally like to shun the java standard camel cased method names and go more of the ruby route and use underscores (my co-workers don't like that at all). In any case, it's clear (at least to me) that test naming is difficult. But the idea that you could get some really valuable reporting out of standardized test method names seems like good reason to embrace standardization (at least a little bit). I'm personally ready to start preceding all my test method names with "test" just to get some of the ad-hoc metrics that moreunit offers. I think once more tools come out that start assuming testing conventions we'll start to get even more value and flexibility out of our test code.

Update - it turns out that hacking the plugin wasn't too hard, I now have the plugin recognizing method names like "foo()" instead of "testFoo()". I'm not going to consider changing my method names after all!

moreUnit

wow... the moreUnit plugin for eclipse is just what I've been looking for, it's amazing. One of the more important things it does is associate Classes with Test Classes - So if you have Foo it knows that FooTest is related to it. From there it does a number of great things.

• Handles renaming correctly - if you rename Foo to Foo2 it will automatically rename FooTest to Foo2Test. For that functionality it's worth it just to use this plugin.
• ^J toggles between Foo and FooTest AND if FooTest doesn't exist yet, it prompts you to create it. Woo hoo! I constantly create a new class - then CRTL N to create the test, this cuts down one anoying step, pretty cool.
• It associates method names with test method names. And does a few awesome things with that association.
  • If you have:
    Foo.bar() and FooTest.testBar()
    you rename Foo.bar() to Foo.foo()
    it will rename your test method to FooTest.testFoo()
    it will even do multiple test methods if you had testBarA() - testBarB() - testBar*() - it will rename all of them.
  • Because it associates the methods with test methods - it can provide you with a view that shows you all the test methods your missing for a class (Emma's great, but it doesn't give you this kind of detail, you can always trick yourself into thinking you covered stuff when you haven't, this is pretty awesome, imagine failing the build if you don't have a testMethod for every method in a class, that I think would be much more valuable than failing on code coverage metrics). In this view you can also create the missing methods in you test class
  • I have a bone to pick though. In Junit4 your method names really shouldn't begin with the word test any longer, because you already have that information in the form of an annotation at the beginning of your test method - BUT for moreUnit to know that a method in a test class is associated to a method in your class under test, you need to prefix with "test"
• Some of the documentation doesn't match up to the most recent version, but 1.2 was just released before Christmas, so I assume it will be updated at some point. For example I don't seems to be able to setup project specific settings - and really the test directory setting, doesn't appear to have any effect (It specifies the directory where you create new tests).

So I've just been playing with it for an hour or so, but this thing is great, I can't wait to start using it day to day

Google Reader

I recently made the switch from Bloglines to Google reader. I'm definitely happy I did. Google reader does a few things much better than bloglines.

1. It has some nice keyboard shortcuts http://www.google.com/help/reader/faq.html#shortcuts , n/p (next previous), o (open close), s (star), m (mark as un /read). The short cuts are simple to use + remember.
2. I really like the Feed / Subscription pane. It shows unread blog post in bold, but then it has all the other blog posts there below it. it's really easy to scan though old posts and find one your looking for.
3. Speaking of finding stuff you're looking for, it also allows you to star a post. There are a few posts that I go back and re-read. My usual MO is to add a bookmark in firefox to the post. But being able to star an item and come back to it is really a great feature for me.
4. Top Recommendations - I really like this, three links that google thinks I'd like. For example I found out No Fluff Just Stuff has a feed - which is an aggregation of all the speakers blog posts it's a really nice feed.

I'm interested to see if bloglines will do a redesign anytime soon. But for me it doesn't compete with Google Readers features and usability

Updated - I just have to add, I've been exploring the rest of the keyboard shortcuts for google reader + they're awesome no more mouse. The commands to browse your subscriptions are great. shift n/p is next previous subscription shift o is open subscription. This is a really nice job of modifying other similar key bindings for a different context... it's very easy to use. Then there are the g? commands. Go somewhere gs goto starred items, gh go home, gu I don't know what the u stands for but it is goto subscription. It brings up an awesome quicksilver esq type completion window, it's pretty and nice to use.

Daily WTF Error'd submission - 16 Million TB

I'm thinking this is worthy of a WTF Error'd, but in case it doesn't make it here it is:

So... one day I'm writing some code, and my computer seems slow this is what the activity manager had to say: