Continuous Testing - Infinitest

If you check out the comments on configure-your-ide-to-run-your-tests-automatically there are a slew of references to Infinitest which is a continuous testing tool, which has a very similar feature list to JUnitMax, which is also free.

I got the plugin up and running on a small - medium project (387 unit-tests). My project is laid out
src/main
src/test
src/integration
Immediately I had a problem when trying out the plugin, it was running all my integration tests. After much rooting around it turns out you can add the special infinitest.filters file:

From readme.pdf of http://infinitest.googlecode.com/files/infinitest-4.0.3.zip

You may have tests in your classpath that you don't want Infinitest to run. These tests can be filtered out by creating a file in the working directory of your project named infinitest.filters. It should contain one regular expression per line. Any class names (not file names) that match any regular expressions in that file will not be run. For example: org\.myproject\.acceptance\..* .*\$.* will filter out all the classes in the com.myproject.acceptance package, and any inner classes (which always contain a $).

I added my filters and everything seems to work. Infinitest doesn't run all your tests, just a subset of them (it does some voodoo to look for dependencies), it tells you how many it ran, but it doesn't tell you which ones it ran... which is fine (I guess) but the anal part of me really wants more insight into what tests it's running. For example, I modified a class, Infinitest informed me that it ran 51 tests, but really which ones did it run... I just really want to know!

Also another interesting tidbit, from http://www.benrady.com/2009/04/comparing-infinitest-and-junitmax.html

The other function that I think a CT runner needs to do is test selection. If you're doing TDD, you're probably running a single test every time you make a change. If you're doing CI, you're running all your tests on a semi regular basis. Somewhere in the middle of these two approaches is a good balance of feedback quality vs speed. I think that's where CT tools should be focused. Infinitest uses dependency analysis to determine what tests need to be run for a given change.

Infinitest isn't particularly speedy, all my unit tests took .6 seconds to run, and the 51 Infinitest chose to run took 3 seconds. So maybe it's important for Infinitest to trim what it's running, but if your unit tests are fast to run than it's more of an Infinitest implementation detail vs a real time to run issue.

Anyway, I can't wait to start trying this tool as part of my workflow.

Running Tests On IDE Save

Misko Hevery just put up a blog post describing how to configure Eclipse to run your test suite after every save. It doesn't seem like a bad idea, and I'm definitely going to try it out, but even with a test suite that runs in under a second, I wonder how annoying it would be. I'm an obsessive saver, I can't seem to help myself. Even if I'm only looking at a file I need to format it then save it... how would that be if it took a second to do.

"Here is a common scenario. Your tests are green and you start doing whole bunch of refactorings which you think are trivial and safe. When you are done you run the tests and it is broken. The problem is that you did ten little things and you don’t know which of the ten things you did broke the code. The solution is to run the test more often, but we just forget."

I'm not sure this really is a common scenario I run into, but it's an intriguing idea to run tests on save. I really believe that a change in process effects your product. Run on save would certainly change my process, but how would it change my product?

Misko's post reminded me of JUnit Max Kent Beck's Eclipse Plugin, which has the similar feature of run tests on save. There are a bunch of interesting features, like displaying test failures like compilation failures, and keeping JUnit unobtrusive. There is one un-interesting feature, it's in beta, and has a monthly subscription. I'd love to try it, even pay for a demo (I think :P), but knowing myself, whenever I subscribe to something, I never unsubscribe, even if I'm not using the software or reading the magazine. So I'll have to wait on trying out JUnit Max.