Friday, February 20, 2009

Java Broke Computer Science

So... yeah, Java broke computer science. But first some background, I'm doing agent based modeling these days, and a common thing you have 2+ nested loops. For example I'm working on a project right now where I'll be looping an outer loop somewhere in the range of 100,000+ times and an inner loop maybe 10,000 - 20,000. Suffice to say, I need to make it as fast as I can. I wrote some little tests so I could get a sense of how long it would take, just to run through the loop and do something small.

A class to do stuff with:

public class Foo {
 private int count;
 private boolean inc;

 public Foo(boolean inc) {
  this.inc = inc;
 }

 public void tick() {
  if (inc) {
   count++;
  }
 }
}

My Test:

private int outerLoop = 100000;
private int innerLoop = 10000;

@Test
public void arrays() {
 Foo[] foos = new Foo[innerLoop];
 for (int i = 0; i < innerLoop; i++) {
  boolean inc = i % 2 == 0 ? true : false;
  foos[i] = new Foo(inc);
 }

 long start = System.currentTimeMillis();
 for (int i = 0; i < outerLoop; i++) {
  for (int j = 0; j < innerLoop; j++) {
   foos[j].tick();
  }
 }
 long fin = System.currentTimeMillis();
 System.out.println("arrays took: " + (fin - start));
}

Takes on my machine in java 1.6 ~2,000 ms I'd like to point out here, that Foo[] foos will not change, so I don't care about speed of modifying a data structure, just the fastest way to iterate.

Right, iterate, what's an arrayList looklike?

@Test
public void arrayList() {
 ArrayList arrayList = new ArrayList();
 for (int i = 0; i < innerLoop; i++) {
  boolean inc = i % 2 == 0 ? true : false;
  arrayList.add(new Foo(inc));
 }
 long start = System.currentTimeMillis();
 for (int i = 0; i < outerLoop; i++) {
  for (Foo node: arrayList) {
   node.tick();
  }
 }
 long fin = System.currentTimeMillis();
 System.out.println("ArrayList took: " + (fin - start));
}

This runs in ~3000 ms

Okay the Java Iterators aren't cutting it, but what about a real linked list? not java.util.LinkedList but, Linked_list the data structure. This as far as I can figure should be ideal. All there is to iteration is pointer de-referencing. There are no lookups to find the a given index in an array, just following the pointers.

Here's my implementation:

public class LinkedListNode {
 public T data;
 public LinkedListNode next;

 public LinkedListNode(T data) {
  this.data = data;
 }
}

public class LinkedList {
 private LinkedListNode first;
 private LinkedListNode last;

 public LinkedList() {}

 public LinkedListNode first() {
  return first;
 }

 public void addToEnd(LinkedListNode linkedListNode) {
  if (first == null) {
   first = linkedListNode;
   last = linkedListNode;
  } else {
   last.next = linkedListNode;
   last = linkedListNode;
  }
 }
}

And the test:

@Test
public void linkedList() {
 LinkedList linkedList = new LinkedList();
 for (int i = 0; i < innerLoop; i++) {
  boolean inc = i % 2 == 0 ? true : false;
  linkedList.addToEnd(new LinkedListNode(new Foo(inc)));
 }

 long start = System.currentTimeMillis();
 LinkedListNode startNode = linkedList.first();
 for (int i = 0; i < outerLoop; i++) {
  LinkedListNode node = startNode;
  while (node.next != null) {
   node.data.tick();
   node = node.next;
  }
 }
 long fin = System.currentTimeMillis();
 System.out.println("linkedList took: " + (fin - start));
}

Any guesses to how this performed? try ~5500 ms Like I said Java broke computer science, they must be doing some amazing compiler optimizations, how is it that node.next is slower than foo[499]. It boggles the mind. What's the take away here? When you can use a primitive array do so?

oh, just a note, yes I know ArrayList is backed by a array, and yes I tested a java.util.LinkedList as well, and it was the worst of the bunch.

Monday, February 9, 2009

Test Data Builders

I've been meaning to post something about this for a little bit, not exactly sure what I had to add to the subject besides, "hey that's a good idea", and after about a month thinking about it I think it's safe to say I don't have much to add except for evangelism.

So without further ado, I'd suggest reading a few articles on test data builders:
http://blog.jayfields.com/2009/01/most-java-unit-tests-consist-of-class.html
Test Data Builders

In short Test Data Builders allow you to minimize maintenance cost in your tests and allow you to easily inject objects into your test objects, it's great, great stuff.

I guess I do have an addition, a story:
On my last project a co-worker made a perfectly reasonable api change, and added a parameter to a constructor, after making the change he realized this would effect ~100 tests (whoa), so he used Eclipses magic refactoring tools and initialized the value to null. The problem was his changes really required that new parameter to be non-null (in fact if it was null a NPE would be thrown), and even though the code compiled all the tests broke (with NPEs). He didn't want to fix the tests. Why should he fix all 100 unrelated tests, maybe 5-10 tests but 100, that was just too much. And really I understood that. And I was lucky, I had just read Jay Fields post about Test Data Builders. 1/2 a day later a co-worker and I added all the Builders we needed for our project and started updating our tests to use those builders. With the builders in place, fixing the NPEs was in one place. And now, if there is a similar situation and one change breaks hundreds of tests, there is one place to go and fix them all.

Anyway, I would not start a new project without using Test Data Builders they really pay for themselves when maintaining your tests and your code.

Friday, February 6, 2009

Stack Overflow #38

There's a big old brouhaha regarding the Stack Overflow podcast #38 Joel summarizes the pod cast: http://www.joelonsoftware.com/items/2009/01/31.html He comes across as very anti-unit test, which is fairly surprising to me. I was surprised, because I feel that unit testing has only made me a better developer and allowed me to deliver higher quality software faster. Anyway I just came across Uncle Bob's response: http://blog.objectmentor.com/articles/2009/01/31/quality-doesnt-matter-that-much-jeff-and-joel. From the comments on Object Mentor, looks Uncle Bob will be participating in a stackoverflow podcast, that should be interesting to to hear.

Another blogger I follow also posted a good response: http://blog.jayfields.com/2009/02/thoughts-on-developer-testing.html

Anyone come across anything else interesting?

Monday, January 12, 2009

More about Mocks and Mockito

I've been digging around for more about Mockito and how it's different from EasyMock / jMock here's what I got:

I think there are very few use cases for mocks. They produce brittle tests and are hard to read. In addition 99% of the time you don't want a mock, you want a test spy. So why if you had the choice would you choose a mocking framework over a test spy framework?

MoreUnit Test Method Names

I got a patch accepted to the MoreUnit project. If you pull down Head and build / install the plugin there is a new option which allows you to specify whether you want your methods to be created in the style of "testFoo" or just "foo"

I've been using moreUnit pretty steadily for about 2 weeks and it is a huge productivity enhancer. if you're doing TDD you should definitely install the plugin!

Thursday, January 8, 2009

EasyMock vs Mockito

I've been meaning to write about this for a little bit, but first check out a Jay Fields post Ubiquitous Assertion Syntax. I really like the idea that tests follow a similar structure and I'm okay with the JUnit style.

@Test public void methodFoo_should_do_X_when_Y() { test setup test setup test setup assertEquals(a, b); }
What's important to me is that the assertions always come at the end of the test and it's clear what you are asserting, without needing to rely on the method name. Here's an example:
public class Foo() { private Bar bar; private String something = null; private String somethingElse = null; public Foo(Bar bar) { this.bar = bar; } public void setStuffUp() { this.something = bar.a(); this.somethingElse = bar.b(); } public Foo callBarC() { return bar.c(something,somethingElse); } }
How a Mockito test might look:
@Test public void callBarC_reallyBadTestName() { Bar bar = mock(Bar.class); Foo foo = new Foo(bar); when(bar.a()).thenReturn("A"); when(bar.b()).thenReturn("B"); foo.setStuffUp(); foo.callBarC(); verify(bar).c("A","B"); }
How an EasyMock example might look:
@Test public void callBarC_reallyBadTestName() { Bar bar = createMock(Bar.class); Foo foo = new Foo(bar); expect(bar.a()).andReturn("A"); expect(bar.b()).andReturn("B"); expect(bar.c("A","B").andReturn(null); replay(bar); foo.setStuffUp(); foo.callBarC(); verify(bar); }

In the Mockito version it's clear what behavior you are testing... it says verify(bar).c("A","B"); But in the EasyMock version the code itself does tell you what you're testing, you have to rely on the bad test name. You could argue that I'm testing all the expectations, but most often you're not. It's not uncommon to have two mocks one that sets up state for another one, the one you're really interested in testing, the focus of your test. Something like:

@Test public void verifyBar2_is_called_correctly() { Bar1 bar1 = createMock(Bar1.class); Bar2 bar2 = createMock(Bar2.class); Foo foo = new Foo(bar1,bar2); bar2.foo(); expect(bar1.a()).andReturn("A"); expect(bar1.b()).andReturn("B"); expect(bar1.c("A","B").andReturn(null); replay(bar1,bar2); foo.setStuffUp(); foo.callBarC(); verify(bar1,bar2); }

In this example except for the test name, it's unclear what the intent of the test is. Here it is in mockito

@Test public void verifyBar2_is_called_correctly() { Bar1 bar1 = mock(Bar1.class); Bar2 bar2 = mock(Bar2.class); Foo foo = new Foo(bar1,bar2); when(bar1.a()).thenReturn("A"); when(bar1.b()).thenReturn("B"); when(bar1.c("A","B").thenReturn(null); foo.setStuffUp(); foo.callBarC(); verify(bar2).foo(); }

In mockito, even if I put in a bad test name, the syntax of the test preserves the intent. This is really important. Easymock was a great library, but I think as Jay put it Mockito starts to bring us to testing 2.0

The google testing blog just published an article Use EasyMock I guess I just disagree. Mockito grew out of EasyMock, and I think they took a great library and made it even better, so I really think the title of the post should have been "Use Mockito" not EasyMock. Anyway... there's a whole lot more that Mockito does check out the examples and get off the EasyMock and start drinking the Mockito :P

Sunday, December 28, 2008

Code Coverage Metrics

Code coverage reports are great, I love the information they give me. I also love the idea of failing a build if you code coverage metrics drop below a certain point. But I think it's generally accepted that code coverage numbers are can be very misleading. A low % of lines coverage is certainly bad, but a high % of lines covered doesn't necessarily mean you've done a good job either. You could have a bunch of tests that really don't exercise many edge cases, but they hit all the lines of code. It doesn't really mean that its been tested well.

I find this sort of thing happens with integration tests. One medium sized integration test, could "cover" lots and lots of code, I could remove a number of unit tests and still have the same coverage % because I have a lot of integration tests. These days I'm not as interested in the coverage % (okay I still want close to 100) but I'm really interested to know if I run Emma (for example) on FooTest, is Foo 100% covered? In my ideal world each Test would cover its related class 100%. I find the Emma plugin for eclipse really helpful to do that kind of analysis. And I'd love a tool that would give me that kind of report.

Sadly that tool doesn't exist. In the current world coverage metrics are great, but they leave something to be desired. After discovering the moreunit plugin I've realized how a tool like that could help enhance coverage metrics. I want to know for every public method in Foo is there a corresponding test method in FooTest. If you had this kind of metric in combination with code coverage % this could put a confidence value on how good your code coverage is. Sadly that tool doesn't exist either.

But what would be even greater than two tools that don't exist a third tool that combines the two. I would love to know that my FooTest.bar*() methods give me 100% coverage on Foo.bar() method. Having something like that would give me very high confidence in my code coverage metrics. I'm guessing as things move along in code quality metrics we'll start seeing tools like that being developed.

One issue with the moreunit tool is that to do it's analysis it requires test method naming conventions, that in my eyes seem to be in continual development. Google around a little bit... lots of people argue for really long names, junit3 required test to start off the test method names, http://blog.jayfields.com/2008/05/testing-value-of-test-names.html kind of believes there should be no test method names, I personally like to shun the java standard camel cased method names and go more of the ruby route and use underscores (my co-workers don't like that at all). In any case, it's clear (at least to me) that test naming is difficult. But the idea that you could get some really valuable reporting out of standardized test method names seems like good reason to embrace standardization (at least a little bit). I'm personally ready to start preceding all my test method names with "test" just to get some of the ad-hoc metrics that moreunit offers. I think once more tools come out that start assuming testing conventions we'll start to get even more value and flexibility out of our test code.

Update - it turns out that hacking the plugin wasn't too hard, I now have the plugin recognizing method names like "foo()" instead of "testFoo()". I'm not going to consider changing my method names after all!

 
Web Statistics