Better test failure feedback with assertThat

It's fairly obvious how each of the assert methods work from their name but what's not so obvious is how they report failures.

All assertions signal verification failure by throwing a java.lang.Error subtype. This allows JUnit to provide the location of the failed assertion in the stacktrace as feedback. Here's assertTrue() failing:

java.lang.AssertionError
   at org.junit.Assert.fail(Assert.java:86)
   at org.junit.Assert.assertTrue(Assert.java:41)
   at org.junit.Assert.assertTrue(Assert.java:52)
   ... the line of your test code where assertTrue was called from ...

But where's the error message? We can guess what the actual and expected values were but only once we visit the test method that failed and see assertTrue(). This adds an extra step (and perhaps a non-trivial amount of time) to the information gathering effort, making it harder to figure out what went wrong and how to fix it.

Here are the error messages generated by pre-Hamcrest JUnit assert methods (JUnit classic?). Most of these messages are not helpful, some are missing (you just get the AssertionError) and some are inconsistent (assertNull() / assertNotNull() reporting isn't symmetrical)

assertArrayEquals(
        array("one", "two", "three"),
        array("one", "three", "four"))

assertArrayEquals("message",
        array("one", "two", "three"),
        array("one", "three", "four"))

assertEquals("aaa", "bbb")

assertEquals("message", "aaa", "bbb")

assertFalse(true)

assertFalse("message", true)

assertNotEquals("aa", "aa")

assertNotEquals("message", "aa", "aa")

assertNotNull(null)

assertNotNull("message", null)

assertNotSame("aa", "aa")

assertNotSame("message", "aa", "aa")

assertNull("aa")

assertNull("message", "aa")

assertSame("aa", "bb")

assertSame("message", "aa", "bb")

assertTrue(false)

assertTrue("message", false)

Even if an optional description parameter is provided to the assert in an attempt to improve the reporting, the description is used inconsistently. Though always appended added before the generated part of the error message, the wording of the error message makes constructing a useful description highly dependent on the assert. It's better not to add descriptions in order to compensate for inadequate assertion reporting, anyway.

Compare that with the assertThat() equivalents which have better and more consistent failure messages.

assertThat(
        array("one", "three", "four"),
        is(array("one", "two", "three")))

assertThat("bbb", is("aaa"))

assertThat(false, is(true))

assertThat("aa", is(not("aa")))

assertThat(null, is(notNullValue()))

assertThat("aa", is(not(sameInstance("aa"))))

assertThat("aa", is(nullValue()))

assertThat("bb", is(sameInstance("aa")))

assertThat(false, is(true))

And because assertThat() leans heavily on Hamcrest to do the comparison, there's no need for custom static assertion methods (e.g. ) to be generated for different operations on different types and the reporting is consistent.

assertThat(3, is(greaterThanOrEqualTo(4)))

assertThat(3, is(greaterThan(4)))

assertThat(3, is(lessThanOrEqualTo(2)))

assertThat(3, is(lessThan(2)))

assertThat("foobar", containsString("baz"))

assertThat(new Point(0, 1),
   hasProperty("x", equalTo(2.2)))

...and collections. Prior to Hamcrest, collection verification looked something like:

List<String> _123 = asList("one", "two", "three");
assertEquals(3, _123.length());
assertTrue(_123.contains("one"));
assertTrue(_123.contains("two"));
assertTrue(_123.contains("three"));

Often people were driven by their dislike of the assertTrue() feedback to do:

assertEquals(3, _123.length());
Collections.sort(_123);
assertEquals(_123.get(0), "one");
assertEquals(_123.get(1), "three");
assertEquals(_123.get(2), "two");

Either way, it required N+1 assertions for collections of size N. One assertion to check the expected size and one for each expected elements. Aside from being cumbersome and ugly, this is an impractical way of checking collections with more than a few elements.

Hamcrest makes this much easier.

assertThat(_123, containsInAnyOrder("three", "one", "two"));

Verification can be expressed as a single assertion without requiring a specific element ordering (though order can also be verified), can match subsets of specific items and in each case gives pretty decent feedback.

assertThat(_123, containsInAnyOrder("one", "two"))

assertThat(_123,
   containsInAnyOrder("one", "two", "three", "four"))

assertThat(_123,
   containsInAnyOrder("one", "three", "four"))

assertThat(_123, contains("one", "two"))

assertThat(_123,
   contains("one", "two", "three", "four"))

assertThat(_123, contains("one", "three", "four"))

assertThat(_123, contains("one", "three", "two"))

assertThat(_123, hasItems("one", "four"))

Choosing the wrong assertion just makes things harder to diagnose test failures. There isn't any reason to use the old 'JUnit classic' assertions.