Better test failure feedback with assertThat

Choose the right kind of assert and make things easier to diagnose and fix broken tests

It's fairly obvious how each of the assert methods work from their name but what's not so obvious is how they report failures.

All assertions signal verification failure by throwing a java.lang.Error subtype. This allows JUnit to provide the location of the failed assertion in the stacktrace as feedback. Here's assertTrue() failing:

java.lang.AssertionError
   at org.junit.Assert.fail(Assert.java:86)
   at org.junit.Assert.assertTrue(Assert.java:41)
   at org.junit.Assert.assertTrue(Assert.java:52)
   ... the line of your test code where assertTrue was called from ...

But where's the error message? We can guess what the actual and expected values were but only once we visit the test method that failed and see assertTrue(). This adds an extra step (and perhaps a non-trivial amount of time) to the information gathering effort, making it harder to figure out what went wrong and how to fix it.

Here are the error messages generated by pre-Hamcrest JUnit assert methods (JUnit classic?). Most of these messages are not helpful, some are missing (you just get the AssertionError) and some are inconsistent (assertNull() / assertNotNull() reporting isn't symmetrical)

Assertion Generated Message
assertArrayEquals(
        array("one", "two", "three"),
        array("one", "three", "four"))
arrays first differed at element [1]; expected:<t[wo]> but was:<t[hree]>
assertArrayEquals("message",
        array("one", "two", "three"),
        array("one", "three", "four"))
message: arrays first differed at element [1]; expected:<t[wo]> but was:<t[hree]>
assertEquals("aaa", "bbb")
expected:<[aaa]> but was:<[bbb]>
assertEquals("message", "aaa", "bbb")
message expected:<[aaa]> but was:<[bbb]>
assertFalse(true)
assertFalse("message", true)
message
assertNotEquals("aa", "aa")
Values should be different. Actual: aa
assertNotEquals("message", "aa", "aa")
message. Actual: aa
assertNotNull(null)
assertNotNull("message", null)
message
assertNotSame("aa", "aa")
expected not same
assertNotSame("message", "aa", "aa")
message expected not same
assertNull("aa")
expected null, but was:<aa>
assertNull("message", "aa")
message expected null, but was:<aa>
assertSame("aa", "bb")
expected same:<aa> was not:<bb>
assertSame("message", "aa", "bb")
message expected same:<aa> was not:<bb>
assertTrue(false)
assertTrue("message", false)
message

Even if an optional description parameter is provided to the assert in an attempt to improve the reporting, the description is used inconsistently. Though always appended added before the generated part of the error message, the wording of the error message makes constructing a useful description highly dependent on the assert. It's better not to add descriptions in order to compensate for inadequate assertion reporting, anyway.

Compare that with the assertThat() equivalents which have better and more consistent failure messages.

Assertion Generated message
assertThat(
        array("one", "three", "four"),
        is(array("one", "two", "three")))
Expected: is ["one", "two", "three"]
   but: was ["one", "three", "four"]
assertThat("bbb", is("aaa"))
Expected: is "aaa"
   but: was "bbb"
assertThat(false, is(true))
Expected: is <true>
   but: was <false>
assertThat("aa", is(not("aa")))
Expected: is not "aa"
   but: was "aa"
assertThat(null, is(notNullValue()))
Expected: is not null
   but: was null
assertThat("aa", is(not(sameInstance("aa"))))
Expected: is not sameInstance("aa")
   but: was "aa"
assertThat("aa", is(nullValue()))
Expected: is null
   but: was "aa"
assertThat("bb", is(sameInstance("aa")))
Expected: is sameInstance("aa")
   but: was "bb"
assertThat(false, is(true))
Expected: is <true>
   but: was <false>

And because assertThat() leans heavily on Hamcrest to do the comparison, there's no need for custom static assertion methods (e.g. ) to be generated for different operations on different types and the reporting is consistent.

Assertion Generated message
assertThat(3, is(greaterThanOrEqualTo(4)))
Expected: is a value equal to or greater than <4>
   but: <3> was less than <4>
assertThat(3, is(greaterThan(4)))
Expected: is a value greater than <4>
   but: <3> was less than <4>
assertThat(3, is(lessThanOrEqualTo(2)))
Expected: is a value less than or equal to <2>
   but: <3> was greater than <2>
assertThat(3, is(lessThan(2)))
Expected: is a value less than <2>
   but: <3> was greater than <2>
assertThat("foobar", containsString("baz"))
Expected: a string containing "baz"
   but: was "foobar"
assertThat(new Point(0, 1),
   hasProperty("x", equalTo(2.2)))
Expected: hasProperty("x", <2.2>)
   but: property 'x' was <0.0>

...and collections. Prior to Hamcrest, collection verification looked something like:

List<String> _123 = asList("one", "two", "three");
assertEquals(3, _123.length());
assertTrue(_123.contains("one"));
assertTrue(_123.contains("two"));
assertTrue(_123.contains("three"));

Often people were driven by their dislike of the assertTrue() feedback to do:

assertEquals(3, _123.length());
Collections.sort(_123);
assertEquals(_123.get(0), "one");
assertEquals(_123.get(1), "three");
assertEquals(_123.get(2), "two");

Either way, it required N+1 assertions for collections of size N. One assertion to check the expected size and one for each expected elements. Aside from being cumbersome and ugly, this is an impractical way of checking collections with more than a few elements.

Hamcrest makes this much easier.

assertThat(_123, containsInAnyOrder("three", "one", "two"));

Verification can be expressed as a single assertion without requiring a specific element ordering (though order can also be verified), can match subsets of specific items and in each case gives pretty decent feedback.

Assertion Generated message
assertThat(_123, containsInAnyOrder("one", "two"))
Expected: iterable over ["one", "two"] in any order
   but: Not matched: "three"
assertThat(_123,
   containsInAnyOrder("one", "two", "three", "four"))
Expected: iterable over ["one", "two", "three", "four"] in any order
   but: No item matches: "four" in ["one", "two", "three"]
assertThat(_123,
   containsInAnyOrder("one", "three", "four"))
Expected: iterable over ["one", "three", "four"] in any order
   but: Not matched: "two"
assertThat(_123, contains("one", "two"))
Expected: iterable containing ["one", "two"]
   but: Not matched: "three"
assertThat(_123,
   contains("one", "two", "three", "four"))
Expected: iterable containing ["one", "two", "three", "four"]
   but: No item matched: "four"
assertThat(_123, contains("one", "three", "four"))
Expected: iterable containing ["one", "three", "four"]
   but: item 1: was "two"
assertThat(_123, contains("one", "three", "two"))
Expected: iterable containing ["one", "three", "two"]
   but: item 1: was "two"
assertThat(_123, hasItems("one", "four"))
Expected: (a collection containing "one" and a collection containing "four")
   but: a collection containing "four" was "one", was "two", was "three"

Choosing the wrong assertion just makes things harder to diagnose test failures. There isn't any reason to use the old 'JUnit classic' assertions.