public class ArithmeticOperation {
public int add(int a, int b) {
return a + b;
}
public int substract(int a, int b) {
return a - b;
}
public int multiply(int a, int b) {
return a * b;
}
public double divide(int a, int b) {
return a / b;
}
}junit5 spock testing mock mockito January 1, 2018
Introduction
For over a decade JUnit 4 has been the go-to test framework for JVM-based software projects. You can find the use of JUnit across the board in organizations from big entprises to small startups. Despite its popularity, the test framework’s features barely moved with the times. Due to its deep penetration among the industry, evolutionary changes became harder and harder to implement without introducing inevidable breakages to the API.
With the rise of newer JVM languages like Groovy and Kotlin, feature-rich test frameworks emerged. The test framework Spock became a welcome alternative for many projects fearless to adopt Groovy as part of their polyglot software stack.
With the first GA release of JUnit 5 in September 2017, the JUnit team brought real innovation to the established space of testing JVM code. Not only is the release packed with new features comparable to the ones provided by Spock, JUnit 5 also serves as a platform for launching other test frameworks on the JVM.
In this blog post, I am going to compare typical testing usage patterns and features available for JUnit 5 and Spock. The content is not going to discuss the fundamental, methodological differences between JUnit 5 and Spock (behavior-driven development (BDD) vs. non-BDD). You can find all sample code in a dedicated repository on Github. All examples are based on JUnit 5.0.2 and Spock 1.1.
For a quick reference, you can directly jump to a specific test framework feature:
Code under test
For the purpose of demonstrating the capabilities of both test frameworks, we’ll use two sets of classes and/or interfaces. The simplest class provides methods for executing arithmetic operations shown in listing 1. Most test framework features in this post can be demonstrated with the help of ArithmeticOperation.java.
ArithmeticOperation.java
Listing 1. Class implementing artithmetic operations
More advanced testing capabilities e.g. code that throws an exception require a slightly more complex setup. The following interface and class read the contents of a file for a given Path instance.
FileReader.java
import java.io.IOException;
public interface FileReader {
String readContent(Path path) throws IOException;
}Listing 2. File reader interface
The implementation of the interface throws an IOException if the provided Path does not exist.
DefaultFileReader.java
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
public class DefaultFileReader implements FileReader {
@Override
public String readContent(Path path) throws IOException {
if (Files.notExists(path)) {
throw new IOException("File does not exist");
}
return new String(Files.readAllBytes(path));
}
}Listing 3. Default file reader implementation
You will find that the class FileManager is referenced in some of the test cases described below. FileManager is just a wrapper around FileReader to demonstrate mocking capabilities. You can find the code for the class in the source code repository.
Test execution
Let’s start with the most simplistic use case: marking test methods or the whole class for execution with the test framework.
JUnit 5
In JUnit 5 you use the annotation @Test to indicate that a method should be executed as a test. JUnit 5 takes a different approach than JUnit 4. The annotation is only applicable to a method but not the class. The annotated test method must not be private or static. Test methods do not return a value.
SimpleTest.java
import org.junit.jupiter.api.Test;
public class SimpleTest {
private final ArithmeticOperation arithmeticOperation = new ArithmeticOperation();
@Test
void canAdd() {
assertEquals(3, arithmeticOperation.add(1, 2));
}
@Test
void canSubstract() {
assertEquals(1, arithmeticOperation.substract(2, 1));
}
@Test
void canMultiply() {
assertEquals(6, arithmeticOperation.multiply(2, 3));
}
@Test
void canDivide() {
assertEquals(3, arithmeticOperation.divide(6, 2));
}
}Listing 4. Using annotation to indicate test execution
Spock
To indicate that all methods of a class should be treated as test methods, you’ll need to extend from Specification. On the one hand, extending from a class requires less work on your end to turn all methods into test methods. On the other hand, it makes it harder to create a hierarchy of parent classes in case you want to formalize reusable fixtures through inheritance. Disabling test cases allows you to be more selective.
SimpleTest.groovy
import spock.lang.Specification
class SimpleTest extends Specification {
@Subject def arithmeticOperation = new ArithmeticOperation()
def canAdd() {
expect:
arithmeticOperation.add(1, 2) == 3
}
def canSubstract() {
expect:
arithmeticOperation.substract(2, 1) == 1
}
def canMultiply() {
expect:
arithmeticOperation.multiply(2, 3) == 6
}
def canDivide() {
expect:
arithmeticOperation.divide(6, 2) == 3
}
}Listing 5. Extending from abstract class Specification
Fixture set up and tear down
Many tests require fixtures to be set up before any of the test methods can run successfully. A fixture can lay out the expected environment, establish a required class composition or ensure that a service can be reached. Fixtures can be costly to create e.g. if a service endpoint needs to be spun up. After the test method finishes, the fixture may need to be cleaned up.
JUnit 5 and Spock provide ways to create fixtures before individual test methods or just once per test class execution. Equivalent hooks are available for cleanup functionality.
JUnit 5
JUnit 5 supports fixture setup and teardown through annotations. The annotated methods for those operations can use any arbitrary name. You can chose from the following annotations: @BeforeEach, @BeforeAll, @AfterEach and @AfterAll. Please refer to the user guide for more information on those annotations.
Using annotations to indicate fixture setup and teardown feels quite natural. The following test class demonstrates the application of @BeforeEach and @AfterEach to create and delete a temporary file so that the code under test can read its contents.
FixtureSetupCleanup.java
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
public class FixtureSetupCleanup {
private final FileReader fileReader = new DefaultFileReader();
private Path testFile;
@BeforeEach
void setup() throws IOException {
testFile = Files.createTempFile("junit5", ".tmp");
}
@AfterEach
void cleanup() {
testFile.toFile().delete();
}
@Test
void canReadFile() throws IOException {
String text = "hello";
Files.write(testFile, text.getBytes());
assertEquals(text, fileReader.readContent(testFile));
}
}Listing 6. Using annotations for setting up and tearing down fixtures
Spock
Spock does not provide annotations to indicate fixture setup and teardown. If a test class implements the methods setup(), setupSpec(), cleanup(), and cleanupSpec() then Spock will automatically use the method body to handle fixtures. You can find more information about those methods in the documentation.
FixtureSetupCleanup.groovy
class FixtureSetupCleanup extends Specification {
@Subject def fileReader = new DefaultFileReader()
def testFile
def setup() {
testFile = Files.createTempFile("junit5", ".tmp")
}
def cleanup() {
testFile.toFile().delete()
}
def "can read file"() {
given:
def text = "hello"
testFile << text
when:
def content = fileReader.readContent(testFile)
then:
content == text
}
}Listing 7. Fixture methods in Spock
Descriptive test names
Test cases should clearly indicate its intent. A camel cased method name proves to be a weak concept to describe its coverage. JUnit 5 and Spock support provide better ways to make test cases human-readable.
JUnit 5
In JUnit 4, you could only rely on the method name to identify the test case. JUnit 5 introduces the annotation @DisplayName that allows users to provide a description of its intent. Check out the user guide for more information on display names.
DescriptiveTest.java
import org.junit.jupiter.api.DisplayName;
public class DescriptiveTest {
private final ArithmeticOperation arithmeticOperation = new ArithmeticOperation();
@Test
@DisplayName("can add two numbers")
void canAdd() {
assertEquals(3, arithmeticOperation.add(1, 2));
}
@Test
@DisplayName("can substract a number from another one")
void canSubstract() {
assertEquals(1, arithmeticOperation.substract(2, 1));
}
@Test
@DisplayName("can multiple two numbers")
void canMultiply() {
assertEquals(6, arithmeticOperation.multiply(2, 3));
}
@Test
@DisplayName("can divide two numbers")
void canDivide() {
assertEquals(3, arithmeticOperation.divide(6, 2));
}
}Listing 8. Providing a descriptive test name by annotation
The executing environment takes the test description into account and uses it instead of the method name. The following screenshot shows the test execution in IntelliJ.
Figure 1. Descriptive test names in IntelliJ
Spock
Spock does not introduce an annotation to support descriptive test names. Instead it simply uses a Groovy language feature to do the heavy lifting. In Groovy, you can provide any String as method name.
DescriptiveTest.groovy
class DescriptiveTest extends Specification {
@Subject def arithmeticOperation = new ArithmeticOperation()
def "can add two numbers"() {
expect:
arithmeticOperation.add(1, 2) == 3
}
def "can substract a number from another one"() {
expect:
arithmeticOperation.substract(2, 1) == 1
}
def "can multiple two numbers"() {
expect:
arithmeticOperation.multiply(2, 3) == 6
}
def "can divide two numbers"() {
expect:
arithmeticOperation.divide(6, 2) == 3
}
}Listing 9. Using a String to provide a readable test description
The runtime environment properly evaluates the provided String as method name. The following screenshot shows how IntelliJ renders the executed Spock test cases.
Figure 2. Descriptive test names in IntelliJ
Disabling tests
Under certain conditions, you might want to disable single test methods. That’s typically the case if a test is failing temporarily or if a test covers functionality that hasn’t been implemented yet.
JUnit 5
JUnit 5 provides the annotation @Disabled to either disable all tests in a test class or individual test methods. A user can provide an optional reason to explain why the test was disabled. The following example demonstrates how to disable the test case canAdd(). More details can be found in the user guide.
IgnoredTest.java
import org.junit.jupiter.api.Disabled;
public class IgnoredTest {
private final ArithmeticOperation arithmeticOperation = new ArithmeticOperation();
@Test
@Disabled("for demonstration purposes")
void canAdd() {
assertEquals(3, arithmeticOperation.add(1, 2));
}
}Listing 10. Disabling test methods by annotation
Spock
Disabling a test in Spock follows the same pattern as JUnit 5. The API introduces the annotation @Ignore. The annotation can apply to a type or a method and allows for providing an optional reason. Groovy-based tests can also use the annotation @NotYetImplemented to pass a test if a test failure occurs for functionality that currently does not work but will be implemented in the future.
IgnoredTest.groovy
import spock.lang.Ignore
class IgnoredTest extends Specification {
@Subject def arithmeticOperation = new ArithmeticOperation()
@Ignore("for demonstration purposes")
def canAdd() {
expect:
arithmeticOperation.add(1, 2) == 3
}
}Listing 11. Using the Ignore annotation to disable a method
Expecting thrown exceptions
Methods can throw an exception. Being a good citizen you’ll want to test those cases as well to verify that the "sad path" behaves as expected.
JUnit 5
The JUnit 5 API provides fine-grained assertion methods for testing thrown exceptions. Any portions of code can be wrapped with the method assertThrows. The test will fail if the wrapped code block does not throw the expected exception. The user guide provides an even more elaborate sample than the one below.
ExpectedExceptionTest.java
import static org.junit.jupiter.api.Assertions.assertThrows;
public class ExpectedExceptionTest {
private final FileReader fileReader = new DefaultFileReader();
@Test
void cannotReadNonExistentFile() {
assertThrows(IOException.class, () -> {
fileReader.readContent(Paths.get("hello.text"));
});
}
}Listing 12. Asserting that a code block throws an expected exception
Spock
In Spock-based tests a class or method can be annotated with @FailsWith to signal that a declared exception should be thrown. If you want more fine-grained control over which portion of the code should throw the exception then you’ll have to implement a try/catch block and assert the exception type.
Alternatively, you can also use the method thrown(Class) to assert a thrown exception. The return value of the method grants access to the exception for further inspection.
ExpectedExceptionTest.java
import spock.lang.FailsWith
class ExpectedExceptionTest extends Specification {
@Subject def fileReader = new DefaultFileReader()
@FailsWith(IOException)
def "throws exception if file contents cannot be read"() {
expect:
fileReader.readContent(Paths.get('hello.text'))
}
def "throws exception if file contents cannot be read and assert message"() {
when:
fileReader.readContent(Paths.get('hello.text'))
then:
def t = thrown(IOException)
t.message == 'File does not exist'
}
}Listing 13. Declaring an expected exception by annotation
Repeating test execution
Sometimes you’ll want to verify that exercising the same logic multiples times leads to the coequal result. There are various examples that come to mind that require repeating a test:
-
Verifying that functionality is idempotent
-
Ensuring that a service endpoint can handle subsequent requests
-
Caching of data works as expected
JUnit 5
Test cases can be executed multiple times in a row by marking them with the @RepeatedTest annotation. Aside from declaring the number of repetitions you can also build a custom test name with the help of built-in variables. Check the user guide for more information.
RepetitionTest.java
import org.junit.jupiter.api.RepeatedTest;
public class RepetitionTest {
private final ArithmeticOperation arithmeticOperation = new ArithmeticOperation();
@RepeatedTest(10)
void canAdd() {
assertEquals(3, arithmeticOperation.add(1, 2));
}
@RepeatedTest(value = 5, name = "Iteration {currentRepetition} of {totalRepetitions}")
void canSubstract() {
assertEquals(1, arithmeticOperation.substract(2, 1));
}
}Listing 14. Repeating a test method by annotation
Spock
Spock does not support test repetition out-of-the-box. You will need to roll your own mechanism. The following example code uses a where statement to repeat the test execution multiple times.
RepetitionTest.groovy
class RepetitionTest extends Specification {
@Subject def arithmeticOperation = new ArithmeticOperation()
@Unroll
def "can add"() {
expect:
arithmeticOperation.add(1, 2) == 3
where:
i << (1..10)
}
@Unroll
def "Iteration #i of 5"() {
expect:
arithmeticOperation.substract(2, 1) == 1
where:
i << (1..5)
}
}Listing 15. Repeating a test method by counter
Declaring test execution timeouts
Code under test may take a little time to finish. Operations can be very costly, calls across network boundaries can take a longer than expected due to latency, load tests should finish in an predefined amount of time. JUnit 5 and Spock provide adequate support for declaring test execution timeout. The test fails if it doesn’t finish within the expected time frame.
JUnit 5
A timeout in JUnit 5 is represented by the assertion method assertTimeout. Timeout declarations expect a Duration instance plus the code block that should finish in the expected timespan. You can learn more about the timeout assertion method in the user guide.
RepetitionTest.java
import static java.time.Duration.ofSeconds;
import static org.junit.jupiter.api.Assertions.assertTimeout;
public class TimeoutTest {
private final ArithmeticOperation arithmeticOperation = new ArithmeticOperation();
@Test
void canAdd() {
assertTimeout(ofSeconds(2), () -> {
assertEquals(3, arithmeticOperation.add(1, 2));
});
}
}Listing 16. Fail a test if it doesn’t finish in expected timeout threshold
Spock
Spock takes the route of providing an annotation to declare a test execution timeout. The annotation can be assigned to the whole test class or just individual test methods. By default the assigned value declares a timeout in seconds. You can provide a different TimeUnit if needed.
RepetitionTest.groovy
import spock.lang.Timeout
class TimeoutTest extends Specification {
@Subject def arithmeticOperation = new ArithmeticOperation()
@Timeout(2)
def canAdd() {
expect:
arithmeticOperation.add(1, 2) == 3
}
}Listing 17. Convenient timeout definition by annotation
Conditional test execution
In a large test suite you don’t necessarily want to execute all tests in every runtime environment. For example some tests should just run in a Windows environment but not on Linux. Or you might implement tests that should only run with a specific JDK version. Being able to control test execution proves to be extremely valuable if your test coverage becomes more complex or multifaceted.
JUnit 5
JUnit 5 offers the assertion method assumeTrue. If the expression evaluates to true then any code that follows will be executed. Should the expression validate to false then any test logic that follows is skipped. Internally, JUnit throws an exception that needs to be handled by the runtime environment.
ConditionalExecutionTest.java
import static org.junit.jupiter.api.Assumptions.assumeTrue;
public class ConditionalExecutionTest {
private final static String SYS_PROP_KEY = "junit5.test.enabled";
private final static String SYS_PROP_TRUE_VALUE = "true";
private final ArithmeticOperation arithmeticOperation = new ArithmeticOperation();
@Test
void testOnlyOnSystemSystemPropertySet() {
assumeTrue(SYS_PROP_TRUE_VALUE.equals(System.getProperty(SYS_PROP_KEY)));
assertEquals(3, arithmeticOperation.add(1, 2));
}
}Listing 18. Conditional test execution using assumeTrue
IntelliJ properly handles conditional execution for JUnit 5. The console renders the message org.opentest4j.TestAbortedException: Assumption failed: assumption is not true for an expression that evaluates to false and records the method as skipped.
Figure 3. IntelliJ handles a TestAbortedException thrown by JUnit 5
For capturing more complex expressions, the JUnit API also allows for writing an implementation of ExecutionCondition. The interface represents an extension to the test framework and can be applied on the class- and method-level. For more information about extending the test framework, see the section below.
ConditionalExecutionTest.java
import org.junit.jupiter.api.extension.ConditionEvaluationResult;
import org.junit.jupiter.api.extension.ExecutionCondition;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.api.extension.ExtensionContext;
import static org.junit.jupiter.api.extension.ConditionEvaluationResult.disabled;
import static org.junit.jupiter.api.extension.ConditionEvaluationResult.enabled;
public class ConditionalExecutionTest {
...
@ExtendWith(SystemPropertyConditionalExtension.class)
@Test
void testOnlyOnSystemSystemPropertySetByExtension() {
assertEquals(3, arithmeticOperation.add(1, 2));
}
private static class SystemPropertyConditionalExtension
implements ExecutionCondition {
@Override
public ConditionEvaluationResult evaluateExecutionCondition(ExtensionContext
context) {
String sysPropValue = System.getProperty(SYS_PROP_KEY);
boolean enabled = SYS_PROP_TRUE_VALUE.equals(sysPropValue);
if (enabled) {
return enabled(String.format("System property '%s' evaluates to true",
SYS_PROP_KEY));
}
return disabled(String.format("System property '%s' evaluates to false",
SYS_PROP_KEY));
}
}
}Listing 19. Conditional test execution using ExecutionCondition
Spock
Spock really shines when it comes to conditional test execution. You can define the condition with the help of a closure return any arbitrary expression. The closure is assigned to either the annotation @Requires or @IgnoreIf. Overall Spock’s support for conditional test execution feels more flexible than JUnit’s capabilities. The following example shows the use of the @Requires annotation.
ConditionalExecutionTest.groovy
import spock.lang.Requires
class ConditionalExecutionTest extends Specification {
private final static String SYS_PROP_KEY = "spock.test.enabled"
private final static String SYS_PROP_TRUE_VALUE = "true"
@Subject def arithmeticOperation = new ArithmeticOperation()
@Requires({ SYS_PROP_TRUE_VALUE == sys[SYS_PROP_KEY] })
def "can add"() {
expect:
arithmeticOperation.add(1, 2) == 3
}
}Listing 20. Expression-based test execution defined as annotation parameter
The following screenshot shows the handling in IntelliJ.
Figure 4. Conditional test execution by annotation in IntelliJ
Data-driven tests
Production source code might behave differently at runtime if provided with varying inputs. Copy-pasting the same test case over and over again with different data sets leads to unmaintainable test classes. The better option is to run the same test case multiple times but with different inputs. Data-driven tests represent a feature I personally use in almost every project. I wouldn’t want to miss it.
JUnit 5
You have various options to feed data to a test case. First of all, you’ll need to mark the method with the annotation @ParameterizedTest to indicate that it receives data and should be executed multiple times. The annotation also allows for building a custom test description based on the parameter values.
One way to feed the data is with the help of the annotation @ValueSource. I don’t find this annotation very helpful as it can only provide a flat list of single input values. To make the functionality useful you’ll most likely also need to provide a second or third input value, a corresponding result value and potentially a textual description that represents the test case. The annotation @MethodSource can refer to a method returning a more complex matrix combination of values. Unfortunately, there’s no way to provide an identifier to any of the values making it hard to parse and understand.
DataDrivenTest.java
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;
import org.junit.jupiter.params.provider.ValueSource;
import java.util.stream.Stream;
public class DataDrivenTest {
private final ArithmeticOperation arithmeticOperation = new ArithmeticOperation();
@ParameterizedTest
@ValueSource(ints = { 1, 2, 3, 4, 5 })
void canAdd(int b) {
assertTrue(arithmeticOperation.add(1, b) >= 2);
}
@ParameterizedTest(name = "can add {0} to {1} and receive {2}")
@MethodSource("additionProvider")
void canAddAndAssertExactResult(int a, int b, int result) {
assertEquals(result, arithmeticOperation.add(a, b));
}
static Stream<Arguments> additionProvider() {
return Stream.of(
Arguments.of(1, 3, 4),
Arguments.of(3, 4, 7),
Arguments.of(10, 20, 30)
);
}
}Listing 21. Declaring a value source and provider to provide data to test
IntelliJ groups data-driven execution per test method. The result is very readable. Furthermore, the IDE also renders the type for each input value in parathesis next to the test name.
Figure 5. JUnit 5 data-driven test execution in IntelliJ
Spock
Spock solves data-driven tests in a very elegant and pragmatic way. The data is provided as a table with descriptive headers in a where clause. Each row in the data table represents a single execution of the test. Each named identifier can be referenced in the method name via the pound character (#) to build a meaningful test case description. The @Unroll annotation tells Spock to treat each test execution as individual test case. From my perspective data-driven tests are the Spock killer feature.
DataDrivenTest.groovy
import spock.lang.Unroll
class DataDrivenTest extends Specification {
@Subject def arithmeticOperation = new ArithmeticOperation()
@Unroll
def "can add"() {
expect:
arithmeticOperation.add(1, b) >= 2
where:
b << [1, 2, 3, 4, 5]
}
@Unroll
def "can add #a to #b and receive #result"() {
expect:
arithmeticOperation.add(a, b) == result
where:
a | b | result
1 | 3 | 4
3 | 4 | 7
10 | 20 | 30
}
}Listing 22. Providing data to test execution by table
For Spock-based tests IntelliJ only renders a flat list of test executions.
Figure 6. IntelliJ renders each data line for Spock-based tests
Mocking
Classes rarely work in isolation. Most classes call off to other class instances, services or subsystems. Mocking enabled you to cut off any integration point, replace it by a stand-in object and make it respond as needed for the test case.
JUnit 5
JUnit 5 does not ship with any mocking capabilities. Many users resort to the popular mocking library Mockito. The following example demonstrates the use of Mockito as part of a test case. The Mockito API calls nicely blend into the logic of the test case. Beginners to testing might have a hard time to visually separate test code from mocking expectations.
MockTest.java
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
public class MockTest {
private Path testFile;
@BeforeEach
void setup() throws IOException {
testFile = Files.createTempFile("junit5", ".tmp");
}
@AfterEach
void cleanup() {
testFile.toFile().delete();
}
@Test
void canMockFileReadOperation() throws IOException {
String text = "hello";
FileReader fileReader = mock(FileReader.class);
when(fileReader.readContent(testFile)).thenReturn(text);
FileManager fileManager = new DefaultFileManager(fileReader);
Files.write(testFile, text.getBytes());
assertEquals(text, fileManager.readContent(testFile));
}
}Listing 23. Using Mockito to inject mock instance
Spock
Mocks and stubs are an integrated part of the Spock API. You do not have to pull in yet another external dependency to fulfill all testing needs. Creating mock objects and defining expected interactions in Spock is straightforward. You can learn more about Spock’s interaction based testing in the user guide.
MockTest.groovy
class MockTest extends Specification {
def fileReader = Mock(FileReader)
@Subject def fileManager = new DefaultFileManager(fileReader)
def testFile
void setup() {
testFile = Files.createTempFile("junit5", ".tmp")
}
void cleanup() {
testFile.toFile().delete()
}
def "can mock file read operation"() {
given:
def text = "hello"
Files.write(testFile, text.getBytes())
when:
def content = fileManager.readContent(testFile)
then:
1 * fileReader.readContent(testFile) >> text
content == text
}
}Listing 24. Built-in mock capabilities
Labeling and filtering test execution
Grouping and executing tests based on functional boundaries becomes inevitable with a growing amount of coverage. For example you might want to split slow- from fast-running tests, unit from integration tests or set up a test suite for load testing. Both test frameworks provide sufficient tooling for labeling tests and running them in dedicated runs.
JUnit 5
In JUnit you can label tests with so-called tags. To tag a test class or method use the @Tag annotation and provide a short description as String. The example below separates slow- from fast-running tests.
TaggedTest.java
import org.junit.jupiter.api.Tag;
public class TaggedTest {
private final ArithmeticOperation arithmeticOperation = new ArithmeticOperation();
@Tag("slow")
@Test
void runsSlowly() {
assertEquals(3, arithmeticOperation.add(1, 2));
}
@Tag("fast")
@Test
void runsFast() {
assertEquals(1, arithmeticOperation.substract(2, 1));
}
}Listing 25. Tagged test methods
The description of a tagged test can serve as documentation for other developers. However, only in combination with test filtering, tagging becomes really beneficial. The following example sets up a test suite for running "fast" tests. Filtering tests requires the dependency org.junit.platform:junit-platform-runner to be available on the test compilation classpath.
FilteredFastTest.java
import org.junit.platform.runner.JUnitPlatform;
import org.junit.platform.suite.api.IncludeTags;
import org.junit.platform.suite.api.SelectPackages;
import org.junit.runner.RunWith;
@RunWith(JUnitPlatform.class)
@SelectPackages("com.bmuschko.test.comparison.junit5.tagged")
@IncludeTags("fast")
public class FilteredFastTest {
}Listing 26. Test suite executing fast tests
Spock
The Spock API does not define the concept of a tag. Nevertheless, it’s very easy to set up labeling yourself. The following two annotation classes serve the purpose of suggesting slow- and fast-running tests.
Slow.groovy
@Retention(RetentionPolicy.RUNTIME)
@Target([ElementType.TYPE, ElementType.METHOD])
@interface Slow {
}Listing 27. Interface annotation indicating slow-running tests
Fast.groovy
@Retention(RetentionPolicy.RUNTIME)
@Target([ElementType.TYPE, ElementType.METHOD])
@interface Fast {
}Listing 28. Interface annotation indicating fast-running tests
Labeling test case methods with the annotations is straightforward. The example below uses both custom annotations.
TaggedTest.groovy
class TaggedTest extends Specification {
@Subject def arithmeticOperation = new ArithmeticOperation()
@Slow
def "runs slowly"() {
expect:
arithmeticOperation.add(1, 2) == 3
}
@Fast
def "runs fast"() {
expect:
arithmeticOperation.substract(2, 1) == 1
}
}Listing 29. Annotated test methods
Labeled test cases can be filtered with the help of a Spock configuration file. Spock searches for a file with the name SpockConfig.groovy on the classpath. The configuration file can tell the test runner which test suite to execute.
SpockConfig.groovy
import com.bmuschko.test.comparison.spock.tagged.Fast
runner {
include Fast
}Listing 30. Spock runner configuration to filter annotated tests
The approach is less convenient than the one provided by JUnit 5 especially if you have to deal with more than one test suite. Please be aware that you can also provide the configuration to the test JVM with the system property spock.configuration.
Extending the test framework
JUnit 5 and Spock are powerful tools with a lot of built-in functionality. Nevertheless, a test framework cannot anticipate all requirements a real world project might have. You can enhance the base functionality with an extension in case you need custom logic that ties into the existing API. This blog post won’t go very deep into the available options for each test framework. I’d highly encourage you to explore the API for yourself.
The simple example below demonstrates basic extension capabilities. The implementation reacts to lifecycle events emitted before and after executing a test case. Each event logs the name of test method to the standard output stream.
JUnit 5
The test framework’s extension model provides callback interfaces to react to prominent lifecycle events. As you can see in listing 31, the extension class implements the interfaces BeforeTestExecutionCallback and AfterTestExecutionCallback. The callback interfaces provide a context object of type ExtensionContext giving you access to detailed information about the executed test method.
BeforeAfterLoggingExtension.java
import org.junit.jupiter.api.extension.AfterTestExecutionCallback;
import org.junit.jupiter.api.extension.BeforeTestExecutionCallback;
import org.junit.jupiter.api.extension.ExtensionContext;
public class BeforeAfterLoggingExtension implements BeforeTestExecutionCallback, AfterTestExecutionCallback {
@Override
public void beforeTestExecution(ExtensionContext context) {
Method testMethod = context.getRequiredTestMethod();
System.out.println(String.format("Starting test method %s.%s", testMethod.getDeclaringClass(), testMethod.getName()));
}
@Override
public void afterTestExecution(ExtensionContext context) {
Method testMethod = context.getRequiredTestMethod();
System.out.println(String.format("Finishing test method %s.%s", testMethod.getDeclaringClass(), testMethod.getName()));
}
}Listing 31. Test execution callback handler
Extensions can be applied to test classes or methods with the help of the annotation @ExtendWith. That’s it! You implemented your first extension.
LoggingTest.java
import org.junit.jupiter.api.extension.ExtendWith;
@ExtendWith(BeforeAfterLoggingExtension.class)
public class LoggingTest {
private final ArithmeticOperation arithmeticOperation = new ArithmeticOperation();
@Test
void canAdd() {
assertEquals(3, arithmeticOperation.add(1, 2));
}
}Listing 32. Using extension in test class
Spock
Spock takes a similar approach as JUnit 5. You need to implement the listener interface IRunListener to intercept test lifecycle events. Thankfully, the API already provides a stub implementation AbstractRunListener so you only have to implement the event methods you are interested in.
BeforeAfterEventListener.groovy
import org.spockframework.runtime.AbstractRunListener
import org.spockframework.runtime.model.FeatureInfo
class BeforeAfterEventListener extends AbstractRunListener {
@Override
void beforeFeature(FeatureInfo feature) {
println "Starting test method ${feature.description.className}.${feature.description.methodName}"
}
@Override
void afterFeature(FeatureInfo feature) {
println "Finishing test method ${feature.description.className}.${feature.description.methodName}"
}
}Listing 33. Event listener implementation in Spock
In our example, the extension should only be available for the test class. Listing 34 implements an annotation class for that very purpose.
BeforeAfterLogging.groovy
import org.spockframework.runtime.extension.ExtensionAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@ExtensionAnnotation(BeforeAfterLoggingExtension.class)
@interface BeforeAfterLogging {
}Listing 34. Exposing an annotation evaluated by extension
With the listener and the annotation in place, you can now create a new extension implementation. Spock provides a convenient abstract implementation named AbstractAnnotationDrivenExtension for registering the listener.
BeforeAfterLoggingExtension.groovy
import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension
import org.spockframework.runtime.model.SpecInfo
class BeforeAfterLoggingExtension extends AbstractAnnotationDrivenExtension<BeforeAfterLogging> {
@Override
void visitSpecAnnotation(BeforeAfterLogging annotation, SpecInfo spec) {
spec.addListener(new BeforeAfterEventListener())
}
}Listing 35. Registering the listener by extension
Any test class can use the annotation to automatically apply the logic implemented by the extension.
LoggingTest.groovy
@BeforeAfterLogging
class LoggingTest extends Specification {
@Subject def arithmeticOperation = new ArithmeticOperation()
def canAdd() {
expect:
arithmeticOperation.add(1, 2) == 3
}
}Listing 36. Using the logging extension in a test class
Conclusion
JUnit has come a long way since version 4. Not only does version 5 present an attractive feature set it also revolutionizes the underlying runtime platform. Almost all features available in Spock have an equivalent in JUnit 5 making it enterprise-ready tooling.
Given JUnit’s popularity I am confident that many organizations will adopt the latest version of the test framework while at the same time keeping legacy, JUnit 4-based tests runnable through the Jupiter Vintage test engine.
Personally, I like the BDD-style of Spock tests better. Test cases are more readable, self-explantory and just a pleasure to write. Data-driven tests and conditional test execution are features I use on a daily basis and hope JUnit 5 will catch up on them in a future version.