Exploring the landscape of Go testing frameworks

golang unit integration testing assertion tdd bdd


June 13, 2018

Recently, JetBrains ran a survey on the state of developer ecosystems. As part of the survey, they asked Go developers about their tools of choice. As I was reading through it, the section on testing frameworks caught my eye. So far I had only used the built-in Go testing support and Testify for my own projects. Time to explore the testing landscape!

During my research I tried to identify the most popular packages and differentiate them in regards to feature set, usability and expressiveness. I did not take into account the topic of mocking as I believe that it deserves a more in-depth discussion.

For a quick reference you can directly refer to the relevant section in this blog post.

  • Go standard library: The testing package that ships with the runtime.

  • Testify: Assertion and mock helper functions.

  • gocheck: Assertion helper functions.

  • gopwt: Power assertion helper functions.

  • Ginkgo and Gomega: A heavyweight BDD testing framework + assertion helpers.

  • Goblin: A Mocha-like BDD testing framework.

  • GoConvey: BDD testing framework with web UI.

You can find all code shown in the blog post in the respective GitHub repository.


Code under test

The code under test doesn’t have to be complicated to demonstrate the usage of testing frameworks. It’s represented by a simple calculator implementation providing exported functions for addition, subtration, multiplication and division.

calc.go

package calc

// Add two numbers.
// Return the result.
func Add(a, b int) int {
	return a + b
}

// Subtract two numbers.
// Return the result.
func Subtract(a, b int) int {
	return a - b
}

// Multiply two numbers.
// Return the result.
func Multiply(a, b int) int {
	return a * b
}

// Divide two numbers.
// Return the result.
func Divide(a, b int) float64 {
	return float64(a / b)
}

Listing 1. Simple calculator functions

The code shown in listing 1 sits in the package calc. All test code is going to live in a different package called calc_test to ensure that the code only verifies the public API. First, we’ll have a look at the testing support that comes with the Go standard library.


Using the Go standard library

There are many voices in the Go community that promote the usage of the testing package as sufficient for writing test code. Developers new to the project will quickly find their way around the code base. However, the drawback is that the code easily becomes repetitive and tedious to implement.

calc_standard_test.go

package calc_test

import (
	. "github.com/bmuschko/go-testing-frameworks/calc"
	"testing"
)

func TestAddWithTestingPackage(t *testing.T) {
	result := Add(1, 2)

	if result != 3 {
		t.Errorf("Result was incorrect, got: %d, want: %d.", result, 3)
	}
}

func TestSubtractWithTestingPackage(t *testing.T) {
	result := Subtract(5, 3)

	if result != 2 {
		t.Errorf("Result was incorrect, got: %d, want: %d.", result, 2)
	}
}

func TestMultiplyWithTestingPackage(t *testing.T) {
	result := Multiply(5, 6)

	if result != 30 {
		t.Errorf("Result was incorrect, got: %d, want: %d.", result, 30)
	}
}

func TestDivideWithTestingPackage(t *testing.T) {
	result := Divide(10, 2)

	if result != 5 {
		t.Errorf("Result was incorrect, got: %f, want: %f.", result, float64(5))
	}
}

Listing 2. Using the Go standard testing package

The console output in verbose mode does the job but looks fairly bland.

$ go test -v ./calc
=== RUN   TestAddWithTestingPackage
--- PASS: TestAddWithTestingPackage (0.00s)
=== RUN   TestSubtractWithTestingPackage
--- PASS: TestSubtractWithTestingPackage (0.00s)
=== RUN   TestMultiplyWithTestingPackage
--- PASS: TestMultiplyWithTestingPackage (0.00s)
=== RUN   TestDivideWithTestingPackage
--- PASS: TestDivideWithTestingPackage (0.00s)
PASS
ok  	github.com/bmuschko/go-testing-frameworks/calc	0.007s

From my perspective, there’s nothing wrong with reusing existing functionality even though it requires an external library or may steepen the learning curve. I would expect that most Go developers are familiar with at least one assertion helper library or a more intricate testing framework. Let’s have a look at Testify, the most popular assertion library.


Simplifying assertions with Testify

Testify is an assertion and mocking toolkit that plays nicely with the standard library. The assert package provides helpful functions for asserting the expected outcome of a test case. Optionally, you can also provide a helpful failure description.

Listing 3 clearly outlines the seamless integration with the standard testing support and the improved readability of the test code. Executing the test code produces the exact same console output as created by the standard library testing support.

calc_testify_test.go

package calc_test

import (
	. "github.com/bmuschko/go-testing-frameworks/calc"
	. "github.com/stretchr/testify/assert"
	"testing"
)

func TestAddWithTestify(t *testing.T) {
	result := Add(1, 2)
	Equal(t, 3, result)
}

func TestSubtractWithTestify(t *testing.T) {
	result := Subtract(5, 3)
	Equal(t, 2, result)
}

func TestMultiplyWithTestify(t *testing.T) {
	result := Multiply(5, 6)
	Equal(t, 30, result)
}

func TestDivideWithTestify(t *testing.T) {
	result := Divide(10, 2)
	Equal(t, float64(5), result)
}

Listing 3. Using the assertion helpers provided by Testify

If you are used to testing frameworks in other languages e.g. JUnit then you might be wondering about set up and tear down functionality or a way to define a suite of tests. Testify supports these concepts with the help of the suite package. From my perspective the functionality of the library is easy to grasp and use.


The alternative assertion library: gocheck

The library gocheck offers similar functionality to Testify. It’s a testing framework with support for rich assertions, the definition of test suites and fixture callbacks. The two features I really like in gocheck are explicit test skipping and the ability to select test execution from the command line via filters. In listing 4, you can see a multiple test cases written with gocheck.

calc_gocheck_test.go

package calc_test

import (
	. "github.com/bmuschko/go-testing-frameworks/calc"
	. "github.com/go-check/check"
	"testing"
)

func Test(t *testing.T) {
	TestingT(t)
}

type MySuite struct{}

var _ = Suite(&MySuite{})

func (s *MySuite) TestAddWithGocheck(c *C) {
	result := Add(1, 2)
	c.Assert(result, Equals, 3)
}

func (s *MySuite) TestSubtractWithGocheck(c *C) {
	result := Subtract(5, 3)
	c.Assert(result, Equals, 2)
}

func (s *MySuite) TestMultiplyWithGocheck(c *C) {
	result := Multiply(5, 6)
	c.Assert(result, Equals, 30)
}

func (s *MySuite) TestDivideWithGocheck(c *C) {
	result := Divide(10, 2)
	c.Assert(result, Equals, float64(5))
}

Listing 4. Using the assertion helpers provided by gocheck

The testing framework condenses the usual test results in the console output. It’s unfortunate that the level of details cannot be controlled by a command line option.

$ go test -v ./calc
=== RUN   Test
OK: 4 passed
--- PASS: Test (0.00s)
PASS
ok  	github.com/bmuschko/go-testing-frameworks/calc	0.007s

The last commit to the source code happened in 2016 which might indicate that active development stopped for good. Despite its complete feature set, I’d probably prefer Testify over gocheck as you can still expect bugfixes and propose feature requests.


Declarative failed assertions with gopwt

Identifying the root cause of a failed assertion can be tedious and may require additional debugging. A power assertion library helps with rendering the evaluated result of every portion of the equation as part of the error message. The library gopwt exposes a single function named OK for asserting an expression as shown in listing 5. You may have noticed that the test function TestAddWithGopwt expects an incorrect result and therefore should fail.

calc_gopwt_test.go

package calc_test

import (
	. "github.com/bmuschko/go-testing-frameworks/calc"
	. "github.com/ToQoz/gopwt/assert"
	. "github.com/ToQoz/gopwt"
	"testing"
	"flag"
	"os"
)

func TestMain(m *testing.M) {
	flag.Parse()
	Empower()
	os.Exit(m.Run())
}

func TestAddWithGopwt(t *testing.T) {
	result := Add(1, 2)
	OK(t, 4 == result)
}

func TestSubtractWithGopwt(t *testing.T) {
	result := Subtract(5, 3)
	OK(t, 2 == result)
}

func TestMultiplyWithGopwt(t *testing.T) {
	result := Multiply(5, 6)
	OK(t, 30 == result)
}

func TestDivideWithGopwt(t *testing.T) {
	result := Divide(10, 2)
	OK(t, float64(5) == result)
}

Listing 5. Power assertions with gopwt

Let’s give it a go. As you can see below, the console output clearly indicates the actual values of every variable used in the expression in the error message.

$ go test -v ./calc
=== RUN   TestAddWithGopwt
--- FAIL: TestAddWithGopwt (0.00s)
	assert.go:85: FAIL calc_gopwt_test.go:20
		OK(t, 4 == result)
		        |  |
		        |  3
		        false

		--- [int] result
		+++ [int] 4
		@@ -1,1 +1,1@@
		-3
		+4


=== RUN   TestSubtractWithGopwt
--- PASS: TestSubtractWithGopwt (0.00s)
=== RUN   TestMultiplyWithGopwt
--- PASS: TestMultiplyWithGopwt (0.00s)
=== RUN   TestDivideWithGopwt
--- PASS: TestDivideWithGopwt (0.00s)
FAIL
exit status 1
FAIL	github.com/bmuschko/go-testing-frameworks/calc	0.006s
exit status 1

In the next section, we’ll have a look at BDD-style testing frameworks.


Expressive BDD-style tests with Ginkgo and Gomega

The BDD testing style is known for its expressiveness and readability. The most well-known BDD testing framework in the Go world is Ginkgo. There’s slightly more code you’ll have to write with Ginkgo compared to the tests that only use an assertion helper library. However, it pays back with a given/when/then pattern that should be understandable by even non-programmers.

Ginkgo really shines when used in conjunction with its sister project, the assertion library Gomega. It allows for writing builder-style assertion logic that can even be used independent of Gingko. It reads like natural language and is easy to write. Additionally, Gomega provides a shortcut symbol for writing assertion via the Ω notation. The notation may align well from a branding perspective but I am personally not a fan of using it as it makes the syntax more obscure and harder to parse. Listing 6 shows test cases written with Ginkgo and the Gomega Expect notation.

calc_ginkgo_test.go

package calc_test

import (
	. "github.com/bmuschko/go-testing-frameworks/calc"
	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
)

func TestCalc(t *testing.T) {
	RegisterFailHandler(Fail)
	RunSpecs(t, "Calculator Suite")
}

var _ = Describe("Calculator", func() {
	Describe("Add numbers", func() {
		Context("1 and 2", func() {
			It("should be 3", func() {
				Expect(Add(1, 2)).To(Equal(3))
			})
		})
	})

	Describe("Subtract numbers", func() {
		Context("3 from 5", func() {
			It("should be 2", func() {
				Expect(Subtract(5, 3)).To(Equal(2))
			})
		})
	})

	Describe("Multiply numbers", func() {
		Context("5 with 6", func() {
			It("should be 30", func() {
				Expect(Multiply(5, 6)).To(Equal(30))
			})
		})
	})

	Describe("Divide numbers", func() {
		Context("10 by 2", func() {
			It("should be 30", func() {
				Expect(Divide(10, 2)).To(Equal(float64(5)))
			})
		})
	})
})

Listing 6. Using the BDD framework Ginkgo and Gomega

Upon execution, Ginkgo renders color-coded console output which makes it extremely easy to parse the results.

ginkgo console output

Figure 1. Console output from Ginkgo

Ginkgo and Gomega have great documentation which makes it real easy for beginners. If you prefer writing BDD-style tests with an expansive feature set then Ginkgo is for you.


Simple and flexible BDD-style testing with Goblin

Goblin is another BDD-style testing framework for Go. Despite its fairly simplistic API, Goblin makes it easy to write declarative and expressive test cases. You don’t necessarily need to go with its built-in assertion functions. Alternatively, you can plug in other assertion libraries e.g. Gomega.

calc_goblin_test.go

import (
	. "github.com/bmuschko/go-testing-frameworks/calc"
	. "github.com/franela/goblin"
	"testing"
)

func TestCalculator(t *testing.T) {
	g := Goblin(t)
	g.Describe("Calculator", func() {
		g.It("should add two numbers ", func() {
			g.Assert(Add(1, 2)).Equal(3)
		})

		g.It("should subtract two numbers", func() {
			g.Assert(Subtract(5, 3)).Equal(2)
		})

		g.It("should multiply two numbers", func() {
			g.Assert(Multiply(5, 6)).Equal(30)
		})

		g.It("should divide two numbers", func() {
			g.Assert(Divide(10, 2)).Equal(float64(5))
		})
	})
}

Listing 7. Using Goblin test framework

What I really like about Goblin is the colored console output which looks similar to the one provided by Mocha, a JavaScript-based testing framework. Figure 2 shows the output for a successful execution of calc_goblin_test.go.

goblin console output

Figure 2. Console output from Goblin


Live test reporting with GoConvey

Good reporting is a must-have for projects with a large test suite. GoConvey is yet another BDD-style testing framework. See listing 8 for a short example.

calc_goconvey_test.go

import (
	. "github.com/bmuschko/go-testing-frameworks/calc"
	. "github.com/smartystreets/goconvey/convey"
	"testing"
)

func TestAddWithGoConvey(t *testing.T) {
	Convey("Adding two numbers", t, func() {
		x := 1
		y := 2

		Convey("should produce the expected result", func() {
			So(Add(x, y), ShouldEqual, 3)
		})
	})
}

func TestSubtractWithGoConvey(t *testing.T) {
	Convey("Subtracting two numbers", t, func() {
		x := 5
		y := 3

		Convey("should produce the expected result", func() {
			So(Subtract(x, y), ShouldEqual, 2)
		})
	})
}

func TestMultiplyWithGoConvey(t *testing.T) {
	Convey("Multiplying two numbers", t, func() {
		x := 5
		y := 6

		Convey("should produce the expected result", func() {
			So(Multiply(x, y), ShouldEqual, 30)
		})
	})
}

func TestDivideWithGoConvey(t *testing.T) {
	Convey("Dividing two numbers", t, func() {
		x := 10
		y := 2

		Convey("should produce the expected result", func() {
			So(Divide(x, y), ShouldEqual, float64(5))
		})
	})
}

Listing 8. Using the BDD testing framework GoConvey

GoConvey comes with an extremely useful web interface. You can start it from the console by executing the command goconvey. Navigating to localhost:8080 shows an up to date view of your test result as shown in figure 3.

goconvey web ui

Figure 3. Live execution and rendering of test results with GoConvey

Summary

Many Go developers are content with the standard testing support. Nevertheless, the Go ecosystem produced a wealth of testing libraries that go far beyond the basics. You can avoid writing unnecessary boilerplate code with the help of an assertion library. My personal favorite testing frameworks in this category are Testify and gopwt.

BDD-style testing frameworks can further improve the expressiveness of your test code. I like Goblin for its simplicity and feature-rich console output. For larger projects, I’d probably go with Ginkgo.

Which testing framework do you prefer? I’d love to hear your opinions.



comments powered by Disqus