package calc
// Add two numbers.
// Return the result.
func Add(a, b int) int {
return a + b
}
// Subtract two numbers.
// Return the result.
func Subtract(a, b int) int {
return a - b
}
// Multiply two numbers.
// Return the result.
func Multiply(a, b int) int {
return a * b
}
// Divide two numbers.
// Return the result.
func Divide(a, b int) float64 {
return float64(a / b)
}golang unit integration testing assertion tdd bdd June 13, 2018
Recently, JetBrains ran a survey on the state of developer ecosystems. As part of the survey, they asked Go developers about their tools of choice. As I was reading through it, the section on testing frameworks caught my eye. So far I had only used the built-in Go testing support and Testify for my own projects. Time to explore the testing landscape!
During my research I tried to identify the most popular packages and differentiate them in regards to feature set, usability and expressiveness. I did not take into account the topic of mocking as I believe that it deserves a more in-depth discussion.
For a quick reference you can directly refer to the relevant section in this blog post.
-
Go standard library: The
testingpackage that ships with the runtime. -
Testify: Assertion and mock helper functions.
-
gocheck: Assertion helper functions.
-
gopwt: Power assertion helper functions.
-
go-testdeep: Deep comparison helper functions.
-
Ginkgo and Gomega: A heavyweight BDD testing framework + assertion helpers.
-
Goblin: A Mocha-like BDD testing framework.
-
GoConvey: BDD testing framework with web UI.
You can find all code shown in the blog post in the respective GitHub repository.
Code under test
The code under test doesn’t have to be complicated to demonstrate the usage of testing frameworks. It’s represented by a simple calculator implementation providing exported functions for addition, subtration, multiplication and division.
calc.go
Listing 1. Simple calculator functions
The code shown in listing 1 sits in the package calc. All test code is going to live in a different package called calc_test to ensure that the code only verifies the public API. First, we’ll have a look at the testing support that comes with the Go standard library.
Using the Go standard library
There are many voices in the Go community that promote the usage of the testing package as sufficient for writing test code. Developers new to the project will quickly find their way around the code base. However, the drawback is that the code easily becomes repetitive and tedious to implement.
calc_standard_test.go
package calc_test
import (
. "github.com/bmuschko/go-testing-frameworks/calc"
"testing"
)
func TestAddWithTestingPackage(t *testing.T) {
result := Add(1, 2)
if result != 3 {
t.Errorf("Result was incorrect, got: %d, want: %d.", result, 3)
}
}
func TestSubtractWithTestingPackage(t *testing.T) {
result := Subtract(5, 3)
if result != 2 {
t.Errorf("Result was incorrect, got: %d, want: %d.", result, 2)
}
}
func TestMultiplyWithTestingPackage(t *testing.T) {
result := Multiply(5, 6)
if result != 30 {
t.Errorf("Result was incorrect, got: %d, want: %d.", result, 30)
}
}
func TestDivideWithTestingPackage(t *testing.T) {
result := Divide(10, 2)
if result != 5 {
t.Errorf("Result was incorrect, got: %f, want: %f.", result, float64(5))
}
}Listing 2. Using the Go standard testing package
The console output in verbose mode does the job but looks fairly bland.
$ go test -v ./calc === RUN TestAddWithTestingPackage --- PASS: TestAddWithTestingPackage (0.00s) === RUN TestSubtractWithTestingPackage --- PASS: TestSubtractWithTestingPackage (0.00s) === RUN TestMultiplyWithTestingPackage --- PASS: TestMultiplyWithTestingPackage (0.00s) === RUN TestDivideWithTestingPackage --- PASS: TestDivideWithTestingPackage (0.00s) PASS ok github.com/bmuschko/go-testing-frameworks/calc 0.007s
From my perspective, there’s nothing wrong with reusing existing functionality even though it requires an external library or may steepen the learning curve. I would expect that most Go developers are familiar with at least one assertion helper library or a more intricate testing framework. Let’s have a look at Testify, the most popular assertion library.
Simplifying assertions with Testify
Testify is an assertion and mocking toolkit that plays nicely with the standard library. The assert package provides helpful functions for asserting the expected outcome of a test case. Optionally, you can also provide a helpful failure description.
Listing 3 clearly outlines the seamless integration with the standard testing support and the improved readability of the test code. Executing the test code produces the exact same console output as created by the standard library testing support.
calc_testify_test.go
package calc_test
import (
. "github.com/bmuschko/go-testing-frameworks/calc"
. "github.com/stretchr/testify/assert"
"testing"
)
func TestAddWithTestify(t *testing.T) {
result := Add(1, 2)
Equal(t, 3, result)
}
func TestSubtractWithTestify(t *testing.T) {
result := Subtract(5, 3)
Equal(t, 2, result)
}
func TestMultiplyWithTestify(t *testing.T) {
result := Multiply(5, 6)
Equal(t, 30, result)
}
func TestDivideWithTestify(t *testing.T) {
result := Divide(10, 2)
Equal(t, float64(5), result)
}Listing 3. Using the assertion helpers provided by Testify
If you are used to testing frameworks in other languages e.g. JUnit then you might be wondering about set up and tear down functionality or a way to define a suite of tests. Testify supports these concepts with the help of the suite package. From my perspective the functionality of the library is easy to grasp and use.
The alternative assertion library: gocheck
The library gocheck offers similar functionality to Testify. It’s a testing framework with support for rich assertions, the definition of test suites and fixture callbacks. The two features I really like in gocheck are explicit test skipping and the ability to select test execution from the command line via filters. In listing 4, you can see a multiple test cases written with gocheck.
calc_gocheck_test.go
package calc_test
import (
. "github.com/bmuschko/go-testing-frameworks/calc"
. "github.com/go-check/check"
"testing"
)
func Test(t *testing.T) {
TestingT(t)
}
type MySuite struct{}
var _ = Suite(&MySuite{})
func (s *MySuite) TestAddWithGocheck(c *C) {
result := Add(1, 2)
c.Assert(result, Equals, 3)
}
func (s *MySuite) TestSubtractWithGocheck(c *C) {
result := Subtract(5, 3)
c.Assert(result, Equals, 2)
}
func (s *MySuite) TestMultiplyWithGocheck(c *C) {
result := Multiply(5, 6)
c.Assert(result, Equals, 30)
}
func (s *MySuite) TestDivideWithGocheck(c *C) {
result := Divide(10, 2)
c.Assert(result, Equals, float64(5))
}Listing 4. Using the assertion helpers provided by gocheck
The testing framework condenses the usual test results in the console output. It’s unfortunate that the level of details cannot be controlled by a command line option.
$ go test -v ./calc === RUN Test OK: 4 passed --- PASS: Test (0.00s) PASS ok github.com/bmuschko/go-testing-frameworks/calc 0.007s
The last commit to the source code happened in 2016 which might indicate that active development stopped for good. Despite its complete feature set, I’d probably prefer Testify over gocheck as you can still expect bugfixes and propose feature requests.
Declarative failed assertions with gopwt
Identifying the root cause of a failed assertion can be tedious and may require additional debugging. A power assertion library helps with rendering the evaluated result of every portion of the equation as part of the error message. The library gopwt exposes a single function named OK for asserting an expression as shown in listing 5. You may have noticed that the test function TestAddWithGopwt expects an incorrect result and therefore should fail.
calc_gopwt_test.go
package calc_test
import (
. "github.com/bmuschko/go-testing-frameworks/calc"
. "github.com/ToQoz/gopwt/assert"
. "github.com/ToQoz/gopwt"
"testing"
"flag"
"os"
)
func TestMain(m *testing.M) {
flag.Parse()
Empower()
os.Exit(m.Run())
}
func TestAddWithGopwt(t *testing.T) {
result := Add(1, 2)
OK(t, 4 == result)
}
func TestSubtractWithGopwt(t *testing.T) {
result := Subtract(5, 3)
OK(t, 2 == result)
}
func TestMultiplyWithGopwt(t *testing.T) {
result := Multiply(5, 6)
OK(t, 30 == result)
}
func TestDivideWithGopwt(t *testing.T) {
result := Divide(10, 2)
OK(t, float64(5) == result)
}Listing 5. Power assertions with gopwt
Let’s give it a go. As you can see below, the console output clearly indicates the actual values of every variable used in the expression in the error message.
$ go test -v ./calc === RUN TestAddWithGopwt --- FAIL: TestAddWithGopwt (0.00s) assert.go:85: FAIL calc_gopwt_test.go:20 OK(t, 4 == result) | | | 3 false --- [int] result +++ [int] 4 @@ -1,1 +1,1@@ -3 +4 === RUN TestSubtractWithGopwt --- PASS: TestSubtractWithGopwt (0.00s) === RUN TestMultiplyWithGopwt --- PASS: TestMultiplyWithGopwt (0.00s) === RUN TestDivideWithGopwt --- PASS: TestDivideWithGopwt (0.00s) FAIL exit status 1 FAIL github.com/bmuschko/go-testing-frameworks/calc 0.006s exit status 1
Deep comparison with go-testdeep
Sometimes you’ll need to write assertions that compare values in data structures. The library go-testdeep offers flexible operators to avoid having to repeat common comparison functionality. For example, you can compare the contents of a map or check if the value of a number is greater or equals than the expected value. The calculator functions under test are not the best show case for demonstrating go-testdeep’s capabilities, however, listing 6 uses some of its comparison functions in action.
calc_gotestdeep_test.go
package calc_test
import (
. "github.com/bmuschko/go-testing-frameworks/calc"
. "github.com/maxatome/go-testdeep"
"testing"
)
func TestAddWithGoTestDeep(t *testing.T) {
result := Add(1, 2)
CmpNotZero(t, result)
CmpDeeply(t, &result, Ptr(3))
CmpDeeply(t, result, Code(func (r int) (bool, string) {
if r == 3 {
return true, ""
}
return false, "Result should be 3"
}))
}
...Listing 6. Power assertions with go-testdeep
Upon encountering a failure, the rendered error messages tries to provide a useful message including the received and expected value.
$ go test -v ./calc === RUN TestAddWithGoTestDeep --- FAIL: TestAddWithGoTestDeep (0.00s) calc_gotestdeep_test.go:12: Failed test *DATA: values differ got: (int) 3 expected: (int) 2 [under TestDeep operator Ptr at calc_gotestdeep_test.go:12] FAIL exit status 1 FAIL github.com/bmuschko/go-testing-frameworks/calc 0.028s
Similar to testify, go-testdeep is one of those libraries you can just add to your tool chain when using the standard testing package. If you are sick and tired of writing the same comparison logic over and over again, then go-testdeep might be for you.
In the next section, we’ll have a look at BDD-style testing frameworks.
Expressive BDD-style tests with Ginkgo and Gomega
The BDD testing style is known for its expressiveness and readability. The most well-known BDD testing framework in the Go world is Ginkgo. There’s slightly more code you’ll have to write with Ginkgo compared to the tests that only use an assertion helper library. However, it pays back with a given/when/then pattern that should be understandable by even non-programmers.
Ginkgo really shines when used in conjunction with its sister project, the assertion library Gomega. It allows for writing builder-style assertion logic that can even be used independent of Gingko. It reads like natural language and is easy to write. Additionally, Gomega provides a shortcut symbol for writing assertion via the Ω notation. The notation may align well from a branding perspective but I am personally not a fan of using it as it makes the syntax more obscure and harder to parse. Listing 7 shows test cases written with Ginkgo and the Gomega Expect notation.
calc_ginkgo_test.go
package calc_test
import (
. "github.com/bmuschko/go-testing-frameworks/calc"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
)
func TestCalc(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Calculator Suite")
}
var _ = Describe("Calculator", func() {
Describe("Add numbers", func() {
Context("1 and 2", func() {
It("should be 3", func() {
Expect(Add(1, 2)).To(Equal(3))
})
})
})
Describe("Subtract numbers", func() {
Context("3 from 5", func() {
It("should be 2", func() {
Expect(Subtract(5, 3)).To(Equal(2))
})
})
})
Describe("Multiply numbers", func() {
Context("5 with 6", func() {
It("should be 30", func() {
Expect(Multiply(5, 6)).To(Equal(30))
})
})
})
Describe("Divide numbers", func() {
Context("10 by 2", func() {
It("should be 30", func() {
Expect(Divide(10, 2)).To(Equal(float64(5)))
})
})
})
})Listing 7. Using the BDD framework Ginkgo and Gomega
Upon execution, Ginkgo renders color-coded console output which makes it extremely easy to parse the results.
Figure 1. Console output from Ginkgo
Ginkgo and Gomega have great documentation which makes it real easy for beginners. If you prefer writing BDD-style tests with an expansive feature set then Ginkgo is for you.
Simple and flexible BDD-style testing with Goblin
Goblin is another BDD-style testing framework for Go. Despite its fairly simplistic API, Goblin makes it easy to write declarative and expressive test cases. You don’t necessarily need to go with its built-in assertion functions. Alternatively, you can plug in other assertion libraries e.g. Gomega.
calc_goblin_test.go
import (
. "github.com/bmuschko/go-testing-frameworks/calc"
. "github.com/franela/goblin"
"testing"
)
func TestCalculator(t *testing.T) {
g := Goblin(t)
g.Describe("Calculator", func() {
g.It("should add two numbers ", func() {
g.Assert(Add(1, 2)).Equal(3)
})
g.It("should subtract two numbers", func() {
g.Assert(Subtract(5, 3)).Equal(2)
})
g.It("should multiply two numbers", func() {
g.Assert(Multiply(5, 6)).Equal(30)
})
g.It("should divide two numbers", func() {
g.Assert(Divide(10, 2)).Equal(float64(5))
})
})
}Listing 8. Using Goblin test framework
What I really like about Goblin is the colored console output which looks similar to the one provided by Mocha, a JavaScript-based testing framework. Figure 2 shows the output for a successful execution of calc_goblin_test.go.
Figure 2. Console output from Goblin
Live test reporting with GoConvey
Good reporting is a must-have for projects with a large test suite. GoConvey is yet another BDD-style testing framework. See listing 9 for a short example.
calc_goconvey_test.go
import (
. "github.com/bmuschko/go-testing-frameworks/calc"
. "github.com/smartystreets/goconvey/convey"
"testing"
)
func TestAddWithGoConvey(t *testing.T) {
Convey("Adding two numbers", t, func() {
x := 1
y := 2
Convey("should produce the expected result", func() {
So(Add(x, y), ShouldEqual, 3)
})
})
}
func TestSubtractWithGoConvey(t *testing.T) {
Convey("Subtracting two numbers", t, func() {
x := 5
y := 3
Convey("should produce the expected result", func() {
So(Subtract(x, y), ShouldEqual, 2)
})
})
}
func TestMultiplyWithGoConvey(t *testing.T) {
Convey("Multiplying two numbers", t, func() {
x := 5
y := 6
Convey("should produce the expected result", func() {
So(Multiply(x, y), ShouldEqual, 30)
})
})
}
func TestDivideWithGoConvey(t *testing.T) {
Convey("Dividing two numbers", t, func() {
x := 10
y := 2
Convey("should produce the expected result", func() {
So(Divide(x, y), ShouldEqual, float64(5))
})
})
}Listing 9. Using the BDD testing framework GoConvey
GoConvey comes with an extremely useful web interface. You can start it from the console by executing the command goconvey. Navigating to localhost:8080 shows an up to date view of your test result as shown in figure 3.
Figure 3. Live execution and rendering of test results with GoConvey
Summary
Many Go developers are content with the standard testing support. Nevertheless, the Go ecosystem produced a wealth of testing libraries that go far beyond the basics. You can avoid writing unnecessary boilerplate code with the help of an assertion library. My personal favorite testing frameworks in this category are Testify and gopwt.
BDD-style testing frameworks can further improve the expressiveness of your test code. I like Goblin for its simplicity and feature-rich console output. For larger projects, I’d probably go with Ginkgo.
Which testing framework do you prefer? I’d love to hear your opinions.