Building Go with Gradle

build golang gradle

July 29, 2017

Introduction

In July 2017 Google Go made a big jump on the TIOBE index. It’s now ranked among the top 10 most popular programming languages. With the rise of Moby aka Docker, Kubernetes and InfluxDB the language has become the go-to tool in the DevOps space. The complexity of automating the process of building, assembling and distributing the source code and binaries for any medium- to large-sized project is high. It’s somewhat shocking to see that the predominant tooling of automating in Go is still a mixture of Make files and shell scripts as it can be observed in the Moby and Kubernetes code base.

Make files and shell script can be powerful tools but they do not provide any support for strong modeling of a domain, are hard to maintain and do not provide any means to testing the automation code.

In this post I’d like to identify if Gradle can live up to the game. We’ll look at a simple Go project, some typical challenges and tasks encountered when building this project and how Gradle can help to automate the process.

How can Gradle help?

go mini Gradle currently does not provide a standard way to build Go project with its core distribution. However, users can write plugins to enhance the functionality by plugins to model new domains.

GoGradle is a plugin that helps with compiling, testing and assembling Go projects. At this year’s Gradle Summit, the plugin was awarded the Gradle plugin of the year 2017. For a deep dive on the plugin functionality check out the recording of the Summit talk on GoGradle.

Disclaimer: I found another Gradle plugin for building Go project but did not have a chance to compare the functionality and their implementation approaches.

The sample project

For the purpose of demonstrating the functionality provided by the GoGradle plugin, we’ll have a look at a Go project called Link Verifier, a program run from the command line. Functionally, Link Verifier recursively iterates over a given directory and identifies plain-text mark-up files like AsciiDoc and Markdown. For each of the found documents, the program extracts URLs and verifies that they can be resolved by executing a HTTP call.

The project depends on external Go packages: xurls for extract URLs out of a text document and testify for conviently making assertions in test code. Both libraries can be resolved with the help of Glide, a package manager for Go. A developer working on the project will have to install Glide to properly resolve the declared dependencies and their transitive dependencies.

The project uses Go’s built-in support for executing tests, the test command. Additionally, code coverage metrics are produced by configuring the The test command. The project uses Codecov to capture and visualize code coverage metrics over time.

The program was designed to run on different OSes e.g. Linux, Windows and MacOSX. For that purpose the project publishes prebuilt libraries with every single release. To adhere to the license agreements of external packages used by the project, I also wrote some logic for extracting the third-party license agreements and packaging them with the corresponding library in a TAR or ZIP file.

Most of the automation steps mentioned above can be executed by invoking a shell script. Every change made to the project runs through Travis CI. To reuse automation logic Travis directly calls the shell script checked into version control with the project’s source code.

Let’s see if the Gradle plugin can fulfill all of those requirements while at the same hiding and simplifying complex implementation logic which would otherwise lives in a shell script. You can find the source code describe below in a dedicated branch.

Initial setup

Getting started with the GoGradle plugin is easy. We just have to create a build.gradle file in the root directory of the existing project. In the build script, apply the plugin and provide some basic configuration that indicates the root path of the package used for the project. For the Link Verifier project, I assigned github.com/bmuschko/link-verifier to the packagePath property of the extension. All sections below directly refer to version 0.6.5. Please be aware that the behavior and/or the configuration options may change in future versions of the plugin.

By default the GoGradle plugin downloads the latest version of Go automatically and stores it in a temporary directory. Alternatively, a user can also configure a concrete Go version as needed. Please refer to the plugin documentation for more information.

build.gradle

plugins {
    id 'com.github.blindpirate.gogradle' version '0.6.5'
}

golang {
    packagePath = 'github.com/bmuschko/link-verifier'
}

Listing 1. Basic project setup

Now that we applied the plugin to our project, we have a bunch of useful tasks to our disposal. Probably the most important one is build which resolves all dependencies, compiles the code, runs the tests and assembles the binaries.

Executing the task with the build script shown in listing 1 fails compilation. Obviously, the project is missing external dependencies referenced in source file. In the next step, we’ll fix the issue.

Managing and resolving dependencies

Glide is my package management tool of choice for building Go projects. The project uses the libraries xurls and testify with a specific version to ensure reproducibility. Listing 2 shows those dependencies in the YAML format processed by Glide.

glide.yaml

package: .
import:
- package: github.com/mvdan/xurls
  version: 1.1.0
testImport:
- package: github.com/stretchr/testify
  version: 1.1.4

Listing 2. The Glide dependency definition file

As you might know Go resolves external packages from Git repositories. In file named glide.lock, Glide maps the concrete version of an external dependency to a Git commit hash of the repository hosting the code. An example of such a lock file can be seen in listing 3.

glide.lock

hash: 806deb3bb1bb02051f152c49856cac37224f623247742a1b8c028b38dff21aef
updated: 2017-06-03T12:38:37.338393246-04:00
imports:
- name: github.com/mvdan/xurls
  version: d315b61cf6727664f310fa87b3197e9faf2a8513
testImports:
- name: github.com/stretchr/testify
  version: 69483b4bd14f5845b5a1e55bca19e954e827f1d0

Listing 3. The Glide dependency lock file

The latest version of Gradle, 4.0, does not support resolving dependencies from a Git repository. So how do we make sure that Gradle understands the information? GoGradle enhances the standard way of declaring dependencies in Gradle. By applying the plugin users can declare Git-based dependencies with a help of a DSL. The exposed DSL seamlessly blends into the existing dependency management DSL as shown in listing 4.

build.gradle

dependencies {
    golang {
        build name:'github.com/mvdan/xurls', version:'d315b61cf6727664f310fa87b3197e9faf2a8513'
        test name:'github.com/stretchr/testify', version:'69483b4bd14f5845b5a1e55bca19e954e827f1d0'
    }
}

Listing 4. Gradle dependency definitions derived from Glide yaml file

As you can imagine typing down the dependency declaration manually is somewhat painful. Thankfully, the plugin introduces the convenience task init. The task derives the information declared in the Glide lock file and translates it into Gradle-based configuration. I’d expect that VCS-based repository formats are going to be introduced by Gradle core natively in the future.

The functionality of the init task is not limited to Glide. It also understands a variety of other Go package management tools. Please refer to the documentation to identify if your package manager is supported by GoGradle.

With all the dependencies in place, the project is able to run through the compilation and test steps. Let’s also have a closer look at the testing capabilities of the plugin.

Executing tests

GoGradle’s testing support follows the same conventions you might know from Java-based projects. The test task provided by the plugin finds all Go files that follow the file name convention <package>_test.go. GoGradle properly detects and runs all tests in the project as shown in the following console output:

$ gradle test

> Task :prepare
Found go 1.8.3 in /usr/local/go/bin/go, use it.
Use project GOPATH: /Users/bmuschko/dev/projects/gradle-playground/link-verifier/.gogradle/project_gopath

> Task :test
Test for github.com/bmuschko/link-verifier/stat finished, 4 completed, 0 failed
Test for github.com/bmuschko/link-verifier/text finished, 6 completed, 0 failed
Test for github.com/bmuschko/link-verifier/http finished, 3 completed, 0 failed
Test for github.com/bmuschko/link-verifier/file finished, 6 completed, 0 failed

BUILD SUCCESSFUL in 3s
5 actionable tasks: 3 executed, 2 up-to-date

By default the task automatically generates test reports as well as coverage profile data. Coverage metrics can be found in the directory .gogradle/reports. Figure 1 shows a sample report for the project.

Test report Figure 1. HTML test report

At the time of writing, the plugin only creates one coverage profile data file per package. Aggregating individual coverage files requires writing some custom logic. Listing 5 demonstrates a simple solution to the problem. I expect this functionality to be worked into the plugin sooner or later.

buildSrc/src/main/groovy/com/bmuschko/linkverifier/AggregateCoverage.groovy

package com.bmuschko.linkverifier

import org.gradle.api.DefaultTask
import org.gradle.api.tasks.TaskAction
import org.gradle.api.tasks.InputDirectory
import org.gradle.api.tasks.OutputFile

class AggregateCoverage extends DefaultTask {
    @InputDirectory
    File inputDir = project.file('.gogradle/reports/coverage/profiles')

    @OutputFile
    File outputFile = project.file("$project.buildDir/reports/coverage-aggregate/coverage.txt")

    AggregateCoverage() {
        group = 'GoGradle'
        description = 'Aggregates coverage profile data.'
    }

    @TaskAction
    void aggregate() {
        StringBuilder aggregatedCoverage = new StringBuilder()

        inputDir.listFiles().each {
            aggregatedCoverage << it.text
        }

        outputFile.text = aggregatedCoverage.toString()
    }
}

Listing 5. Custom task for aggregating coverage profiles

With the custom task implementation in place, the build script can be enhanced by the task aggregateCoverage. As shown in listing 6, necessary task dependencies have been established to hook into the typical lifecycle of a Go-based Gradle project.

build.gradle

import com.bmuschko.linkverifier.AggregateCoverage

task aggregateCoverage(type: AggregateCoverage) {
    mustRunAfter test
}

build.dependsOn aggregateCoverage

Listing 6. Hooking coverage aggregration task into task lifecycle

After exploring the plugin’s testing capabilities, let’s round out the discussion by having a look at the cross-compilation functionality.

Cross-compiling binaries

Link Verifier creates a matrix-combination of cross-compiled binary files. The plugin takes away the burden of having to implement the logic manually for every single project. With the help of the build task, a user can declare a list of target platforms as shown in listing 7.

build.gradle

build {
    targetPlatform = 'darwin-amd64, netbsd-amd64, netbsd-386, openbsd-amd64, openbsd-386, freebsd-amd64, freebsd-386, linux-amd64, linux-386, linux-arm, windows-amd64, windows-386'
}

Listing 7. Target platforms definitions

Upon execution, the task creates the resulting prebuilt libraries in the directory .gogradle. Each file name consists of the provided target platform and the name of the project. Currently, the project version is not taken into account.

.
└── .gogradle
    ├── darwin_amd64_link-verifier
    ├── freebsd_386_link-verifier
    ├── freebsd_amd64_link-verifier
    ├── linux_386_link-verifier
    ├── linux_amd64_link-verifier
    ├── linux_arm_link-verifier
    ├── netbsd_386_link-verifier
    ├── netbsd_amd64_link-verifier
    ├── openbsd_386_link-verifier
    ├── openbsd_amd64_link-verifier
    ├── windows_386_link-verifier
    └── windows_amd64_link-verifier

In my shell script, I also extracted all third-party licenses and bundled them with the binary in a tar.gz or .zip file. The plugin does not provide this functionality. However, it’s very easy to achieve this with the help of Gradle’s built-in capabilities as shown in listing 8.

buildSrc/src/main/groovy/com/bmuschko/linkverifier/Distributions.groovy

package com.bmuschko.linkverifier

import org.gradle.api.DefaultTask
import org.gradle.api.Project
import org.gradle.api.tasks.TaskAction
import org.gradle.api.tasks.InputDirectory
import org.gradle.api.tasks.OutputDirectory

class Distributions extends DefaultTask {
    @InputDirectory
    File inputDir = project.file('.gogradle')

    @OutputDirectory
    File outputDir = project.file("$project.buildDir/distributions")

    Distributions() {
        group = 'GoGradle'
        description = 'Builds packaged distributions for all prebuilt binaries.'
    }

    @TaskAction
    void create() {
        inputDir.listFiles(new CrossCompiledFileFilter(project)).each { preBuiltLib ->
            if (preBuiltLib.name.contains('windows')) {
                ant.zip(destfile: "${outputDir}/${preBuiltLib.name}.zip") {
                    fileset(dir: inputDir) {
                        include(name: preBuiltLib.name)
                    }
                }
            } else {
                def tarFile = "${outputDir}/${preBuiltLib.name}.tar"
                ant.tar(destfile: tarFile) {
                    tarfileset(dir: inputDir) {
                        include(name: preBuiltLib.name)
                    }
                }
                ant.gzip(destfile: "${tarFile}.gz", src: tarFile)
                project.delete(tarFile)
            }
        }
    }

    private static class CrossCompiledFileFilter implements FilenameFilter {
        private final Project project

        CrossCompiledFileFilter(Project project) {
            this.project = project
        }

        boolean accept(File f, String filename) {
            filename.endsWith("_${project.name}")
        }
    }
}

Listing 8. Packaging prebuilt binaries into archive files

In the build script, we just need to create a new task of type Distributions and ensure that it can only run after the prebuilt libraries have been created.

build.gradle

import com.bmuschko.linkverifier.Distributions

task dist(type: Distributions) {
    mustRunAfter build
}

Listing 9. Instantiating a task for creating distributions

Summary

Go’s popularity is growing. Gradle can help to automate the build process for Go-based projects. In this post, we had a look at the community plugin GoGradle. GoGradle is a powerful and easy-to-use addition to the Gradle plugin ecosystem. It provides tasks for compiling and testing Go sources. It can also help with creating prebuilt libraries for distribution.

We’ve seen that the plugin can reduce a lot of boiler plate code which normally would have to be written as shell scripts. In some areas, the plugin can be improved even further. To drive innovation, I opened a list of issues on the plugin’s GitHub repository.