Best practices for writing Jenkins shared libraries

jenkins cicd build

October 31, 2019

Introduction

Pipeline definitions in Jenkins start small and maintainble. You write a Jenkinsfile that declares a couple of stages. Nothing dramatic, simple, understandable code. As your adoption of Jenkins and "pipelines as code" grows within the organization, you’ll find out that other teams are copy-pasting pipeline code all over the place. What works for one project should work for other projects, right?! Soon requirements become more complex and your organization will enter a world of pain of unmaintainable, duplicated code.

Jenkins provides the concept of reusable pipeline functionality through shared libraries. With the help of shared libraries, you can implement more complex logic that can be shared across multiple pipelines. Shared libraries are somewhat comparable to libraries in other languages e.g. JARs in the JVM world or Go packages.

The Jenkins user guide explains the mechanics of shared libraries but gives very little guidance on best practices. In this blog post, I am going to explain what I consider to be best practices. Many of the recipes described here are not really specific to Jenkins shared libraries but are applicable to software development in general.

As a quick reference, you can directly jump to a specific aspect:

Designing shared libraries
Building shared libraries
Testing shared libraries
Versioning shared libraries
Documenting shared libraries

Designing shared libraries

Global variables vs. class implementations

Shared libraries offer two ways to implement reusable logic:

Global variables represent a loosely-defined script with little structure. Those scripts usually containing one or many methods and/or variables. Essentially, global variables are just externalized scripts that can be imported into a Jenkinsfile to break down logic. The naming doesn’t fully express its purpose and can lead to issues among team members when talking about shared library terminology.
Class implementations represent the alternative to scripts. They support a much more structured approach to breaking down functionality into packages and classes, a coding approach you are likely already familiar with if you are writing application source code. One of the major benefits of class implementation is the cabilility to declare and download external libraries via Groovy Grape.

Personally, I am not a fan of using global variables. The ability to expose variables with a global scope often times leads to confusion when tracking down its definition and the place in the code that assign new values. Morever, a script is not well-suited for implementing more elaborate logic as it can easily become spaghetti code.

For the most part, I start implementing Jenkins shared libraries as classes right away. The approach feels much more natural to JVM programmers, helps with structuring and evolving the code over time and puts you in a good position to actually writing tests for the code. You can read more about testing aspects in the decicated section below.

Declarative vs. scripted

Shared libraries can even define a templated pipeline definition with the purpose of standardizing typical project types. For example, you might decide that a Java project in your organization should require the change to pass through the stages compilation, unit testing, integration testing and publishing.

There are some intricate differences between the syntax of declarative and scripted pipelines e.g. a stage in a scripted pipeline does not need to specify a nested steps blocks. Syntax differences (especially when imported from shared libraries) can lead to a lot of confusion among consumers and result in unexpected runtime errors. Try to implement shared libraries with the declarative syntax as the preferred choice. The declarative syntax will likely see more support and new features by CloudBees in the future. Most importantly, document this decision for any of your consumers.

API design

Independent of your choice to use global variables or class implementations, you will have to think about the method signatures you want to expose to consumers. Try to put yourself into the shoes of other developers that are calling the functionality from their pipeline. As a general guideline, I’d recommend to ask yourself the following questions when designing the API of your shared library.

Does the naming express the functionality it provides?
Is the signature of the functionality expressive enough?
Do I maybe require the end user to provide a long list of parameters? Can I minimize the number of parameters? Should I potentially introduce a data object for providing input values?
Is the functionality documented with the help of Groovydoc? For more information on documentation, see the dedicated section below.

Groovy as a language does not enforcing static typing of variables and methods. You can happily just mark everything with def or omit the type altogether. I would highly advise against it as the typing can act as subtle documentation to consumers. Try to provide a type whenever you can. It will give consumers a hint on what kind of value your are expecting.

Limitations

Initially, Jenkins shared libraries might give you the impression that you are writing plain old Groovy code. That’s only true to a certain extend. While the pipeline uses the Groovy compiler and parser, it runs the pipeline and shared libraries with a special interpreter. This interpreter induces certain limitations that affect the way you need to structure your code.

It doesn’t handle inheritance or method overrides very well which can lead to runtime issues. The resulting error messages are hard analyze and debug, also known as the notorious CPS mismatch errors. In most cases, I replace inheritance with delegation to work around this limitation.
Given my recommendations about static typing above, you might feel tempted to use Groovy’s @CompileStatic annotation to enforce the coding style. Unfortunately, Jenkins doesn’t deal well with the annotation and generates a runtime error.
Not so much a limitation but more of a requirement. Don’t forget to implement java.io.Serializable by any of your classes to avoid issues for pipelines in case the Jenkins server needs to be restarted.

Building shared libraries

Building a Jenkins shared library becomes so much easier if you work on it in an IDE. Especially when writing Groovy classes, you will want features like auto-completion, easy navigation between classes and compilation support. IntelliJ does a great job of deriving the project setup from a build definition.

Listing 1 shows a sample Maven build script. Pointing IntelliJ to the build script when opening the project will automatically derive the source directories, set up the proper JDK version and configure the Groovy compiler. Please note that the source directory conventions of a shared library does not follow the standard Maven conventions and therefore has to be reconfigured.

pom.xml

<?xml version="1.0" encoding="UTF-8"?>

<project>
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.bmuschko.jenkins</groupId>
    <artifactId>jenkins-shared-lib</artifactId>
    <name>jenkins-shared-lib</name>
    <version>1.0.0</version>

    <properties>
        <jdk.version>8</jdk.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <groovy.cps.version>1.30</groovy.cps.version>
        <groovy.version>2.4.12</groovy.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.cloudbees</groupId>
            <artifactId>groovy-cps</artifactId>
            <version>${groovy.cps.version}</version>
        </dependency>
        <dependency>
            <groupId>org.codehaus.groovy</groupId>
            <artifactId>groovy</artifactId>
            <version>${groovy.version}</version>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>src</sourceDirectory>
        <resources>
            <resource>
                <directory>resources</directory>
            </resource>
        </resources>

        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <compilerId>groovy-eclipse-compiler</compilerId>
                    <source>${jdk.version}</source>
                    <target>${jdk.version}</target>
                    <encoding>${project.build.sourceEncoding}</encoding>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-eclipse-compiler</artifactId>
                        <version>3.5.0-01</version>
                    </dependency>
                    <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-eclipse-batch</artifactId>
                        <version>2.5.8-02</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>
</project>

Listing 1. Building a shared library with Maven

I tried to locate the compatible Jenkins versions used to compile and parse a Jenkins pipeline. The only hint I could find was under Manage Jenkins > About Jenkins. For my version of Jenkins, the Maven GAV is org.codehaus.groovy:groovy-all:2.4.12. In the build script for your shared library, you should rely on that exact version to ensure optimal version compatibility. You will also get a hint about the compatible Groovy version by looking at the parent POM of the dependency com.cloudbees:groovy-cps.

There’s nothing speaking against setting up a similar build with Gradle. While the syntax is completely different, the essence of the configuration would be the same. In a nutshell, apply the Groovy plugin, declare the relevant dependencies and reconfigure the source directory. There’s no need to configure the Groovy compiler explicitly. The Groovy plugin already takes care of it.

Testing shared libraries

Any code used in production should be tested. And by that I do not necessarily mean by hand. The Jenkins documentation doesn’t provide any hints on how to approach this problem. Here are the possible ways to tackle the testing for shared libraries.

Setting up pipeline job for the sole purpose of consuming shared library code to see how things pan out. Sooner or later, you will have to go through this type of testing as there’s no way to emulate the runtime behavior of Jenkins.
Write unit tests and mock out every portion of the code that calls off to the Jenkins API. This approach is really only possible if you are writing shared libraries as class implementations so that you can put the proper abstractions in place.

Point 2 requires a little bit of extra work from your end. I am going to describe the setup below. There’s also a project called "Jenkins Pipeline Unit testing framework", however, I didn’t manage to even execute a single working test case with it successfully.

Setting up the build

For writing unit tests, you have to decide on a test framework. The most prominent options are JUnit and Spock. Additionally, you will want to pull in a mock framework if you decide to go with JUnit. The Maven build shown in listing 2 uses JUnit 5 in combination with Mockito. You can also see that I am configuring the build to look at a non-standard test sources directory.

pom.xml

<project>
    ...
    <properties>
        ...
        <junit.jupiter.version>5.5.2</junit.jupiter.version>
    </properties>

    <dependencies>
        ...
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>${junit.jupiter.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-params</artifactId>
            <version>${junit.jupiter.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit.jupiter.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-core</artifactId>
            <version>3.0.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        ...
        <testSourceDirectory>test</testSourceDirectory>

        <plugins>
            ...
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
            </plugin>
        </plugins>
    </build>
</project>

Listing 2. Testing a shared library with Maven

Mocking the Jenkins API

You will want to put yourself into a good position for mocking calls to the Jenkins API. I recommend introducing an interface that can hide all those calls. You can find an example in listing 3. You will likely only need to add a couple of methods and not the full Jenkins API.

JenkinsExecutor.groovy

package com.bmuschko.jenkins

interface JenkinsExecutor extends Serializable {
    void stage(String name, Closure config)
    String sh(String command)
    void echo(String message)
    ...
}

Listing 3. Hiding the Jenkins API behind an interface

The implementation of the interface looks straightforward, as shown in listing 4. First of all, you’ll have to inject the reference to the Jenkins script. This is done in the constructor of the class. The method simply uses the script reference to call the relevant Jenkins APIs.

DefaultJenkinsExecutor.groovy

package com.bmuschko.jenkins

class DefaultJenkinsExecutor implements JenkinsExecutor {
    private final script

    DefaultJenkinsExecutor(script) {
        this.script = script
    }

    @Override
    String sh(String command) {
        script.sh(script: command, returnStdout: true)
    }

    @Override
    void echo(String message) {
        script.echo(message)
    }

    @Override
    void stage(String name, Closure config) {
        script.stage(name, config)
    }
    ...
}

Listing 4. Calling to the Jenkins API via the script reference

Any code in your shared library that needs to call the Jenkins API requires a reference to the interface JenkinsExecutor. For example, the class below uses the sh and the echo method.

MyCustomSteps.groovy

package com.bmuschko.jenkins

class MyCustomSteps implements Serializable {
    private final JenkinsExecutor jenkinsExecutor

    MyCustomSteps(JenkinsExecutor jenkinsExecutor) {
        this.jenkinsExecutor = jenkinsExecutor
    }

    void execute() {
        jenkinsExecutor.sh('ls -l')
        jenkinsExecutor.echo('Done!')
    }
}

Listing 5. Using the Jenkins API facade

Now that we hid the Jenkins implementation details behind an interface, we can simply create a mock object for it. The test case in listing 6 creates a mock object for JenkinsExecutor with Mockito, injects the instance into the class under test and emulates its behavior as needed.

DefaultJenkinsExecutor.groovy

package com.bmuschko.jenkins

import com.bmuschko.jenkins.JenkinsExecutor
import org.junit.jupiter.api.Test

import static org.mockito.Mockito.*

class Test {
    JenkinsExecutor jenkinsExecutor = mock(JenkinsExecutor)
    MyCustomSteps myCustomSteps = new MyCustomSteps(jenkinsExecutor)

    @Test
    void "can execute custom steps"() {
        when(jenkinsExecutor.sh('ls -l')).thenReturn("""total 1
-rw-r--r--@  1 bmuschko  staff  889 Jun 13  2018 README.adoc""")
        myCustomSteps.execute()
        verify(jenkinsExecutor).sh('ls -l')
        verify(jenkinsExecutor).echo('Done!')
    }
}

Listing 6. Mocking Jenkins API calls in a test

Versioning shared libraries

Jenkins shared libraries do not need to be bundled or published like typical libraries in the JVM ecosystem. In the Jenkins management section, you create a reference to the SCM repository hosting the code. It might sound very tempting at first to point to the master branch for the library, however, the result is a potential unreliable build. Any changes made to the branch will be pulled down automatically by the consuming pipeline. While that might seem convenient for rolling out new features, the same concept also applies to bugs.

I would highly recommend tagging your commits in version control and pinning to those tags from your pipeline. Effectively, the tag acts as the version of the shared library. I used semantic versioning with great success in the past.

Jenkinsfile

@Library('deployment@4.2.6')

import com.bmuschko.jenkins.Deployment
...

Listing 7. Using a shared library by referencing a concrete tag

It goes without saying that every "release" aka tag should be documented with the help of release notes. Release notes can look as simple as a Markdown or Asciidoc file in the root directory of the shared library. If you are rolling out new releases to a wider audience, those release notes will help consumers to decide which feature set they actually want to adopt.

Documenting shared libraries

For most consumers, a Jenkins shared library looks like a black box. You might understand the purpose of the shared library but without any documentation you have no idea what to call. There are multiple levels of documentation I found useful:

High-level documentation that answers the question "What problem does the shared library solve?".
Groovydoc to document the API of the shared library that answers the question "How can I used it?".
Usage examples that show code snippets of the shared library as part of a pipeline.

Documentation for 1 and 3 can easily be added as Markdown and Asciidoc files to the same repository or can reside on a Wiki page. Groovydoc needs to be generated and published for later reference. Adding Groovydoc support to a Maven build is not very hard. You can add one of the available plugins, as shown in listing 8.

pom.xml

<project>
    ...
    <build>
        <plugins>
            ...
            <plugin>
                <groupId>com.bluetrainsoftware.maven</groupId>
                <artifactId>groovydoc-maven-plugin</artifactId>
                <version>2.1</version>
            </plugin>
        </plugins>
    </build>
</project>

Listing 8. Generating Groovydoc in a Maven build

Publishing the Groovydoc files is a bit more complicated. Hosting the API documentation on Jenkins may be a good way to start. Assuming you already have a build pipeline in place for your shared library, adding another step for generating and publishing the API docs is easy. The listing below demonstrates such a stage in an example pipeline. Additionally, you will have to configure Jenkins' content security policy.

Jenkinsfile.groovy

pipeline {
    ...
    stage('Publish API Docs') {
        when {
            branch 'master'
        }
        steps {
            sh './mvnw groovydoc:generate'
        }
        post {
            success {
                publishHTML(target: [
                    allowMissing: false,
                    alwaysLinkToLastBuild: false,
                    keepAll: true,
                    reportDir: 'target/groovydoc',
                    reportFiles: 'index.html',
                    reportName: 'API Docs'])
            }
        }
    }
}

Listing 9. Generating and publishing Groovydoc in a Jenkins pipeline

Summary

Shared libraries can be a powerful tool for organizations interested in writing reusable pipeline logic or even standardizing full pipeline definitions. Jenkins does not take a strong stance on best practices. This article identifies recipes that worked well for me. We covered design, build, test, versioning and documentation aspects. I hope you can apply those recipes to your own projects to avoid common pitfalls.